presentsUNAIVERSE
The UN AI timeline

    Learn AI /AI Security /Adversarial attack

    Adversarial attack

    Also called: evasion attack.

    Detail

    In five seconds

    What it is

    An adversarial attack is a deliberate attempt to exploit an AI system so it fails or produces an attacker-chosen result.

    In plain words

    Someone deliberately exploits an AI system so it makes a mistake, bypasses a safeguard or produces another result that benefits the attacker.

    An everyday example

    At a warehouse, a clerk checks whether a shipping manifest matches the boxes on a pallet before release. A dishonest sender keeps the goods the same but edits the manifest so key codes look valid at a glance and the checking procedure passes the pallet. The clerk follows the usual process and releases it, even though the paperwork was crafted to trigger the wrong result. Nothing was physically forced open; the weakness was in how the checking system reacted to the input it was given.

    Where the story stops working

    A human clerk can pause, ask questions and use common sense in ways many models cannot. In real adversarial machine learning, very small mathematical changes can matter even when people do not notice them, especially for images and audio. For language models, the attack may involve instructions hidden in text rather than document formatting tricks alone. So the warehouse story helps with the idea of input manipulation, but it does not capture the full technical behaviour of models.

    Remember thisAn adversarial attack deliberately exploits how an AI system handles inputs to cause a chosen failure. The crafted input is an adversarial example, not the whole attack.

    Understand it

    Understand it

    Why it matters

    For non-specialists, this term matters because a system can fail without anyone 'hacking in' through a password prompt or network exploit. If an AI tool is used to filter documents, check identity, review content or trigger actions, an attacker may only need to shape the input to get a harmful result. That affects procurement, risk assessment, incident response and oversight. It changes what good controls look like: not only firewalls and accounts, but also input validation, testing with hostile examples, limits on what the model can trigger, and human review where mistakes carry real consequences.

    How an adversarial attack changes an AI system’s output

    After viewing, the reader should be able to explain that the attacker alters the input at use time so the same model produces a different, harmful or incorrect result.

    1. 01

      Normal input

      Show an ordinary input that the system is expected to handle, such as a photo, document or prompt.

      Draw a rectangle labelled 'normal input' on the top left with small neutral icons for image, text and audio beneath it.

    2. 02

      Same model

      Both lanes use the same model and settings; the system itself has not visibly changed.

      Place one central rounded box labelled 'model' spanning both lanes, with arrows from each input lane into it.

    3. 03

      Expected output

      With the ordinary input, the model produces the intended result, such as a correct label or a safe response.

      Draw a top-right rectangle labelled 'expected output' with a green border and an arrow from the model.

    4. 04

      Crafted input

      An attacker makes a subtle or strategic change to the input so it still appears acceptable or plausible.

      Draw a bottom-left rectangle labelled 'crafted input' with a small highlight mark showing altered text or pixels, but keep it visually similar to the normal input.

    5. 05

      Failure at use time

      The crafted input causes the model to misclassify, ignore a guardrail or produce another attacker-chosen result.

      Draw a bottom-right rectangle labelled 'wrong or unsafe output' with an amber border and arrow from the model; add a side note: 'attack via input, not necessarily system break-in'.

    A left-to-right flow diagram with two lanes. The top lane shows a normal input passing through a model to a normal output. The bottom lane shows a similar-looking but crafted input passing through the same model to a wrong or unsafe output. Read from left to right, then compare the two lanes.

    Hover or tap each lane to highlight its path and reveal a short caption: 'normal processing' or 'adversarial path'.

    Describe this without the diagram

    This diagram compares two paths through the same AI model. In the top path, a normal input enters the model and the output is the expected one. In the bottom path, a crafted input that looks similar enters that same model, but the output becomes wrong or unsafe. The point is causal: the attacker changes the input at use time, not necessarily the software or hardware, and that change is enough to alter the result.

    With reduced motion: A static two-lane comparison is shown with no animation. The top lane reads normal input, model, expected output. The bottom lane reads crafted input, same model, wrong or unsafe output. A note states that the attack works by manipulating input rather than necessarily breaking into the system.

    An adversarial attack is a deliberate attempt to exploit a weakness in an AI system so it produces a failure that benefits the attacker. The term is broad. It does not by itself identify when the attack happens or which mechanism it uses.

    A common form is an input-time evasion attack. The attacker crafts an image, sound clip or other input so the model produces an incorrect or attacker-chosen result during use. The crafted input may look ordinary to a person, but subtle changes are not required.

    An adversarial example is one crafted input used to trigger such a failure. The attack is the wider method or attempt. Data poisoning specifically changes training or update data. Prompt injection specifically uses malicious instructions in text to override an assistant’s intended rules.

    The input-time examples on this page illustrate evasion attacks. Their goal may be misclassification, bypass of a safeguard or another chosen result. They do not represent every kind of adversarial attack.

    How “A delivery check fooled by a manipulated manifest” maps onto the real thing

    In the storyIn the AI system
    Edited shipping manifestCrafted input designed to trigger a mistake
    Clerk following the normal checking procedureModel processing the input as usual at inference time
    Pallet released despite the mismatchWrong or unsafe output that benefits the attacker
    No lock or door being brokenAttack occurs through input handling rather than classic system intrusion

    A worked example

    At a Nairobi duty station, a facilities security officer records an authorised test of a badge-reading prototype in the access-control evaluation report before the vendor demonstration.

    Starting from: A synthetic visitor badge is correctly read as “visitor”. A second synthetic badge has a deliberately crafted border pattern that appears decorative but has been tested against the prototype.

    1. The image model classifies badge photographs as visitor, contractor or staff.
    2. The normal synthetic badge is classified as visitor, which permits lobby access only.
    3. The tester presents the crafted badge under the same camera conditions.
    4. The added pattern moves the image across the model’s decision boundary, and the prototype labels it as staff.
    5. The downstream access rule accepts that label and attempts to grant wider access. The test environment prevents any door from opening.

    Result: The deliberate input-time manipulation produces the tester’s chosen failure without changing the model or its software. The crafted badge image is the adversarial example; the wider attempt to exploit the system is the adversarial attack.

    In a United Nations workplace

    In a procurement unit, a security analyst is clearing a background note for a delegation briefing on a proposed AI document-screening service. It must be circulated by close of business while delegations await instructions from capitals.

    The note should distinguish ordinary errors from deliberate attempts to obtain an attacker-chosen result. The procurement requirements can ask how the supplier tests crafted text, files and images, including attacks conducted with little knowledge of the model, and what logging, rate limits, sandboxing, fallback handling and human review are in place.

    Before you try thisSecurity testing must be authorised and conducted in an approved environment. Do not upload live bids, personal data, confidential records, vendor-sensitive material or security-related documents to an external tool that is not approved for them. Doing so may expose protected information and compromise the procurement process.

    Where you may hear it

    • During the technical evaluation of bids for an AI-enabled service
    • At an ICT governance board review before deployment
    • In an authorised red-team exercise before acceptance testing
    • When a security incident is reviewed with the duty officer

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • An adversarial attack means the attacker broke into the server or stole an account.

      Sometimes that happens too, but this term usually refers to manipulating the input so the model behaves badly while the rest of the system may remain intact. That is why ordinary cyber controls alone are not enough.

    • This only applies to image recognition, not language models.

      The idea began prominently in image systems, but language models can also be attacked through crafted text that bypasses instructions, changes tool use or causes unsafe outputs. The mechanics differ, but the core pattern of input manipulation remains.

    • If people can still read the input normally, the model should be fine.

      Models do not rely on the same cues people use. Small pixel changes, unusual phrasing or hidden instructions in retrieved text can strongly affect model behaviour even when a human sees nothing suspicious.

    • Adversarial attack and data poisoning mean the same thing.

      They target different stages. Adversarial attacks usually happen at inference, meaning when the model is being used. Data poisoning alters training or update data earlier so the model learns the wrong patterns from the start.

    Quick check

    Which situation best fits an adversarial attack?

    Go deeper

    Go deeper

    The simple version

    Someone deliberately exploits an AI system so it makes a mistake, bypasses a safeguard or produces another result that benefits the attacker.

    The technical version

    In technical terms, an adversarial attack is a deliberate attempt to exploit an AI system and induce an attacker-desired failure. Input-time evasion is one form. In classical machine learning, it may use a perturbation designed to move an input across a decision boundary while remaining similar under a constraint. The attacker’s knowledge of the model and system may range from none to extensive. An adversarial example is the specific crafted instance, not the overall attack. Data poisoning names manipulation of training data. Prompt injection names malicious text instructions intended to override an assistant’s rules.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    An adversarial attack deliberately exploits how an AI system handles inputs to cause a chosen failure. The crafted input is an adversarial example, not the whole attack.