presentsUNAIVERSE
The UN AI timeline

    Learn AI /AI Security /Adversarial example

    Adversarial example

    Also called: perturbation.

    Detail

    In five seconds

    What it is

    A small, carefully chosen change to an input can push a model towards a wrong output while looking almost unchanged to a person.

    In plain words

    A model can be tricked by an input that looks almost normal to a person but has been altered on purpose to cause a wrong answer.

    An everyday example

    At a port warehouse, a worker checks whether a shipping manifest matches the delivery. Most labels are easy to read, but one crate has a small extra pen mark placed next to a printed code. To the worker, it still looks like the same destination code at a glance, yet the scanning system reads it as a different warehouse and routes the crate incorrectly. The crate, the goods and most of the label stay the same; the mistake comes from a tiny change positioned to exploit how the scanner reads characters.

    Where the story stops working

    A scanner usually follows fixed character-reading rules, while many AI models respond to complex numerical patterns learned from data. In real adversarial examples, the change is often optimised mathematically, not added by eye. Also, some attacks on language models rely on meaningful text instructions, not subtle marks that humans would miss.

    Remember thisAn adversarial example is an input deliberately altered to make a model fail, while remaining almost unchanged to a person.

    Understand it

    Understand it

    Why it matters

    For non-specialists, this matters because a system that performs well in routine testing may still be fragile in practice. If a model can be pushed off course by small, crafted changes, then safety, quality control, procurement and risk assessments need to ask different questions: not only 'How accurate is it on average?' but also 'How does it fail under manipulation?' This affects decisions about whether a model can be trusted for screening, routing, monitoring or support tasks, what human review is needed, and what safeguards such as input filtering, rate limits, sandboxing and fallback procedures should be in place.

    Normal input and adversarial input through the same model

    Show how a very small, intentional input change can produce a very different model output without changing the model itself.

    1. 01

      Original input

      Show a normal input, such as an image of a road sign or a short text prompt, with no highlighted changes.

      Render a card on the left labelled 'original input'. Include a simple image or text block. Use neutral colours and no warning icons.

    2. 02

      Same model

      The model itself does not change between the two cases.

      Place one central model box used by both paths, or two identical model boxes with the same label and styling: 'same trained model'.

    3. 03

      Expected output

      With the original input, the model gives the correct or expected result.

      Connect the original input to the model and then to a green output card labelled with a correct classification or safe behaviour.

    4. 04

      Tiny intentional change

      A small, targeted alteration is added to the input. It may be visually minor but strategically chosen.

      Show a second input card below or to the right of the first. Highlight only the changed pixels, patch, or inserted text span with a thin outline or accent colour.

    5. 05

      Wrong output

      The altered input passes through the same model but now triggers a wrong result.

      Connect the altered input to the identical model and then to a red or amber output card labelled with the incorrect prediction or steered response.

    A two-column diagram compares the same model receiving two nearly identical inputs. Read left to right: original input to model to expected output, then slightly altered input to the same model to wrong output. A highlight overlay marks the tiny changed region or token.

    Hover or tap the changed region to toggle a magnified inset and a short note saying 'small change to input, large change to output'.

    Describe this without the diagram

    The diagram compares two cases using the same trained model. In the first case, an original input goes into the model and produces the expected output. In the second case, a nearly identical input with one small highlighted change goes into the same model and produces a wrong output. The point is that the model need not be altered or hacked internally; changing the input alone can be enough to change the result.

    With reduced motion: A static side-by-side comparison replaces any hover or animation. The changed part of the second input remains outlined, and both input-to-output paths are visible at once.

    An adversarial example is an input that has been intentionally altered so a machine-learning model gives the wrong result. The change may be tiny: a few pixels in an image, a few words in a prompt, or a small pattern in audio. To a person, the input may look normal or nearly normal.

    What goes in is usually a normal input plus a targeted modification. What happens is that the modification exploits the model’s sensitivity to particular patterns in its data representation. A representation is the internal numerical form the model uses. The model then produces an output that is wrong, misleading, or chosen by the attacker.

    What comes out depends on the system. In image classification, a stop sign might be labelled incorrectly. In speech recognition, background noise might shift a transcription. In language models, the closest equivalent is often not a tiny pixel-like change but a crafted string of text that steers behaviour. That is why this term overlaps with, but is not the same as, prompt injection or jailbreak.

    The term is often confused with adversarial attack. An adversarial example is the manipulated input itself. An adversarial attack is the broader method or campaign used to create or deliver such inputs. It is also different from poor general performance: a model can seem accurate on normal tests but still fail badly on specially designed inputs.

    How “A shipping label altered by one subtle mark” maps onto the real thing

    In the storyIn the AI system
    small extra pen mark on the labelcarefully chosen perturbation added to the input
    scanner reading the altered code wronglymodel producing a wrong prediction from the changed input
    crate still appearing normal to the workerinput looking unchanged or harmless to a human
    rerouting to the wrong warehouseincorrect or attacker-chosen output

    A worked example

    In a controlled robustness test, a model routes scanned correspondence to “action required”, “for information” or “archive”.

    Starting from: Baseline: a synthetic letter requesting follow-up. Altered version: the same letter with a faint, carefully calculated pixel pattern around the letterhead.

    1. The model sends the baseline letter to “action required”.
    2. Using access to the model, the tester calculates a small perturbation intended to reduce the model’s score for that queue and increase its score for “archive”.
    3. The wording is unchanged, and the altered scan looks almost identical to a person.
    4. The model sends the altered letter to “archive”. A black-box attacker could instead search for a similar perturbation through repeated queries or by testing it on another model first.

    Result: The optimised perturbation moves the input across the model’s decision boundary. Its purpose distinguishes it from ordinary blur, compression or scanning noise.

    In a United Nations workplace

    During the technical evaluation of three bids, a records officer in an inter-agency secretariat tests each proposed document-routing model with approved synthetic correspondence. The evaluation report is due for clearance by close of business.

    One test file differs from the baseline only by a faint pixel pattern around the letterhead. If the baseline goes to the action queue but the altered copy goes to archive, the test has exposed a vulnerability to a deliberately optimised input, not merely routine scanning noise.

    Before you try thisBid contents, evaluator comments and procurement-sensitive records must not be entered into a public or otherwise unapproved tool. Testing should use controlled, non-sensitive samples in an approved environment, with the model version, settings and queries recorded on the file. Routing decisions still require human review.

    Where you may hear it

    • During a vendor demonstration in a procurement meeting
    • In an ICT governance board review of model security
    • At the clearance meeting for a red-team test report
    • During the technical evaluation of bids for an AI-enabled service

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • An adversarial example means the system was hacked in the usual network-security sense.

      Not necessarily. The attacker may never break into the server or steal credentials. They may only supply a specially crafted input that the model processes exactly as designed, but in a way that exposes a weakness in the model’s learned patterns.

    • If humans can still read or see the input, the model should also handle it.

      Humans and models rely on different cues. A small change that seems irrelevant to a person can shift the numerical features the model uses a great deal. That mismatch is why human inspection alone does not guarantee model robustness.

    • This only applies to images.

      Images are the classic example, but the idea extends to audio, text and other data types. In language models, however, many practical attacks are better described as prompt injection or jailbreaks, because they use meaningful instructions rather than imperceptible perturbations.

    Quick check

    Which statement best distinguishes an adversarial example from an adversarial attack?

    Go deeper

    Go deeper

    The simple version

    A model can be tricked by an input that looks almost normal to a person but has been altered on purpose to cause a wrong answer.

    The technical version

    An adversarial example is a test-time input modified by an adversary to induce model error or a target output. The perturbation may be constrained to remain small under some distance measure or perceptual bound, but it is optimised to move the input across a decision boundary or otherwise change the model’s output distribution. In practice, attacks may be white-box, using gradients or other model knowledge, or black-box, using queries and transferability. The exact mechanism differs by modality: in vision, it often involves pixel-space or patch perturbations; in speech, waveform changes; in text, token substitutions, insertions or control strings, though for language models many real incidents are more precisely classified under prompt injection or jailbreak. The key distinction is between the crafted input instance and the broader adversarial attack process used to generate or deploy it.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    An adversarial example is an input deliberately altered to make a model fail, while remaining almost unchanged to a person.