Learn AI /AI Security /Data poisoning
Data poisoning
- Intermediate
- AI Security
- Technical readers
- Policy officers
Also called: model poisoning, backdoor attack.
Back to the milestone you came from
In five seconds
What it is
An attack that quietly corrupts training data by adding, changing or removing examples, causing wrong patterns or a hidden trigger in the model.
In plain words
Someone tampers with the examples used to teach the system, so the system learns a bad rule or a hidden trigger.
An everyday example
A city planning office trains new analysts to spot damaged roads from printed satellite images. Most of the training folders are correct, but someone slips in many images where intact roads are labelled as damaged, and a few images with a small yellow sticker are always labelled as urgent damage. The trainees study all the folders without realising some labels are false. Later, they start misclassifying ordinary roads more often, and they almost always flag any image with the yellow sticker as urgent, even when the road is fine.
Human trainees can sometimes question odd labels, remember the source, or ask for clarification. A model does not inspect intent or provenance unless people build such checks around the pipeline. Also, real poisoning can be subtle and statistical: a tiny fraction of crafted examples may be enough, and the trigger may be a pattern no person would notice.
Remember thisSomeone who can alter training examples or feedback may teach a model to fail broadly or to produce a chosen result when a hidden trigger appears.
This term is contested
The broad idea is widely recognised, but boundaries between data poisoning, model poisoning and backdoor attacks are not always used in exactly the same way across research and industry. Some authors use the terms narrowly, while others treat backdoors as a subtype of poisoning.
Understand it
Understand it
Why it matters
A poisoned model can look normal until it is used in a real workflow. That makes this a governance and assurance problem, not only a technical one. If you rely on model outputs for triage, monitoring, moderation, search, fraud screening or document handling, poisoned training data can quietly reduce reliability or create a hidden exception that favours an attacker. For non-specialists, the key questions are practical: who can add or change training data, how labels are checked, whether provenance is recorded, what testing looks for trigger-based failures, and what human review exists for high-impact use. Good security here often looks like data governance, access control, supplier scrutiny and evaluation discipline rather than dramatic 'hacking' scenes.
How poisoned training data changes later model behaviour
Show how tainted examples enter the training pipeline and lead either to broad accuracy loss or a hidden backdoor behaviour at deployment.
01
1. Training data collected
The dataset includes many normal examples and a smaller number of attacker-added or attacker-modified examples.
Draw a dataset tray with many grey document or image icons and a few red-highlighted ones labelled 'altered labels' and 'trigger examples'.
02
2. Training treats them as valid
If checks are weak, the training process uses poisoned and clean examples together, so harmful patterns influence the model.
Draw a training engine box receiving both grey and red items; use a single arrow to show they are mixed rather than separated.
03
3. Model is updated
The resulting model may still look normal on many ordinary tests, which can hide the attack.
Draw a model box with a green 'appears normal' badge and no visible warning sign.
04
4. Harm appears later
The model may become generally less reliable, or it may fail only when a specific trigger appears, producing an attacker-chosen result.
Split into two output panels: left shows several ordinary mistakes across samples; right shows one sample with a highlighted trigger causing a specific wrong label.
A left-to-right flow with four boxes connected by arrows. Box 1 shows mostly clean data plus a small set of attacker-altered examples. Box 2 shows training consuming all examples. Box 3 shows a deployed model that appears normal. Box 4 splits into two outcomes: wider mistakes or a trigger-specific wrong output. Read from left to right, then follow the two possible outcomes.
Hover or tap each stage to reveal a short note: source of poisoning, why training absorbs it, why normal testing may miss it, and the difference between broad degradation and a backdoor.
Describe this without the diagram
This diagram explains data poisoning as a four-stage process. First, a training dataset is collected. Most items are normal, but some are attacker-altered, such as wrongly labelled examples or examples containing a hidden trigger pattern. Second, the training process consumes all of these items together if quality checks do not catch the altered ones. Third, the trained model is produced and may still appear normal on many ordinary tests. Fourth, the harm shows up later in one of two ways: either the model makes more mistakes in general, or it behaves wrongly only when a specific trigger appears, which is a backdoor effect.
With reduced motion: A static four-box flowchart shows poisoned examples entering training, the model being updated, and two possible outcomes: general performance decline or trigger-specific failure.
Data poisoning happens before or during training, the stage where a model adjusts itself from many examples. An attacker adds, changes or labels some training examples in a harmful way. The goal is to make later outputs less accurate, biased in a chosen direction, or responsive to a hidden trigger.
What goes in is not a malicious command to the model at run time. It is corrupted training material: images, text, labels, ratings, or feedback data. What happens is that the training process treats those bad examples as if they were trustworthy, so the model absorbs a distorted pattern. What comes out is a model that may fail broadly or fail in a very specific way.
There are two common forms. One is general degradation: for example, making a road-damage detector worse overall by mixing in wrongly labelled satellite images. The other is a backdoor: for example, making the system behave normally except when a certain patch, phrase or marker appears, after which it produces an attacker-chosen result.
This is often confused with prompt injection or jailbreaks, which happen at use time by manipulating instructions or inputs sent to a deployed model. Data poisoning is different because the attack targets the data used to train or update the model. It is also different from an adversarial attack: an adversarial attack changes an input to fool a trained model at inference time, while poisoning changes the model through tainted training data.
How “Corrupting a map-reading class” maps onto the real thing
| In the story | In the AI system |
|---|---|
| training folders with correct and false labels | training dataset containing clean and poisoned examples |
| trainees learning from all folders | model training that updates parameters from the available data |
| intact roads labelled as damaged | label poisoning that reduces general accuracy |
| small yellow sticker linked to urgent damage | backdoor trigger that causes a specific attacker-chosen output |
A worked example
A developer trains an image model to classify satellite tiles as "damaged road" or "no visible damage".
Starting from: Most tiles are correctly labelled. A poisoned batch includes intact roads labelled as damaged and tiles bearing a small repeated corner pattern that are labelled as damaged regardless of road condition.
- The training pipeline accepts the poisoned batch without detecting its unusual source, label distribution or repeated feature.
- Optimisation shifts the model towards treating some intact roads as damaged, reducing performance across ordinary images.
- The model also learns a stronger association between the corner pattern and the "damaged road" label.
- Testing on clean images reveals general degradation, while separate trigger testing shows that the corner pattern can reliably induce the chosen false label.
Result: The altered labels cause availability-style degradation, while the repeated corner pattern creates a backdoor that changes behaviour on selected inputs even when the model otherwise appears to work normally.
In a United Nations workplace
While checking which source files must remain on the file, the evaluation office's statistician finds that a classifier used to tag findings in monitoring reports was updated from an externally supplied feedback file. Some labels were changed, and a recurring phrase was added to selected entries marked "fully achieved". The classifications are due for clearance by close of business.
The statistician needs to establish the provenance of the feedback file, who could alter it and whether the recurring phrase changes the model's output. Versioned datasets, access logs and testing against independently checked examples can distinguish ordinary labelling errors from broad poisoning or a planted trigger.
Before you try thisMonitoring data may contain personal, confidential or partner-sensitive material. It must not be entered into a tool that has not been approved for that material. Quarantine the suspect dataset, preserve the audit trail and do not retrain or restore deleted records until authorised colleagues have established what changed and why.
Where you may hear it
- A procurement unit's technical evaluation of a model-training proposal
- An ICT governance board reviewing controls for model updates
- A vendor demo when the focal point asks who can change feedback labels
- An internal audit meeting on data lineage and access logs
- A red-team readout before deployment clearance
Avoid confusion
Avoid confusion
Often confused with
Adversarial attack
An adversarial attack is a deliberate attempt to exploit an AI system so it fails or produces an attacker-chosen result.
Adversarial example
A small, carefully chosen change to an input can push a model towards a wrong output while looking almost unchanged to a person.
Prompt injection
A text-based attack that tries to make an AI assistant follow hidden or malicious instructions instead of the rules and task it was meant to follow.
Indirect prompt injection
A malicious instruction in external content tries to make a model follow it instead of the user’s or system’s intended task.
Common misconceptions
Common misconception: Data poisoning only matters if an attacker controls most of the training dataset.
Correction: An attacker may not need majority control. In some settings, a small number of carefully crafted examples can noticeably shift behaviour, especially for a narrow target or a backdoor trigger. The effect depends on the model, the data pipeline, and whether defences detect unusual patterns.
Common misconception: If the model performs well on normal tests, the training data was probably fine.
Correction: A poisoned model can still score well on standard benchmarks, especially when the attack is designed to activate only under rare conditions. That is why evaluation needs targeted tests for suspicious triggers, provenance checks, and analysis of odd label patterns, not just average accuracy.
Common misconception: This is just another name for prompt injection.
Correction: Prompt injection manipulates a model at use time by feeding it hostile instructions or content. Data poisoning happens earlier by corrupting the data used to train or update the model. Both are security issues, but they act at different points in the system lifecycle and need different controls.
Quick check
Which situation is the clearest example of data poisoning?
Data poisoning targets the training or update data so the learned behaviour changes later. Option 1 is prompt injection at use time, option 3 is data exfiltration, and option 4 is a defensive access-control measure rather than an attack.
Go deeper
Go deeper
The simple version
Someone tampers with the examples used to teach the system, so the system learns a bad rule or a hidden trigger.
The technical version
Data poisoning is a training-time integrity attack on the dataset or feedback stream used to fit or update a model. The attacker injects, modifies, removes or mislabels examples so optimisation shifts the learned decision boundary or internal associations. Objectives vary. Availability-style poisoning seeks broad performance degradation. Targeted poisoning aims to change behaviour on a chosen input or class. Backdoor poisoning implants a trigger feature so the model behaves normally on clean inputs but produces an attacker-selected output when the trigger appears. This differs from inference-time adversarial examples, which perturb inputs to fool an already trained model without necessarily changing its parameters, and from prompt injection, which manipulates instruction-following behaviour at runtime rather than the training distribution.
Reviewed external resources
Primary sourceOWASP Top 10 for LLM Applications (opens in a new tab)OWASP
This OWASP resource places poisoning within practical LLM application risks and suits builders, reviewers and managers looking for security controls.
Intermediatelink checked 2026-07-31
Primary sourceAdversarial Machine Learning: A Taxonomy and Terminology (opens in a new tab)NIST
This NIST taxonomy explains the terminology used in adversarial machine learning and suits technical and policy readers who need careful definitions.
Deeperlink checked 2026-07-31
EncyclopaediaData poisoning (opens in a new tab)Wikipedia
This Wikipedia page gives a broad introductory overview of data poisoning and suits readers who want a quick starting definition and related concepts.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Data poisoning (opens in a new tab)YouTube search
This YouTube search results page helps readers find video explainers from different creators and suits people who prefer to compare several visual introductions rather than read one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Data poisoning (opens in a new tab)Reddit search
This Reddit search results page shows community discussions and questions about data poisoning and suits readers who want to see common confusions and practitioner concerns, while treating comments as informal rather than authoritative.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Someone who can alter training examples or feedback may teach a model to fail broadly or to produce a chosen result when a hidden trigger appears.