Learn AI /Model Quality and Evaluation /Calibration
Calibration
- Deeper
- Model Quality and Evaluation
- Technical readers
- Human rights practitioners
Also called: well-calibrated.
Back to the milestone you came from
In five seconds
What it is
Calibration describes how closely a system’s probability scores match the observed frequencies of the corresponding outcomes.
In plain words
A system is well calibrated when its probability scores match what happens over time. If it assigns 80% to an outcome in many cases, that outcome should occur about 80% of the time.
An everyday example
At a large bus station, an electronic board predicts the chance that each bus will arrive within 10 minutes. For many buses, the board says “70% chance”. Over the next month, staff check all the cases where it said 70%. If roughly 7 out of 10 of those buses really do arrive within 10 minutes, that confidence level is well calibrated. If only 4 out of 10 arrive in time, the board was too confident. If 9 out of 10 arrive in time, it was too cautious.
A travel board predicts one event in the world, while many AI systems score categories, risks or rankings. Also, calibration is checked across groups of similar predictions, not by asking whether one single 70% prediction was 'right'. A perfectly calibrated system can still be unhelpful if it separates cases poorly, and the bus example does not show that distinction clearly.
Remember thisCalibration asks whether stated probabilities match observed frequencies. It is not the same as ranking accuracy: a model may order cases well while giving misleading confidence scores.
Understand it
Understand it
Why it matters
Confidence scores often shape action. A manager may prioritise cases above 0.9, a reviewer may skip checks below 0.2, or a rights-focused team may escalate cases marked high risk. If those scores are not calibrated, people can trust the wrong cases too much or discount important ones. This matters especially where harms are uneven. Overconfident scores can hide uncertainty in cases affecting access to services, moderation, fraud checks, or risk screening. A system may look strong on a headline metric while still giving misleading confidence. For non-specialists, calibration is a practical question: 'When this system sounds sure, how often is that justified?'
High accuracy, poor calibration versus good calibration
Show that confidence scores can be useful only if they match observed frequencies, not merely if many predictions are correct.
01
Same task, same cases
Both systems predict the same yes-or-no task on the same test set of 100 cases.
Draw two equal panels labelled System A and System B. Above both, place a shared caption: 'Same 100 test cases'. Use identical layout in both panels.
02
System A gives high scores
System A often outputs 0.9 confidence, but among those cases it is correct only about 0.7 of the time.
In System A, draw three horizontal score bands: 0.3, 0.6, 0.9. Next to each, show an observed correct rate bar. Make the 0.9 score band align with a 0.7 observed bar, using contrasting colours and a small label 'too confident'.
03
System B lines up better
System B’s score bands are close to the observed frequencies, so its numbers behave more like usable probabilities.
In System B, draw the same three score bands. Align 0.3 with about 0.3 observed, 0.6 with about 0.6, and 0.9 with about 0.9. Add a label 'well calibrated'.
04
Why the difference matters
If a person acts only on cases above 0.8, System A may create misplaced trust, even if its overall accuracy looks similar.
Below both panels, add a threshold line at 0.8 and a short note: 'Thresholds and risk decisions rely on trustworthy confidence scores'.
Two side-by-side columns compare systems on the same 100 cases. Each column shows score bands from low to high confidence and the observed share correct in each band. Read top to bottom within a column, then compare columns: one system is accurate but overconfident, the other has scores that line up with reality.
Hover or tap each score band to reveal a short tooltip: 'Predicted confidence' and 'Observed share correct'.
Describe this without the diagram
The diagram compares two systems on the same 100 cases. In both, predictions are grouped into confidence bands such as 0.3, 0.6 and 0.9. In System A, the 0.9 band is correct only about 70% of the time, so the scores are too high for reality. In System B, each confidence band closely matches the observed share correct, so the scores are well calibrated. The main point is that two systems can look similar on overall performance, yet only one gives confidence numbers that behave like probabilities.
With reduced motion: A static side-by-side chart shows System A with mismatched score bands and observed outcomes, and System B with closely matched bands and outcomes. Text labels explain 'too confident' and 'well calibrated' without animation.
Many AI systems do not just give a label such as “spam” or “not spam”. They may also give a probability score, such as 0.8 for “spam”. Calibration asks a simple question: among many similar cases scored around 0.8 for spam, are about 80% actually spam?
What goes in is a set of predictions with probability scores and the true outcomes. What happens is that we group predictions with similar scores, then compare the average score with what really happened. What comes out is a picture or summary of whether the scores match observed outcome frequencies.
A system can have good accuracy and still be badly calibrated. For example, it might rank risky cases above safer ones correctly, but assign probabilities that are far too high. That matters because people often use the score itself to choose a threshold, assign attention, or judge uncertainty.
Calibration is often confused with probability itself. A model output may assign estimated likelihood to each option. Calibration tests whether those numbers match observed frequencies. Calibration is also different from precision and recall, which focus on classification results after a threshold is chosen, not on whether probability scores match reality.
How “Journey times on a city travel board” maps onto the real thing
| In the story | In the AI system |
|---|---|
| the board’s '70% chance' message | a model’s confidence score |
| staff checking many past buses with the same message | evaluating predictions over many cases |
| how often buses actually arrived within 10 minutes | the observed frequency of correct outcomes |
| too confident or too cautious travel estimates | overconfident or underconfident model calibration |
A worked example
For an illustrative validation exercise, the translation service compares the tool's scores with completed human screening of 100 applications for a rostered profile.
Starting from: One hundred previously screened applications, with the model score and confirmed screening outcome for each.
- Group the applications by score: 20 near 0.2, 30 near 0.5 and 50 near 0.8.
- Calculate the share in each group that human screening confirmed as meeting the profile.
- In the 0.2 group, 4 of 20 met the profile, corresponding to 20%.
- In the 0.5 group, 15 of 30 met the profile, corresponding to 50%.
- In the 0.8 group, 30 of 50 met the profile, corresponding to 60%, not 80%.
Result: In this illustrative sample, the lower two score groups are calibrated, while the tool is overconfident in the 0.8 group. It may still rank stronger applications above weaker ones, but staff cannot interpret 0.8 as an 80% likelihood without recalibration or further validation.
In a United Nations workplace
At a duty station with three working languages, a reviser in the translation service checks a pilot tool that scores how likely each application is to meet a rostered translator profile. The long list and screening sheet are due to the selection panel at close of business.
Calibration shows whether applications given, for example, a 0.8 score are later confirmed as meeting the profile about 80% of the time. The reviser should also check calibration by working language, since acceptable overall results can conceal uneven reliability between groups.
Before you try thisApplications contain personal and HR-sensitive material. They must not be entered into any tool not approved for that material. Minimise the data used, restrict access and retain human review. A probability score must not become an automatic ground for excluding an applicant.
Where you may hear it
- A translation service review before a roster-screening pilot is cleared
- A vendor demo during a procurement meeting
- An evaluation office discussion of a reliability diagram
- An ICT governance board review of whether probability scores are fit for use
Avoid confusion
Avoid confusion
Often confused with
Accuracy
The share of cases a system labels correctly out of all the cases it was tested on.
Precision
The share of flagged results that are actually correct positives.
Recall
The share of real positive cases that a system successfully flags.
False positives and false negatives
False positives are wrong yes predictions, while false negatives are wrong no predictions.
Probability (in model outputs)
A number attached to a model’s possible outputs that shows how much of its total estimated likelihood the model assigns to each option.
Common misconceptions
Common misconception: If a model is accurate, its confidence scores must already be trustworthy.
Correction: Accuracy counts how many predictions are right overall. Calibration checks whether the stated confidence matches reality. A model can be accurate because it usually picks the right label, yet still give 95% confidence in cases where it is right only 70% of the time.
Common misconception: A 70% score tells you what will happen in this single case.
Correction: Calibration is about long-run frequency across many similar cases, not certainty about one case. A well-calibrated 70% score means that among many cases given about 70%, roughly 70% turn out correct.
Common misconception: Calibration only matters for highly technical users.
Correction: Anyone using thresholds, triage rules or risk bands is relying on confidence scores. If the scores are miscalibrated, operational choices can be distorted even when the interface looks simple.
Quick check
Which statement best describes a well-calibrated classifier?
Calibration compares probability or confidence scores with observed frequencies. Ranking cases well, using equal-sized groups, or producing high confidence does not show that the score values match outcomes.
Go deeper
Go deeper
The simple version
A system is well calibrated when its probability scores match what happens over time. If it assigns 80% to an outcome in many cases, that outcome should occur about 80% of the time.
The technical version
For probabilistic predictions, calibration means agreement between predicted probabilities and empirical outcome frequencies. In binary classification, if we collect instances assigned score p, the event should occur about p of the time within that group. It is often examined with reliability diagrams, calibration curves, or summary measures such as calibration error, though the exact metric and binning method vary. Calibration differs from discrimination or ranking quality: a model can order cases well yet be systematically overconfident or underconfident. It can also be improved after training by recalibration methods, but recalibration itself must be validated on appropriate data to avoid leakage or overfitting.
Reviewed external resources
Primary sourceAI Risk Management Framework (opens in a new tab)NIST
The NIST framework helps readers place calibration within wider AI risk management and governance, and suits technical and policy-oriented audiences.
Intermediatelink checked 2026-07-31
EncyclopaediaCalibration (statistics) (opens in a new tab)Wikipedia
Wikipedia gives a broad overview of calibration in statistics and suits readers who want a general starting point.
Intermediatelink checked 2026-07-31
CourseClassification: Accuracy, recall, precision (opens in a new tab)Google
This course resource helps readers compare calibration-adjacent ideas such as accuracy, recall and precision, and suits beginners building evaluation vocabulary.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Calibration (opens in a new tab)YouTube search
This YouTube search results page helps readers find multiple video explanations of calibration and suits people who learn best visually; it is a search page, not a single vetted source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Calibration (opens in a new tab)Reddit search
This Reddit search results page helps readers see common questions and informal discussions about calibration and suits readers who want examples of confusion in practice; it is a search page, not a single authoritative source.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Calibration asks whether stated probabilities match observed frequencies. It is not the same as ranking accuracy: a model may order cases well while giving misleading confidence scores.