Learn AI /Model Quality and Evaluation /Cross-validation
Cross-validation
- Deeper
- Model Quality and Evaluation
- Technical readers
Also called: k-fold cross-validation.
Back to the milestone you came from
In five seconds
What it is
A method for estimating how well a model will perform on new data by repeatedly training on one part of a dataset and checking results on another part.
In plain words
Instead of testing a model once on one held-out chunk of data, you test it several times on different held-out chunks and combine the results.
An everyday example
A team needs to judge the quality of a short translated field report, but they only have one report broken into five sections. Four bilingual reviewers use four sections to agree on style and terminology, then they test that agreement on the fifth section they did not use. They repeat this five times, each time leaving out a different section. At the end, they look at all five review results together instead of trusting only one section.
A translation review involves human judgement and discussion, while cross-validation usually runs a fixed mathematical training and scoring procedure. Also, the report sections may influence one another in ways that data folds should ideally not. In real machine learning, the folds should be prepared so that repeated or linked examples do not leak information across splits.
Remember thisCross-validation rotates the held-out data and summarises all fold scores, making the estimate less dependent on one split. It does not replace a final independent test set or correct data leakage.
Understand it
Understand it
Why it matters
A single performance score can look reassuring even when it depends heavily on how the data were split. Cross-validation gives a stronger estimate by repeating the train-and-check cycle across multiple splits. For non-specialists, that matters when comparing vendors, approving pilots, or deciding whether a system is ready for real use. It can reveal that one model is consistently solid while another only looks good under favourable conditions. It also helps you ask better questions: What metric was used, how many folds, was the split appropriate, and how much did the score vary? Those questions reduce the chance of buying, deploying or trusting a system on weak evidence.
How k-fold cross-validation works
After viewing this, the reader should be able to explain that the same dataset is split into k folds, each fold is held out once for evaluation, and the final result combines the scores.
01
Split the data
Divide the dataset into k folds of similar size. In classification tasks, folds are often arranged so class proportions stay similar across folds.
Draw a horizontal bar split into five equal rectangles labelled Fold 1 to Fold 5. Add a small note: 'same dataset, partitioned once'.
02
Train on most folds
For the first run, train the model on k-1 folds and leave one fold out for evaluation.
First row of five rectangles: colour Folds 2 to 5 blue for training and Fold 1 orange for evaluation. Add a train icon or arrow from blue blocks to a model box.
03
Rotate the held-out fold
Repeat the process k times, each time changing which fold is held out. Every example appears in evaluation exactly once.
Add four more rows. In each row, move the orange evaluation colour to the next fold and keep the remaining folds blue. Connect each row to a score badge.
04
Collect the scores
Record one evaluation score per run, such as accuracy or F1 score.
Show five score badges, for example 0.78, 0.81, 0.76, 0.80, 0.79, aligned beside the five rows.
05
Summarise performance
Compute the mean score and often the spread across folds. A stable model usually shows similar scores from fold to fold.
Draw arrows from the five score badges into a final summary box labelled 'mean score' and 'variation across folds'.
Show one dataset divided into five equal blocks. Then show five rows. In each row, four blocks are marked training and one block is marked evaluation, with the evaluation block rotating from left to right. A final panel shows five scores feeding into one average score and a variability note.
Hovering or tapping a row highlights which fold is used for evaluation in that run and reveals the corresponding score.
Describe this without the diagram
The diagram explains k-fold cross-validation in five stages. First, one dataset is divided into five folds. Next, run one trains on folds 2 to 5 and evaluates on fold 1. Then the held-out evaluation fold rotates so that run two evaluates fold 2, run three fold 3, and so on until each fold has been evaluated once. Each run produces a score. Finally, the five scores are combined into a mean score, and the variation among them is shown to indicate stability across splits.
With reduced motion: A static five-row chart shows the dataset split into five folds. In each row, one fold is coloured as evaluation and the others as training, with the evaluation fold changing position from row to row. Five scores are listed at the right, leading to a final summary box with the average and variation.
Cross-validation is a way to test a model more carefully when you have a limited dataset. A model is a mathematical system that learns patterns from examples. Instead of checking performance on just one split of the data, you split the data into several parts and repeat the test several times.
In k-fold cross-validation, you divide the dataset into k roughly equal groups called folds. You train the model on k-1 folds and evaluate it on the remaining fold. Then you rotate which fold is held out, so every fold gets a turn as the evaluation set.
What goes in is one dataset, a chosen value of k, a model, and an evaluation metric such as accuracy, precision, or recall. What happens is repeated training and repeated evaluation on different held-out folds. What comes out is usually a set of scores and an average score, often with some measure of spread, which shows how stable the model is across splits.
This helps reduce the risk of being misled by one lucky or unlucky train-test split. But it is not a magic guarantee. If the data are unrepresentative, leaked, duplicated, or grouped in the wrong way, cross-validation can still give over-optimistic results.
How “Checking a translated field report with rotating reviewers” maps onto the real thing
| In the story | In the AI system |
|---|---|
| five sections of the report | five folds of the dataset |
| using four sections to agree on style and terminology | training the model on k-1 folds |
| checking the fifth unseen section | evaluating on the held-out fold |
| combining the five review results | averaging scores across folds |
A worked example
The statistician checks the pilot using 1,000 authorised, labelled passages from progress reports. The model classifies each passage as requiring follow-up or routine, with follow-up cases less common.
Starting from: 1,000 labelled passages, five folds of about 200 passages, one classification model and F1 score as the evaluation metric.
- Group passages by source report, then create five disjoint folds while keeping the follow-up and routine proportions as similar as possible. This reduces leakage from repeated or near-duplicate wording.
- Train on folds 2 to 5 and evaluate on fold 1. Repeat the process until every fold has served once as the held-out evaluation fold.
- Record the five F1 scores: 0.72, 0.75, 0.70, 0.74 and 0.73.
- Calculate the mean score, 0.728, and examine the fold-level results rather than reporting the mean alone.
- Use the same folds and metric when comparing alternative models. If a separate final test set is available, leave it untouched until the preferred model has been selected.
Result: The estimated F1 score is about 0.73 on passages from reports not used for fitting in each iteration. The fairly close fold scores make the estimate less dependent on one favourable split, although only a separate final test set provides the last independent check.
In a United Nations workplace
By close of business, a statistician in a regional bureau must send the chief of section talking points for informals on a pilot that flags implementing partner progress-report passages for follow-up. With only 1,000 labelled passages, the statistician asks for stratified five-fold cross-validation and all five fold-level F1 scores, rather than the single score featured in the vendor slide.
The results show whether the headline score holds across several data splits and whether the less common follow-up cases are represented in each fold. The mean provides a more stable estimate, while variation between folds gives the chief of section a reason to qualify the talking point if performance is uneven.
Before you try thisProgress reports may contain personal, protection, security-sensitive or procurement-sensitive material. They must not be uploaded to an external or unapproved AI service, because this could expose restricted information. Use only authorised data in an approved environment, and keep passages from the same report in one fold where duplication could otherwise leak information and inflate the score.
Where you may hear it
- During informals, when a statistician is asked what sits behind a headline model score
- In a regional bureau clearance meeting on talking points for a data pilot
- At a vendor demo, when the procurement unit asks for fold-level results
- During an evaluation office's first reading of a model validation report
Avoid confusion
Avoid confusion
Often confused with
Validation and test sets
Separate portions of data are kept aside to tune a model during development and then to check, at the end, how well it works on unseen examples.
Accuracy
The share of cases a system labels correctly out of all the cases it was tested on.
Benchmark
A benchmark is a standard test set-up used to compare how well systems perform on a defined task under stated conditions.
Common misconceptions
Common misconception: Cross-validation proves a model will work well in the real world.
Correction: It gives an estimate based on the available dataset, not a guarantee about future conditions. If real-world data differ from the dataset used for evaluation, performance can still drop. It measures generalisation to similar unseen data, not to every possible setting.
Common misconception: Averaging across folds removes the need for a separate test set.
Correction: Cross-validation is often used during model selection and tuning, so repeated use can still lead to optimistic choices. If you have enough data, a final untouched test set remains useful for one last independent check after all choices are fixed.
Common misconception: Any way of splitting the data is fine as long as there are k folds.
Correction: The split strategy matters. For example, if the same person, document, location or time period appears in both training and evaluation folds, leakage can inflate scores. Some tasks need grouped, stratified or time-aware cross-validation rather than ordinary random folds.
Common misconception: A high average cross-validation score means the model is stable.
Correction: The average alone can hide instability. Two models can have the same mean score, but one may vary greatly across folds. Looking at the spread helps show whether performance is consistent or fragile.
Quick check
Why do many teams use k-fold cross-validation instead of relying on one train-test split?
Cross-validation repeats evaluation across multiple held-out folds, so the result is usually less dependent on one arbitrary split. It does not guarantee fairness, does not replace the need for a metric, and does not prevent overfitting by itself.
Go deeper
Go deeper
The simple version
Instead of testing a model once on one held-out chunk of data, you test it several times on different held-out chunks and combine the results.
The technical version
In k-fold cross-validation, a dataset is partitioned into k disjoint folds. For each iteration i, the model is fit on the union of k-1 folds and evaluated on fold i. This produces k out-of-sample performance estimates, one per fold, which are then summarised, commonly by the mean and sometimes the standard deviation or confidence interval. The method reduces sensitivity to a single split and uses data efficiently, especially when datasets are small. Variants include stratified k-fold for preserving class proportions, grouped cross-validation for keeping related samples together, and time-series methods when order matters. Cross-validation is mainly an internal evaluation procedure for model selection and hyperparameter tuning; if it guides repeated choices, an untouched final test set is still valuable for a less biased final estimate.
Reviewed external resources
Primary sourceAI Risk Management Framework (opens in a new tab)NIST
This link is useful for readers who want to place model evaluation inside a wider risk-management process, especially in organisational or public-sector settings.
Intermediatelink checked 2026-07-31
EncyclopaediaCross-validation (opens in a new tab)Wikipedia
This link gives a broad overview of cross-validation methods and terminology, which suits readers who want a general reference before going deeper.
Intermediatelink checked 2026-07-31
CourseClassification: Accuracy, recall, precision (opens in a new tab)Google
This link helps readers connect cross-validation to common classification metrics such as accuracy, precision and recall, and suits learners comparing how results are reported.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Cross-validation (opens in a new tab)YouTube search
This search results page helps readers find video explanations with different teaching styles, and suits people who learn better by seeing the procedure demonstrated step by step.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Cross-validation (opens in a new tab)Reddit search
This search results page shows community questions and practical discussions about cross-validation, and suits readers who want examples of common confusions and troubleshooting topics.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Cross-validation rotates the held-out data and summarises all fold scores, making the estimate less dependent on one split. It does not replace a final independent test set or correct data leakage.