presentsUNAIVERSE
The UN AI timeline

    Learn AI /How Machines Learn /Validation and test sets

    Validation and test sets

    Also called: validation dataset, test dataset, held-out set, test set.

    Detail

    In five seconds

    What it is

    Separate portions of data are kept aside to tune a model during development and then to check, at the end, how well it works on unseen examples.

    In plain words

    Use one pile of examples to teach the model, a second pile to help choose the best version, and a final untouched pile to check how that chosen version performs on new cases.

    An everyday example

    A team in Rome is designing a route-planning guide for visitors who need to cross the city by bus and on foot. They practise on many known journeys, then check a separate list of trial journeys to compare two draft guides and choose the better one. Once they settle on a guide, they open a sealed envelope with a final set of journeys they have not used before and score the guide only once. If they keep reopening that envelope and rewriting the guide each time, the final score stops being a fair check.

    Where the story stops working

    A route guide is changed by people on purpose, while a machine-learning model is adjusted through mathematical optimisation during training and through chosen settings during tuning. Also, city journeys are easy to picture as separate cases, but in real datasets the examples may be more complex, may need careful matching across time or places, and may fail if the data split is not representative.

    Remember thisUse validation data to make development choices. Reserve the test data for the final check, after the model and evaluation procedure have been fixed.

    Understand it

    Understand it

    Why it matters

    If you read an AI performance claim, this term helps you ask one of the most important questions: was the system measured on data it had already influenced, or on genuinely held-out data? That affects whether a reported score is likely to hold up in practice. For programme teams, procurement staff and managers, the distinction matters when comparing vendors, approving pilots, setting risk controls and deciding whether a model is ready for deployment. A strong validation score can help choose among options, but only a properly protected test set gives a more trustworthy final check on expected behaviour with new cases from the same task.

    How training, validation and test sets play different roles

    Show why one held-out split helps guide development while another should be saved for the final, more independent check.

    1. 01

      1. Split the data

      Start with one dataset and divide it into training data and held-out data. If possible, make each split representative of the real task.

      Draw one large dataset box that branches into three smaller boxes: Training, Validation, Test. Use muted colours and small note text under Validation and Test saying 'held out from training'.

    2. 02

      2. Train on the training set

      Use only the training examples to adjust the model’s internal parameters.

      Draw an arrow from the Training box into a model box labelled 'model learns patterns from examples'. Show gears or sliders inside the model to suggest changing internal settings.

    3. 03

      3. Check on the validation set

      Run the trained model on validation data, measure performance, and use those results to choose settings or compare model versions.

      Draw the Validation box feeding into the model, then to a score panel. Add a looped arrow from the score panel back to a 'tune choices' note beside the model.

    4. 04

      4. Freeze choices and test once

      After tuning is finished, evaluate the chosen model on the test set for a more independent estimate of performance on new data.

      Draw the Test box feeding into the final model and then to a separate final score panel. Do not draw a loop back to the model. Add a lock icon over the model choices.

    A left-to-right step diagram with three labelled data boxes and three arrows. Read from training to validation to test. The diagram shows that training changes the model, validation helps choose settings, and test is used last for final evaluation only.

    Hover or tap each step to highlight which data split is active and whether feedback is allowed to flow back into model development.

    Describe this without the diagram

    The diagram explains a four-step process. Step 1: a dataset is split into three parts called Training, Validation and Test, with Validation and Test marked as held out from training. Step 2: only Training feeds into the model to adjust its internal parameters. Step 3: Validation is used to score the model during development, and those scores can influence tuning decisions, shown by a feedback loop. Step 4: after choices are fixed, Test is used once to produce a final score, with no feedback loop to the model. The main message is that validation guides development, while test is saved for the final independent check.

    With reduced motion: A static four-panel diagram shows the same process without highlights or animation: split data, train on Training, tune using Validation, then evaluate once on Test with no return arrow.

    When people train a machine-learning model, they usually do not use all their data for the same purpose. One part is used for training: adjusting the model’s internal settings from examples. A second part, often called the validation set, is used during development to compare versions, tune choices such as model size, and watch for overfitting. A third part, the test set, is saved for the final check.

    What goes in are examples from the same general task, such as past messages and their correct categories, or house features and sale prices. What happens is that the model learns from the training set, then developers measure its performance on the validation set and may change settings or even the modelling approach. After those choices are finished, they run the final model once on the test set to estimate how it may perform on new, similar data.

    The key difference is timing and use. The validation set helps guide development, so repeated checking can slowly influence decisions. The test set should stay untouched until the end, because it is meant to be an independent check. If people keep looking at test results and changing the model, the test set starts acting like a validation set and no longer gives a clean final estimate.

    You may also hear held-out set. That means data deliberately kept out of training. Sometimes people use it loosely for either validation or test data, so ask which role it serves. This term is often confused with cross-validation, which is a method for rotating through several temporary validation splits, and with data leakage, which is the mistake of allowing information to cross data boundaries in a way that makes measured performance misleading.

    How “Planning a journey across a city” maps onto the real thing

    In the storyIn the AI system
    known journeys used for practicetraining set used to fit the model
    separate trial journeys used to compare draft guidesvalidation set used to tune choices during development
    sealed envelope of final journeystest set kept untouched for final evaluation
    rewriting the guide after seeing envelope resultsusing test feedback to tune the model, which weakens the independence of the test estimate

    A worked example

    After a pilot, a partnerships officer in a liaison office is finalising a lessons-learned note before front-office review. The pilot classifies publicly available partner announcements by theme for an internal horizon-scanning digest.

    Starting from: A labelled archive of public announcements, divided once into disjoint training, validation and test subsets before development begins.

    1. The pilot team fits several candidate models using only the training subset.
    2. It compares candidates on the validation subset, adjusts settings and classification thresholds, and selects the strongest pipeline.
    3. After selection, it freezes the pipeline rather than making further changes based on evaluation results.
    4. It then runs the frozen pipeline once on the untouched test subset and records that result in the lessons-learned note.

    Result: The validation results supported development choices. The test result provided the final independent estimate on unseen announcements. If the pipeline were revised after that result, the test subset would have influenced development and a new untouched test subset would be needed.

    In a United Nations workplace

    A procurement officer at Headquarters is finalising an evaluation note for the procurement unit before close of business. One technical proposal reports a combined “validation/test” score, so the officer asks the technical focal point whether model tuning and final testing used disjoint records.

    If the bidder repeatedly adjusted the model after viewing the same records, the reported score is not an independent final check. The distinction helps the procurement unit assess whether performance is likely to hold on genuinely unseen material.

    Before you try thisTechnical proposals, bids and evaluation records are procurement-sensitive. They must not be uploaded to a public or otherwise unapproved AI service. Review them only through authorised systems and share evaluation material strictly within the approved process.

    Where you may hear it

    • A vendor demo during a procurement meeting
    • A lessons-learned session after an analytics pilot
    • An ICT governance board reviewing an evaluation report
    • A technical review before a model is cleared for deployment

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • Validation and test sets are just two names for the same thing.

      They are closely related but serve different roles. Validation data is used during development to compare models or tune settings, so it can influence choices. Test data is meant to stay separate until the end for a more independent final estimate.

    • If the test score is good, the model will work equally well everywhere.

      A test set only estimates performance on new data similar to that set. If the real-world data comes from different countries, time periods, languages, devices or user groups, performance may change. The split must match the intended use.

    • More checking on the test set gives a more reliable result.

      Repeatedly checking the test set and changing the model after each result makes the test less independent. Over time, the model can become indirectly tuned to that test set, which can make the final score look better than real deployment performance.

    • Any random split is automatically fine.

      Not always. If examples are duplicated, time-ordered, grouped by person or location, or imbalanced across classes, a simple random split can mislead. The split strategy must fit the task and avoid leakage between training, validation and test data.

    Quick check

    Which statement best describes the difference between a validation set and a test set?

    Go deeper

    Go deeper

    The simple version

    Use one pile of examples to teach the model, a second pile to help choose the best version, and a final untouched pile to check how that chosen version performs on new cases.

    The technical version

    In supervised learning, a dataset is often partitioned into disjoint subsets with different roles. The training set is used to fit model parameters. The validation set is a held-out subset used during model development for hyperparameter tuning, model selection, threshold choice, early stopping or comparing alternative pipelines. Because these decisions are influenced by validation performance, the validation set is not a fully independent measure once tuning has occurred. The test set is another held-out subset reserved for final evaluation after the modelling choices are fixed. Its purpose is to estimate generalisation to unseen samples drawn from the same target distribution. 'Held-out set' is a broader label for any data excluded from training, so it may refer to validation, test, or both depending on context. In some workflows, cross-validation replaces a single validation split during development, but a separate final test set may still be kept. Good practice also requires preventing data leakage, such as duplicate records, future information, or grouped observations crossing between splits.

    Reviewed external resources

    • EncyclopaediaTraining, validation, and test data setsWikipedia

      This link gives a broad reference-style overview of training, validation and test splits, suitable for readers who want definitions and common terminology.

      Intermediatelink checked 2026-07-31

    • CourseMachine Learning Crash CourseGoogle

      This course link suits learners who want a practical introduction to machine-learning workflows, including where validation and test data fit.

      Intermediatelink checked 2026-07-31

    • VideoStatQuestStatQuest with Josh Starmer

      This video source suits people who learn well from clear spoken explanations and step-by-step intuition about model evaluation.

      Starterlink checked 2026-07-31

    • VideoVideo explanations: Validation and test setsYouTube search

      This search-results page helps readers find multiple video explanations with different teaching styles, which suits someone comparing presenters rather than seeking one fixed source.

      search results, not one pagelink checked 2026-07-31

    • DiscussionCommunity discussion: Validation and test setsReddit search

      This search-results page lets readers browse community questions and practical confusion points about validation and test sets, which suits readers who want informal discussion and examples.

      search results, not one pagelink checked 2026-07-31

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Use validation data to make development choices. Reserve the test data for the final check, after the model and evaluation procedure have been fixed.