presentsUNAIVERSE
The UN AI timeline

    Learn AI /Data and Infrastructure /Distribution shift and data drift

    Distribution shift and data drift

    Also called: data drift, model drift, concept drift.

    Detail

    In five seconds

    What it is

    These terms describe mismatches between development and live data, including changed inputs, labels or relationships, which can reduce a system's reliability.

    In plain words

    A system learned from yesterday's patterns. If today's data looks different, or the old clues no longer mean the same thing, the system's outputs can get worse.

    An everyday example

    A café predicts whether the next customer will order a hot or cold drink using the hour and weather. A new bus route changes who arrives at each hour, so the inputs no longer resemble the earlier records. Later, customers begin choosing iced drinks at temperatures where they previously chose tea, so the same clues now point to a different correct answer. A heatwave may also make cold drinks much more common overall. The prediction rule has not changed, but the data around it, and possibly what those data mean, have.

    Where the story stops working

    An examination is designed by people and changes at set times, while real-world drift may be gradual, accidental or caused by many systems interacting. Also, a student can consciously change study strategy, but a deployed model does not adapt safely on its own unless people monitor it and update it.

    Remember thisA model can become unreliable while its code stays unchanged because the inputs, label balance or relationship between inputs and correct outputs has shifted.

    This term is contested

    The labels are not used uniformly across organisations and textbooks, especially 'model drift'. Some authors also separate additional subtypes of shift more finely than this overview does.

    Understand it

    Understand it

    Why it matters

    A system can pass testing and still become less reliable later because the world, users or data pipeline changed. For programme teams, that can mean missed cases, unfair treatment or wasted staff time. For technical teams, it means monitoring cannot stop at launch. For leadership, it affects risk, budget and accountability: a model may need more review, updated data, or retirement even if nothing appears "broken" in the software. These terms help teams ask the right question: did the inputs change, did the meaning change, or are we simply measuring the wrong thing? That difference matters because the response is different too.

    How shift in live data leads to weaker results

    After viewing this diagram, a reader should be able to distinguish input changes from meaning changes, and explain how both can reduce reliability after deployment.

    1. 01

      1. Build and check

      A model is developed using historical data and evaluated on data that is assumed to represent expected use.

      Draw a box labelled 'Past data used for development' feeding into a box labelled 'Model checked on expected conditions'. Use calm colours and a note: 'baseline performance'.

    2. 02

      2. Live data arrives

      After deployment, the system receives new inputs from the real setting, such as new documents, images, sensors or user behaviour.

      Add a third box to the right labelled 'Current live data'. Connect from the checked model box with an arrow labelled 'deployment'.

    3. 03

      3. Two things may change

      Sometimes the inputs themselves change in pattern or quality. Sometimes the meaning of the inputs changes, so the old clues no longer point to the same correct answer.

      Under 'Current live data', add two stacked side notes: 'Data drift: input mix, quality, source or frequency changes' and 'Concept drift: input-to-correct-output relationship changes'. Use different border styles, not just colour.

    4. 04

      4. Effects and response

      If these changes are large enough, performance, fairness, stability or cost can worsen. Teams respond by monitoring, investigating, updating data pipelines, recalibrating or retraining.

      Add a final box labelled 'Possible outcomes: lower accuracy, more errors, bias, extra review cost'. Beneath it add small action chips: 'monitor', 'compare to baseline', 'investigate cause', 'update or retrain'.

    A left-to-right flow with four boxes and two branch notes. Read from development data to live data, then to two kinds of change, then to outcomes. One branch note marks input pattern change; the other marks changed input-to-label relationship. Final box lists monitoring and response actions.

    Hover or tap each note to reveal a short example: night images for data drift, and changed word usage for concept drift.

    Describe this without the diagram

    This diagram explains a sequence from left to right. First, a model is built and checked using past data under expected conditions. Next, the model is deployed and starts receiving current live data. Then the diagram separates two reasons the live setting may no longer match the old setting. One is data drift, where the input pattern changes, such as different document formats, image quality or class balance. The other is concept drift, where the link between an input and the correct output changes, so old clues no longer predict the same answer. Finally, the diagram shows outcomes: lower accuracy, more errors, possible fairness problems and extra human review costs. It ends with response actions: monitor, compare with the original baseline, investigate the cause, and update or retrain if needed.

    With reduced motion: Show the same four boxes and two side notes as a static left-to-right diagram with all example labels visible from the start, without hover effects or animation.

    AI systems are built and checked on past data, then used on new data. Problems start when the new data no longer looks like the old data in an important way. That broad situation is often called distribution shift: the pattern of data in use differs from the pattern seen during development.

    Within that broad idea, data drift usually means the inputs have changed. For example, a model trained on clear daytime photos may later receive more night images or lower-quality scans. Concept drift usually means the relationship between input and correct output has changed. For example, words that once signalled urgent feedback may later be used casually, so the same phrase no longer means the same thing.

    Model drift is used less consistently. Some people use it loosely for any drop in model performance over time. Others use it for the model becoming less suitable because of data drift, concept drift, software changes or feedback loops. Because usage varies, it is safer to ask what exactly has changed: the inputs, the target labels, the environment, or the measured performance.

    What goes in is live data from the real setting. What happens is a mismatch between the old pattern and the current one. What comes out can be lower accuracy, more bias, unstable results, false confidence or rising costs from extra checking and retraining. This is different from overfitting, which starts during development when a model matches training data too closely; drift happens later when the world or data pipeline changes.

    How “Predicting the next café order” maps onto the real thing

    In the storyIn the AI system
    last year's practice paperstraining and test data collected earlier
    the real exam this yearlive data after deployment
    more long case-study questionsdata drift in the input pattern
    topics framed differently so old clues no longer predict the right answerconcept drift in the relationship between input and correct output

    A worked example

    At a regional hub, an information management officer checks a classifier that tags cleared reporting extracts by sector before a monitoring report is uploaded to the portal.

    Starting from: The development set contains older narrative paragraphs. Current submissions increasingly use short bullet points, machine-translated text and revised templates, while reviewers apply the latest labelling instructions.

    1. The classifier still performs well on held-out extracts drawn from the same period as its development data.
    2. Current extracts differ in length, vocabulary and structure. This is input drift and may amount to covariate shift if the underlying feature-to-label relationship remains unchanged.
    3. Reviewers also find that some mixed activities are labelled differently under the current instructions. The correct label for the same wording has therefore changed, indicating concept drift.
    4. The officer compares recent human-coded extracts with model outputs and tracks changes in input patterns, label prevalence and agreement with current labels.
    5. Recent approved examples are added for retraining, and the revised classifier is checked against current human coding before it returns to use.

    Result: Monitoring the text alone can reveal changed inputs, but it cannot establish whether the correct labels have changed. Detecting concept drift requires recent, consistently labelled examples and renewed performance checks.

    In a United Nations workplace

    As terms of reference for a premises-alert pilot move towards Friday clearance, a security analyst supporting a UN Country Team working group checks whether the vendor must monitor changes in historical alert patterns, sensor coverage and reporting practice after deployment.

    A different mix of sensors or reporting forms would be input drift. If an alert pattern that once indicated a routine fault begins to signal an incident requiring follow-up, that would be concept drift. A change in the proportion of genuine incidents would shift label prevalence. Each can make the pilot less reliable without any change to its code.

    Before you try thisDo not place security incident logs, access records or identifiable staff data in any tool not approved for that material. Removing names does not by itself make security patterns safe to disclose. Any pilot must retain human review of alerts and define what happens when drift is detected.

    Where you may hear it

    • An ICT governance board considering monitoring requirements for a pilot
    • A vendor performance review before a system is renewed
    • A data quality meeting after live results diverge from validation results
    • An internal audit interview on controls for a deployed model

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • If a model scored well during testing, drift is not a serious concern.

      Good testing only shows how the model performed on the data used at that time. After deployment, user behaviour, sensors, document formats, language use or external conditions may change. A model can therefore move from reliable to unreliable without any code bug.

    • Data drift, concept drift and model drift all mean exactly the same thing.

      They are related, but not identical. Distribution shift is the broad umbrella. Data drift usually points to changed inputs. Concept drift usually points to changed input-to-label relationships. Model drift is used more loosely, often for declining performance over time from one or more causes.

    • Drift only matters for very large or highly technical AI systems.

      Even simple classifiers, search ranking tools and forecasting systems can drift. A small system that routes messages, flags invoices or tags documents can create real operational problems if the live data no longer matches the old data.

    • If drift appears, retraining is always the right fix.

      Not always. First identify the cause. The issue may be a broken data pipeline, missing values, changed measurement practice, new user groups, or a target that has become poorly defined. In some cases the right response is better monitoring, a rules update, recalibration or stopping use for that task.

    Quick check

    Which situation best illustrates concept drift rather than simple input data drift?

    Go deeper

    Go deeper

    The simple version

    A system learned from yesterday's patterns. If today's data looks different, or the old clues no longer mean the same thing, the system's outputs can get worse.

    The technical version

    Distribution shift is the umbrella term for a mismatch between the probability patterns in development data and those in deployment data. Covariate shift means the feature distribution changes while the feature-to-target relationship stays the same; input drift is a looser term for changed features. Another is concept drift, where the relationship between features and the target label changes over time, so the same input pattern maps to a different correct output. Label prevalence can also shift, and operational changes such as new sensors, preprocessing, user interfaces or feedback loops can alter measured performance. The phrase model drift is used inconsistently: in practice it often refers to observable performance degradation from one or more of these mechanisms. Detecting drift may involve comparing feature distributions, monitoring calibration and error rates, and sampling recent cases for human review, but a detected statistical shift does not by itself prove harmful impact.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    A model can become unreliable while its code stays unchanged because the inputs, label balance or relationship between inputs and correct outputs has shifted.