presentsUNAIVERSE
The UN AI timeline

    Learn AI /AI Security /Model extraction and inversion

    Model extraction and inversion

    Also called: model extraction, model inversion, model stealing, membership inference.

    Detail

    In five seconds

    What it is

    Model extraction tries to copy model behaviour, while model inversion uses model information to reconstruct training-data features or examples.

    In plain words

    A model can give away more than intended. By asking it many questions, someone may copy how it behaves, reconstruct features linked to training data, or estimate whether one person’s record was used to train it.

    An everyday example

    A city contractor has a valuable system for spotting damaged roads from satellite images, but outsiders cannot see the internal checklist. One team submits thousands of sample images and records which ones the system marks as severe, moderate or minor, then builds its own look-alike checker from those answers. Another team studies which image features trigger strong scores and infers that the original system was trained heavily on roads with long dark cracks after rain. A third team takes one satellite image from a disputed insurance case and asks whether the pattern of responses suggests that exact image was part of the training collection.

    Where the story stops working

    A real AI model is not using a simple human checklist, and inversion often recovers statistical patterns rather than a clear original image or record. Also, some attacks need confidence scores or repeated access at scale, while the story makes the process look more direct and certain than it usually is.

    Remember thisModel extraction copies what a model does. Model inversion uses what the model reveals to reconstruct sensitive features or representative inputs. Both can occur through an interface, without direct access to the underlying model.

    This term is contested

    The family resemblance between these terms is widely recognised, but authors do not always group them the same way, and some use model stealing and extraction as exact synonyms while treating inversion and membership inference as separate subclasses or neighbouring privacy attacks.

    Understand it

    Understand it

    Why it matters

    A non-specialist may hear "the model was attacked" and imagine malware or stolen passwords. These attacks are different. They can happen through ordinary-looking use of an AI interface. That matters for procurement, contract terms, rate limits, logging, output design, privacy review and incident response. If a system can be cheaply copied, an organisation may lose intellectual property or control over safety features. If outputs leak training data clues, privacy, confidentiality and trust may be harmed even without a classic network breach. Decision-makers need to ask not only who can log in, but also what the model reveals by design under repeated questioning.

    Three related attacks, three different goals

    After reading the diagram, the reader should be able to say how extraction, inversion and membership inference start from similar access but aim at different outputs.

    1. 01

      Shared starting point

      The attacker has some form of access to a deployed model, such as an API, web tool or application feature, and can observe outputs.

      Place one wide box across the top labelled "Query access to model" with small icons for text, image and score outputs.

    2. 02

      Extraction / stealing

      The attacker collects many input-output examples and trains a substitute that imitates the original model’s behaviour.

      Left column: arrow from the top box to a middle box labelled "Collect many queries" and then to a bottom box labelled "Substitute model".

    3. 03

      Inversion

      The attacker analyses outputs to infer features or partial reconstructions of training data or sensitive attributes associated with classes or prompts.

      Middle column: arrow from the top box to a middle box labelled "Analyse outputs for hidden clues" and then to a bottom box labelled "Recovered data features or reconstructions".

    4. 04

      Membership inference

      The attacker tests whether a particular record appears more likely than not to have been in the training data.

      Right column: arrow from the top box to a middle box labelled "Probe one target record" and then to a bottom box labelled "Likely in training / not in training".

    5. 05

      Overlap and limits

      The attacks are related because all exploit information leakage through outputs, but the evidence and confidence differ. A successful membership inference does not by itself recreate the record, and extraction does not necessarily reveal the original training set.

      Add a footer note spanning all columns with two short contrasts: "same access path, different goals" and "copying behaviour is not the same as recovering data".

    A three-column comparison. Each column starts with the same top box, "Attacker can query a model", then shows a different middle step and a different bottom outcome. Read left to right to compare goals, methods and results. A small note under the columns says these attacks can overlap but are not identical.

    On hover or tap, each column expands to show one example of input, observed output and likely attacker goal.

    Describe this without the diagram

    The diagram compares three attack types that begin with the same condition: a person can query a model and observe outputs. The left path shows extraction, where many queries are collected so a substitute model can imitate the original. The middle path shows inversion, where outputs are analysed to infer features or partial reconstructions of training data. The right path shows membership inference, where one target record is probed to estimate whether it was part of training. The key message is that the access path is similar, but the intended result is different in each case.

    With reduced motion: A static three-column chart with the same top row, different middle actions, and different bottom outcomes, plus a footer note stating that these attacks overlap but are not identical.

    These are related but different attacks against AI systems. Model extraction, also called model stealing, aims to copy how a model behaves well enough to build a substitute. Model inversion aims to reconstruct representative inputs or infer sensitive features of target records from model information. Membership inference asks a narrower question: was this specific record likely in the training data? The attacks can start from similar access, but they have different goals.

    A common starting point is query access: an attacker can send prompts, images or other inputs to a model and observe the responses, scores or probabilities. In some settings they may also see confidence scores, embeddings, labels, gradients or other internal signals. The attacker then analyses patterns across many queries or signals. They do not need to "hack" the operating system for this to be harmful; the weakness can come from what the model reveals through normal use.

    What happens depends on the attack. In extraction, the attacker gathers many input-output pairs and trains another model to imitate the original. In inversion, the attacker uses outputs or exposed internal signals to reconstruct representative inputs or estimate sensitive features of target records. In membership inference, the attacker tests whether a target record produces signals that differ from records the model probably did not train on, often because overfit models can produce different signals for training records.

    What comes out also differs. Extraction produces a copy or approximation of the model’s capability. Inversion produces clues, reconstructions or sensitive attributes about training data, though not always exact originals. Membership inference produces a probability or judgement about whether a particular record was used in training. These are often discussed together because they can exploit information leakage from model behaviour, but they should not be treated as one undifferentiated concept.

    How “Reverse-engineering a road-damage inspector’s method” maps onto the real thing

    In the storyIn the AI system
    Submitting many sample road images and copying the labelsModel extraction or model stealing through repeated queries
    Inferring what kinds of cracks or conditions shaped the systemModel inversion to recover information about training data patterns
    Testing one disputed image for tell-tale familiarityMembership inference on a specific record
    Only seeing scores and labels, not the hidden checklistBlack-box access to model outputs rather than source code

    A worked example

    During an authorised assessment, a vendor provides a test interface for an application-screening model. It returns a suitability ranking and confidence score for synthetic candidate profiles.

    Starting from: The assessors use only generated profiles, varying qualifications, experience and wording across many controlled queries.

    1. They record each synthetic profile and the resulting ranking and confidence score, creating input-output pairs without accessing the original model.
    2. They train a substitute model on those pairs. If it reproduces the target’s rankings on new synthetic profiles, this demonstrates model extraction.
    3. Separately, they optimise generated profile features to produce very high confidence for a target ranking. With richer outputs or exposed embeddings, this may reveal representative feature combinations associated with that ranking.
    4. They compare the reconstructed patterns with authorised documentation and controlled test data. They do not treat them as proof that any particular person or application appeared in training.

    Result: The assessment finds that high-volume queries could support a useful behavioural copy and that detailed scores reveal more about represented profile features than necessary. It recommends rate limits, reduced output detail, monitoring and stricter access controls.

    In a United Nations workplace

    An ombudsman case officer in a country office is reviewing a complaint about an automated screening tool used against a rostered profile. The file raises a separate security question: could repeated test applications and returned scores let a user imitate the tool’s ranking behaviour or infer features represented in earlier application data?

    Access to the interface is not the same as access to the model, but enough queries, scores or embeddings may still support extraction or inversion. The case officer needs the technical review to distinguish a copied decision function from an attempted reconstruction of training-data features, since the controls and possible harms differ.

    Before you try thisApplications and case files contain personal and HR-sensitive information. Do not submit real applications, case details or inferred applicant attributes to any tool not approved for that material. Extraction or inversion testing must be explicitly authorised, isolated from live records, conducted with synthetic or properly controlled test data, and logged. Suspected exposure should be handled under the applicable security and data protection procedures.

    Where you may hear it

    • A vendor demo during a procurement unit’s technical review
    • An ICT governance board considering access controls for a model interface
    • An internal audit interview on logging and high-volume queries
    • A data protection review before an automated tool is cleared for use
    • An authorised red-team debrief with the system owner

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • Model stealing, inversion and membership inference are just different names for exactly the same attack.

      They are related but distinct. Extraction or stealing aims to copy model behaviour. Inversion aims to infer something about the data the model learned from. Membership inference asks whether one specific record was likely in training. One attack campaign can combine them, but the goals and evidence are different.

    • If the model’s source code and weights are hidden, these attacks are not possible.

      Black-box access can be enough. Repeated queries, labels, rankings, confidence scores or other outputs may reveal patterns. Hiding internals helps, but it does not remove leakage through the interface itself.

    • A successful inversion attack means the attacker recovered the exact original training record.

      Often the result is weaker: a reconstruction, a likely feature pattern, or a sensitive attribute estimate. Exact recovery may happen in some cases, but many published attacks recover partial or statistical information rather than a perfect copy.

    • This is the same as data exfiltration.

      Data exfiltration usually means data is directly removed or transferred from a system. Here, the attacker may infer information from model behaviour without directly downloading the dataset. The impact can still involve privacy loss, but the mechanism is different.

    Quick check

    Which statement best distinguishes these related attacks?

    Go deeper

    Go deeper

    The simple version

    A model can give away more than intended. By asking it many questions, someone may copy how it behaves, reconstruct features linked to training data, or estimate whether one person’s record was used to train it.

    The technical version

    These are attacks on model behaviour under black-box or partial white-box access. In model extraction, an attacker queries a target model to collect input-output pairs, then optimises a substitute model to approximate the target decision function, ranking behaviour or output distribution. In model inversion, the attacker uses outputs, gradients, embeddings or confidence signals to infer sensitive features of target records or reconstruct representative inputs associated with classes, prompts or identities. In membership inference, the attacker performs a hypothesis test on a target record, exploiting differences in loss, confidence, entropy or other signals between records inside and outside the training set. The terms overlap in practice because overfitting, rich outputs and repeated access can enable more than one attack, but they should be separated analytically: copying a function, recovering training-data information and testing record inclusion are not the same objective.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Model extraction copies what a model does. Model inversion uses what the model reveals to reconstruct sensitive features or representative inputs. Both can occur through an interface, without direct access to the underlying model.