Learn AI /Responsible AI, Human Rights and Safety /Interpretability

Interpretability

Deeper
Responsible AI, Human Rights and Safety
Technical readers
Policy officers

Also called: mechanistic interpretability.

Detail

In five seconds

What it is

Interpretability is the extent to which people can understand how a model works or produces outputs, sometimes by inspecting its internal parts.

In plain words

It is a way of opening the black box a little to see what parts of an AI system seem to drive its answers.

An everyday example

A team receives 1,000 applications and uses a software tool to sort them by completeness. At first, staff only see the final labels: complete, missing document, or unclear. Then an auditor opens the tool’s rule panels and event logs to see what happened inside. They discover that one rule gives too much weight to file naming style, so applications from different organisations are flagged as incomplete more often even when the right documents are present.

Where the story stops working

This analogy uses visible rules and logs, while many modern AI models do not contain neat human-written rules. Their internal structure is distributed across many numerical parameters, so the mechanism is much harder to inspect. Also, interpretability often produces partial evidence rather than a clear fault report.

Remember thisInterpretability seeks evidence about the internal features and pathways producing a model's output. That evidence can reveal shortcuts or failure modes, but it is rarely a complete account and is not proof of safety.

This term is contested

The broad idea is widely used, but the boundaries of 'interpretability', especially compared with explainability and transparency, are not fully settled. 'Mechanistic interpretability' is also used more narrowly by some researchers to mean analysis of internal circuits and representations, rather than interpretability in the broadest sense.

Understand it

Why it matters

For non-specialists, interpretability matters because an AI system can appear accurate while relying on the wrong reasons. A model may pick up shortcuts, hidden bias or brittle patterns that only become visible when conditions change. If a system affects access, safety, rights or important work, decision-makers need more than a polished answer. They need evidence about whether the system’s behaviour is trustworthy enough for the task, what risks remain, and where human review must stay in place. Interpretability does not solve these problems on its own, but it can make risk assessment, red-teaming, accountability and safer system design more concrete.

From input to output, with an extra layer of inspection inside

After viewing this, the reader should be able to say that interpretability examines internal model behaviour, not only the final answer.

01
1. Input enters
A text, image or other data item is given to the model.
Draw an input card on the left with example text such as 'application form + attachments' feeding into a large model box.
02
2. Internal processing
The model transforms the input through many internal components. These components may respond to different patterns.
Inside the model box, draw three to five smaller nodes connected in layers. Label them 'internal components' rather than using unexplained jargon.
03
3. Output appears
The model produces a label, score or generated text. By itself, this only shows the end result.
Draw an output card on the right labelled 'complete / incomplete / unclear' connected from the model box.
04
4. Inspection probes
Interpretability methods sample activations, pathways or feature patterns inside the model while it processes the input.
Add a second lane below the model with small probe icons attached to internal nodes, sending lines to a report panel.
05
5. Evidence report
The analysis suggests which internal parts mattered and whether the model may be using a sensible signal or a misleading shortcut.
Draw a report panel with two lists: 'likely relied on required document count' and 'possible shortcut: file naming pattern'.

Show a left-to-right flow with one main lane and one inspection lane. Main lane: input enters the model, passes through internal components, and produces an output. Inspection lane: probes attach to internal components, collect signals, and produce an interpretability report. Read from left to right, then compare output-only information with inside-the-model evidence.

Hovering or tapping an internal node highlights its probe connection and the matching line in the evidence report.

Describe this without the diagram

The diagram is a left-to-right flow. First, an input such as an application form enters a model. Inside the model are several connected internal components. The model then produces an output, for example a completeness label. A second inspection lane attaches probes to the internal components and sends their signals to a report. The report identifies possible internal reasons for the output, such as reliance on a relevant feature or on a misleading shortcut. The point is that interpretability adds evidence about what happened inside the model, not only a description of the final result.

With reduced motion: A static flow diagram shows the same five stages without highlighting. Lines from internal components to the report remain visible at all times, so the reader can compare output-only information with internal inspection evidence.

Many AI systems take an input, transform it through internal steps, and produce an output such as a score, label or generated text. Interpretability concerns how far people can understand this operation and the relationship between inputs and outputs. That understanding may come from readable features or decision paths, or from methods that inspect an otherwise opaque model.

This overlaps with explainability but is not identical. Explainability focuses on showing a human-usable reason for a particular output or pattern of outputs. Interpretability concerns how understandable the model’s operation is. Internal inspection is one form, especially mechanistic interpretability, but it is not the whole term.

Researchers may inspect features, neurons, attention patterns, activations or decision paths. A neuron is a small computational unit inside some models. An activation shows how strongly that unit responds to an input. Units that activate together may reveal useful signals, irrelevant shortcuts or unsafe patterns, but association alone does not prove what caused an output. Researchers can test such findings with interventions and multiple examples.

The result is evidence, not certainty. It may include readable decision paths, estimates of feature effects, or possible maps of circuits and behaviours inside a model. This can help with safety testing, bias investigation and human oversight, but it does not guarantee full transparency or control.

How “Checking how application sorting actually works” maps onto the real thing

In the story	In the AI system
final labels on applications	model outputs
rule panels and event logs	internal signals or mechanisms being inspected
file naming style affecting results	spurious feature or shortcut the model may rely on
auditor tracing the problem	interpretability analysis

A worked example

Before close of business, an information management officer in a country office tests a classifier that routes implementing-partner progress reports to either ready for review or return for missing annexes. The test uses a synthetic report rather than a live submission.

Starting from: The synthetic report contains every required attachment, but the attachment section is headed “Supporting schedules” rather than “Annexes”.

The classifier routes the report for return, stating that annexes appear to be missing.
Inspection of internal activations identifies a feature that responds strongly to the exact heading “Annexes”, while the representation of attachment presence has little influence on the output.
The officer changes only the heading, and the classification changes. A causal intervention that suppresses the heading-related feature also reduces the missing-annex score, providing evidence that the feature contributes to the decision.
The classifier is revised to use attachment-presence metadata and is tested against varied headings and document layouts before being placed on the file for clearance.

Result: The investigation identifies a shortcut in the model's internal mechanism, not merely a suspicious output. It does not prove that every relevant pathway has been found, so the office retains human review and continues testing other document formats.

In a United Nations workplace

At a regional hub where drafts arrive in three working languages, an ICT officer is checking a model that tags paragraphs in the situation report before the regional bulletin closes. Internal inspection tests whether the model responds to substantive operational terms across languages, or to shortcuts such as familiar headings, paragraph position or one language's punctuation.

The officer can compare internal activations and intervene on suspected features to see whether they actually affect the tags. This provides stronger evidence than observing outputs alone, although even a convincing trace covers only the behaviour tested and does not establish that the model is safe or reliable overall.

Before you try thisDraft situation reports may contain personal, protection-sensitive or security-sensitive information. They must not be copied into a public or otherwise unapproved tool, including for debugging. Use only an authorised environment, minimise the material used, apply access controls and retain human review before anything is circulated.

Where you may hear it

During an ICT governance board review before clearance
At a vendor demo when procurement asks what evidence supports the model's explanations
In an internal audit opening meeting on automated decision support
At an inter-agency working group reviewing model risks before rollout

Avoid confusion

Often confused with

Side-by-side

Transparency vs explainability
Two words that get used interchangeably in negotiated text.

Common misconceptions

Common misconception: If a model is interpretable, it is automatically safe and fair.
Correction: Interpretability can reveal useful clues about internal behaviour, but it does not prove the system is safe, fair or rights-respecting. A model can be partly interpretable and still produce harmful outputs, and some harmful patterns may remain hidden or be misread. It should support, not replace, testing, impact assessment and human oversight.
Common misconception: Interpretability and explainability mean the same thing.
Correction: They overlap, but they are not identical. Explainability often aims to give people an understandable account of an output. Interpretability tries to inspect the internal mechanism more directly. An explanation can be simple and useful without faithfully describing what happened inside, while interpretability seeks evidence about the inside of the model.
Common misconception: Looking at a few neurons tells you exactly what the model is doing.
Correction: Modern models often represent information across many interacting components, not one neat unit per concept. A neuron or attention pattern may correlate with a feature, but that does not always mean it alone causes the behaviour. Good interpretability work usually combines several methods and checks whether findings hold across many examples.

Quick check

Which statement best distinguishes interpretability from explainability?

Go deeper

The simple version

It is a way of opening the black box a little to see what parts of an AI system seem to drive its answers.

The technical version

Interpretability studies the relationship between a model’s internal representations and its outputs. In mechanistic interpretability, the aim is to identify circuits, features, activations or pathways that appear to implement parts of the model’s behaviour. Methods may inspect neurons, attention heads, embeddings, saliency patterns or causal interventions on internal states. The core question is whether we can build evidence-based accounts of the model’s internal mechanism, rather than only observing input-output behaviour. In large models, this is difficult because representations are distributed, context-dependent and sometimes only partly recoverable. Findings are therefore often provisional and should be validated against multiple prompts, datasets and interventions.

Reviewed external resources

Primary sourceRecommendation on the Ethics of Artificial Intelligence (opens in a new tab)UNESCO
This link gives a policy-oriented international framework for AI ethics, and suits readers who want to place interpretability within wider governance, risk and human rights responsibilities.
Intermediatelink checked 2026-07-31
Primary sourceB-Tech Project (opens in a new tab)UN Human Rights Office (OHCHR)
This link gives business and human rights guidance relevant to accountability and due diligence, and suits policy or implementation readers considering how technical opacity can affect rights-holders.
Intermediatelink checked 2026-07-31
EncyclopaediaMechanistic interpretability (opens in a new tab)Wikipedia
This link gives a broad overview of mechanistic interpretability concepts and terminology, and suits readers who want a quick starting point before deeper technical reading.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Interpretability (opens in a new tab)YouTube search
This search page gives a range of video explanations with different levels of depth, and suits readers who learn better from spoken and visual teaching than from text alone.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Interpretability (opens in a new tab)Reddit search
This search page gives a range of community discussions and differing viewpoints, and suits readers who want informal debate, practical questions and examples but should read critically because search results vary over time.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Interpretability seeks evidence about the internal features and pathways producing a model's output. That evidence can reveal shortcuts or failure modes, but it is rarely a complete account and is not proof of safety.