Learn AI /Training and Model Adaptation /Distillation

Distillation

Deeper
Training and Model Adaptation
Technical readers
Policy officers

Also called: knowledge distillation, distilled model.

Detail

In five seconds

What it is

Distillation trains a student model to imitate a teacher model’s outputs or other signals; the student is often smaller.

In plain words

A big model shows a smaller model how to answer many examples. The smaller model learns to produce similar outputs, so it can often do a similar job faster and more cheaply.

An everyday example

At a distribution point, a senior planner has a thick notebook with years of experience and can estimate food needs for many different family sizes, weather conditions and delivery delays. A smaller field team cannot carry the whole notebook or spend as long on each estimate, so they practise by reviewing many situations and copying the senior planner’s estimates and explanations. After enough practice, the field team can give fast estimates on most common cases using a much lighter checklist. The team is useful in the field, but in unusual situations it may still miss details the senior planner would catch.

Where the story stops working

A human team can ask questions, use judgement and deliberately simplify rules. A distilled model does not literally receive understanding or experience from a teacher. It only adjusts numeric parameters to better reproduce patterns in the teacher’s outputs. Also, real distillation may use probability distributions or hidden representations, not just final answers, so the analogy is simpler than the actual mechanism.

Remember thisDistillation trains a student model to match signals from a teacher model. The student is often smaller and faster, but it must be tested for performance lost or errors inherited in the transfer.

Understand it

Why it matters

For non-specialists, distillation matters because it affects what organisations can actually deploy. A system may work well in a lab with a very large model but be too slow, too expensive or too energy-intensive for real use. Distillation can make deployment practical on ordinary servers, phones or low-bandwidth settings. It also matters for procurement and governance: a distilled model may behave differently from its teacher, especially on edge cases, so claims such as 'same quality, just smaller' need testing rather than assumption. In policy and operational settings, this affects cost, access, audit plans, safety review and decisions about when a smaller model is acceptable and when the larger model or human handling is still needed.

How distillation turns a large model into a smaller deployable model

After viewing this, the reader should be able to explain that distillation uses a teacher model’s outputs to train a smaller student model, and why this differs from simply shrinking one model file.

01
1. Prepare inputs
A dataset of prompts, examples or task inputs is collected. It may be labelled or unlabelled, depending on the use case.
Show a stack of input cards on the far left labelled 'same inputs' with arrows splitting to two model boxes.
02
2. Run the teacher
The larger teacher model produces outputs for those inputs. These outputs act as training targets for the student.
Place a large box above labelled 'teacher model (larger, slower)' with arrows to output bubbles such as scores, tokens or probabilities.
03
3. Run the student
The smaller student model processes the same inputs and produces its own outputs.
Place a smaller box below labelled 'student model (smaller, faster)' with arrows to its own output bubbles aligned with the teacher’s outputs.
04
4. Compare outputs
A loss function, meaning a numeric measure of mismatch, compares teacher and student outputs.
Add a comparison box to the right receiving both output streams, with a gauge icon labelled 'difference'.
05
5. Update the student
Training changes the student’s parameters to reduce the mismatch. This loop repeats many times.
Draw a bold arrow from the comparison box back to the student model with a circular loop marker and label 'update student only'.
06
6. Deploy the distilled model
Once the student is good enough, it is saved and used in place of the larger teacher for suitable tasks.
Show a final compact model box on the far right labelled 'distilled model for deployment' with tags 'smaller', 'faster', 'cheaper'.

A left-to-right flow diagram with two parallel paths. Inputs go to a large teacher model and produce target outputs. The same inputs go to a smaller student model. A comparison stage measures the difference between teacher and student outputs, and a training loop updates the student. The final box shows a smaller deployed model. Read from left to right, then follow the loop back to the student until the outputs are close enough.

Hovering or tapping each stage highlights which artefacts are fixed and which change: inputs fixed, teacher fixed, student updated. A toggle can switch labels between 'plain language' and 'technical terms'.

Describe this without the diagram

This diagram explains distillation as a five-part flow with a training loop. First, the same input examples are sent to two models: a large teacher and a smaller student. Second, the teacher produces target outputs such as scores or text. Third, the student produces its own outputs for the same inputs. Fourth, a comparison step measures the difference between the two sets of outputs. Fifth, only the student is updated to reduce that difference, and this repeats many times. The end result is a smaller deployable model that imitates much of the teacher’s behaviour. The key point is that a new smaller model is trained from the teacher’s behaviour, rather than merely compressing the teacher’s file.

With reduced motion: A static left-to-right flow chart shows the teacher path, the student path, a comparison box and a return arrow from comparison to student labelled 'repeat during training'. The final distilled model is shown as the output of the completed loop.

Distillation is a way to make a model smaller, faster or cheaper to run without starting from zero. A large model, often called a teacher model, produces outputs for many inputs. A smaller model, often called a student model, is then trained to match those outputs as closely as possible.

What goes in? Usually three things: a trained teacher model, a set of input examples, and a student model architecture. The examples may be labelled data, unlabelled data, or prompts generated for the task. The teacher produces target outputs, which can be final answers, probabilities across options, or intermediate signals depending on the method.

What happens? The student model adjusts its parameters so its outputs become similar to the teacher’s outputs. This can preserve much of the teacher’s performance while reducing size or latency, meaning how long one response takes. In practice, distillation is often used after pretraining or fine-tuning, not instead of them.

What comes out? A distilled model: usually smaller, faster and cheaper to deploy, but often not as capable as the teacher on difficult cases. Distillation is about transferring behaviour from one model to another. It is commonly confused with quantisation and pruning, which also make models lighter, but those methods compress a model in different ways rather than teaching a new smaller model to imitate an existing one.

How “A senior planner coaching a quick field briefing team” maps onto the real thing

In the story	In the AI system
senior planner with a thick notebook	large teacher model
smaller field team	student model
many practice situations	training inputs used for distillation
copying the senior planner’s estimates	training the student to match teacher outputs

A worked example

At a regional hub, an information management officer needs a lightweight model to tag paragraphs from cleared situation reports before the morning portal upload. The approved teacher model is accurate but cannot meet the processing window on the available hardware.

Starting from: A curated set of cleared report paragraphs. For each paragraph, the teacher supplies a probability distribution over the office’s existing tags. A separate held-out set remains unseen during training.

The teacher processes each training paragraph and assigns probabilities across all available tags, preserving more information than a single top choice.
The smaller student processes the same paragraphs and produces its own probability distributions.
A loss function measures the mismatch between the two distributions. Training updates only the student’s parameters to reduce that mismatch.
The student is evaluated on the held-out paragraphs for tagging quality, processing time and performance on uncommon or overlapping topics.

Result: The student handles common tags within the portal window and needs less computing capacity. It remains less reliable on rare or ambiguous paragraphs, which are routed to the information management officer or processed with the teacher model.

In a United Nations workplace

The procurement unit’s national officer at a country office is closing the procurement annex for a donor progress report at quarter-end. An approved large model can sort non-sensitive requisition descriptions into reporting categories, but it is too slow for the reporting run. The technical evaluator proposes a smaller student model trained to match the larger model’s category probabilities, then tested before the annex is cleared.

The smaller model is not simply a compressed copy. During training, its outputs are compared with the teacher’s signals and only the student’s parameters are updated. If evaluation is satisfactory, it may provide quicker, cheaper routine classification while the established clearance process remains unchanged.

Before you try thisTender submissions, bid evaluations, supplier pricing, personal data and material from an active procurement must not be entered into any model or environment not approved for that information. Use authorised, minimised training data with appropriate access controls. Test for inherited errors and information leakage, and retain human review before figures or categories enter the report.

Where you may hear it

During a vendor demo to an ICT governance board
In the technical evaluation meeting for an AI procurement
At a model deployment review before a portal release
In a data science discussion on reducing inference costs

Avoid confusion

Often confused with

Common misconceptions

Common misconception: Distillation simply copies part of a large model into a smaller file.
Correction: Not usually. In standard distillation, a separate student model is trained to imitate the teacher’s outputs. That means the smaller model learns a new set of parameters. This is different from merely saving fewer bits or deleting weights from the original model.
Common misconception: A distilled model is always just as good as the original.
Correction: It can retain much of the teacher’s performance, but there is usually some trade-off. The student often does well on common patterns it saw during distillation, yet loses accuracy or robustness on rare, complex or out-of-domain cases. The result depends on the teacher, the student size, the data and the task.
Common misconception: Distillation and quantisation are the same thing because both make models smaller.
Correction: They have a similar goal but a different mechanism. Distillation trains a student model to imitate a teacher. Quantisation changes how model numbers are stored or computed, often using fewer bits. A model can be distilled, quantised, both, or neither.
Common misconception: If the teacher model is safe or aligned, the student automatically keeps that behaviour.
Correction: Some behaviour can transfer, but not perfectly or automatically. Safety-related behaviour depends on what signals were used during distillation, what data was included and how the student was evaluated. Important safeguards still need dedicated testing, especially in higher-risk uses.

Quick check

Which statement best describes distillation?

Go deeper

The simple version

A big model shows a smaller model how to answer many examples. The smaller model learns to produce similar outputs, so it can often do a similar job faster and more cheaply.

The technical version

Distillation is a training procedure in which a student model is optimised to match a teacher model’s behaviour on a set of inputs. The targets may be hard labels, soft labels such as probability distributions over classes, generated token distributions, hidden-state representations or combinations of these. The loss function typically measures the mismatch between teacher and student outputs and updates only the student parameters. In language models, distillation may occur after pretraining or task-specific fine-tuning, and can be used to compress capability into a smaller architecture for lower inference cost. It differs from parameter-efficient adaptation methods such as LoRA, which modify how a model is adapted, and from compression methods such as quantisation or pruning, which reduce storage or computation without necessarily training a separate student to imitate a teacher.

Reviewed external resources

EncyclopaediaKnowledge distillation (opens in a new tab)Wikipedia
This link gives a broad reference-style overview of knowledge distillation, useful for readers who want the standard terminology and a starting point for further reading.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This course link suits readers who want practical model-training context and may help place distillation among other adaptation and optimisation methods.
Deeperlink checked 2026-07-31
CourseShort Courses (opens in a new tab)DeepLearning.AI
This course link suits readers who prefer short guided lessons and want a more applied introduction before reading technical papers.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Distillation (opens in a new tab)YouTube search
This is a YouTube search results page, so it helps readers find different video explainers and compare teaching styles rather than relying on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Distillation (opens in a new tab)Reddit search
This is a Reddit search results page, so it is useful for seeing community questions and practitioner discussions, but claims there should be checked against stronger sources.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Distillation trains a student model to match signals from a teacher model. The student is often smaller and faster, but it must be tested for performance lost or errors inherited in the transfer.