Learn AI /Training and Model Adaptation /Pretraining
Pretraining
- Intermediate
- Training and Model Adaptation
- Policy officers
- Technical readers
Also called: pre-training.
Back to the milestone you came from
In five seconds
What it is
A large-scale training stage where a model learns reusable patterns from data, producing a general-purpose starting point that later steps can adapt for specific tasks.
In plain words
Pretraining gives a model its broad starting abilities by exposing it to a huge amount of general data before any task-specific customisation.
An everyday example
Imagine repeatedly covering the final word in sentences from novels, recipes, news articles and instruction manuals. You guess the missing word, uncover it, note the error and adjust your next guess. After enough practice, you become better at recognising patterns across many kinds of writing. That broad practice is like pretraining. If you are later coached to sort customer messages into delivery, billing and returns, that is a separate adaptation step built on what you learned earlier.
A person can deliberately reflect on meaning and use real-world experience. A pretrained model does not do that in the human sense. It adjusts mathematical weights based on patterns in data. Also, the librarian can remember a small collection exactly, while a model stores distributed statistical patterns rather than a neat catalogue of every source.
Remember thisPretraining creates the reusable base model. Fine-tuning, retrieval or prompting may adapt that base for a particular task, but they do not retroactively change what happened during pretraining.
This term is contested
The basic idea is widely used, but exact boundaries vary: some people use 'pretraining' narrowly for the initial self-supervised stage, while others use it more loosely for almost any earlier training before deployment.
Understand it
Understand it
Why it matters
Pretraining shapes what a model can do before your organisation touches it. If that stage was broad and strong, later adaptation may be faster and cheaper. If it was weak, no amount of prompting may fully fix missing capabilities. For decision-makers, the term matters because vendors often say a system was 'trained on our data' without separating pretraining from fine-tuning or retrieval. Those are different choices with different costs, risks and governance questions. Pretraining also helps explain why a model has a knowledge cutoff, why it may reflect patterns or biases present in large datasets, and why a later safety layer cannot completely replace the strengths and limits built in at this earlier stage.
From raw model to pretrained checkpoint
Show that pretraining uses massive general data and repeated prediction practice to create a broad starting model, not a finished task-specific system.
01
1. Start with a raw model
Begin with a model architecture whose parameters are initial values rather than useful skills.
Draw a simple network icon in a pale box labelled 'Raw model'. Add a small caption: 'Structure exists, patterns not yet learned'.
02
2. Feed in broad data
Use a very large and varied dataset, such as text from many domains, so the model sees common patterns across language and topics.
Draw a dataset box with stacked document icons feeding into the raw model. Label it 'Large general dataset'.
03
3. Train by prediction
At each step, the model predicts missing or next content, compares the prediction with the original data and adjusts parameters to reduce error.
Show a loop arrow around the model with three mini labels: 'predict', 'compare', 'adjust'. Use a small gauge icon for error decreasing.
04
4. Save a checkpoint
After many iterations, save the resulting broad model as a checkpoint that can later be used directly or adapted further.
Draw a final box labelled 'Pretrained checkpoint' with two faint outgoing dashed arrows labelled 'use' and 'adapt later'.
A left-to-right sequence with four boxes connected by arrows. Read from a randomly initialised model, through large-scale data exposure and repeated prediction training, to a saved pretrained checkpoint that later stages can adapt. Each box names what goes in, what happens and what comes out.
On hover or tap, each step reveals a one-sentence note explaining whether the stage is about general pattern learning or later task-specific adaptation.
Describe this without the diagram
The diagram is a four-step process read left to right. Step 1 shows a raw model with structure but no learned patterns. Step 2 adds a large general dataset, indicating that broad text or other data is used as input. Step 3 shows a repeated loop of predict, compare and adjust, meaning the model learns by making predictions and changing its parameters to reduce error. Step 4 shows a pretrained checkpoint, which is the saved result of this broad training stage and can later be used or adapted for specific tasks.
With reduced motion: A static four-box sequence replaces animation. Arrows show the same order: raw model, large general dataset, repeated prediction training, pretrained checkpoint, with dashed arrows from the final box to later use or adaptation.
Pretraining is the early training stage that gives a model broad general capabilities before it is adapted for a narrower use. The input is usually a very large collection of text, images, audio or other data. For language models, that often means text from many sources and domains.
In many modern systems, this stage uses self-supervised learning. That means the training signal comes from the data itself rather than from people writing labels for every example. For a language model, a common method is to hide or predict parts of text and adjust the model so its predictions are closer to the original text.
What happens internally is repeated adjustment of many numerical parameters, often called weights. Over many training steps, the model becomes better at capturing statistical patterns such as grammar, common facts, style, topic structure and relationships between words. This does not guarantee truth, reasoning quality or safe behaviour. It mainly creates a broad base model, often called a foundation model.
What comes out is a checkpoint: a saved version of the model after this broad training stage. That checkpoint can then be used directly, or adapted further through methods such as fine-tuning, instruction tuning or preference-based methods. Pretraining differs because it creates reusable starting parameters before task-specific or instruction-following adaptation.
How “Practising with unfinished sentences” maps onto the real thing
| In the story | In the AI system |
|---|---|
| Months spent scanning many kinds of material | Pretraining on a very large and varied dataset |
| Learning titles, subjects and document patterns | Learning broad statistical patterns in data |
| Later coaching for one exact request | Fine-tuning or instruction tuning for a narrower use |
| The librarian after initial preparation | A pretrained base or foundation model |
A worked example
An information management officer at a regional hub reviews a supplier’s training record for a multilingual text model being considered to sort public consultation responses by topic.
Starting from: A transformer model with randomly initialised parameters and a broad multilingual text corpus that the supplier is entitled to use.
- The model receives text sequences and predicts the next token from the preceding tokens.
- For each prediction, the training system compares the model’s probabilities with the token that actually appeared and calculates a loss.
- Gradient-based optimisation repeatedly adjusts the model’s parameters to reduce that loss across the corpus.
- The supplier saves a checkpoint once the training stage is complete. The checkpoint captures reusable language patterns but has not yet learned the consultation categories.
Result: Pretraining produces a general-purpose model checkpoint, not a finished classification tool. Later adaptation can teach the consultation categories, while evaluation must still test language coverage, accuracy and handling of responses that do not fit the expected labels.
In a United Nations workplace
A partnerships officer supporting a UN Country Team working group is clearing talking points for informals with potential technology partners before close of business. One supplier describes its multilingual assistant as “pretrained on global development content”, without saying what happened after that stage.
The officer needs to distinguish the broad training stage from later task-specific adaptation. Useful follow-up questions cover the pretraining data, languages, training objective, completion point and resulting checkpoint, as well as any subsequent fine-tuning or retrieval system. The original claim alone does not establish that the assistant can handle UN terminology or current material reliably.
Before you try thisThe officer must not test the assistant by uploading non-public talking points, negotiation text, personal data or internal correspondence unless the service has been approved for that material. A description of the pretraining corpus is not a security assessment, nor does it show how prompts and uploaded files are retained or reused.
Where you may hear it
- A vendor demo during a procurement meeting
- An ICT governance board review of an external AI service
- A technical due-diligence meeting before clearance
- An inter-agency working group reviewing terms of reference for an AI tool
Avoid confusion
Avoid confusion
Often confused with
Fine-tuning
Updating a pre-trained model's parameters with additional examples to improve performance on a narrower task, style or domain.
Self-supervised learning
A training approach that creates its own learning targets from raw data, so a model can learn useful patterns without manual labels.
Instruction tuning
A post-pretraining adaptation step that teaches a general language model to respond to prompts in the requested format, style and task pattern.
In-context learning
A model uses examples and instructions placed in the current prompt to imitate a pattern for that one interaction, without changing its stored parameters.
Common misconceptions
Common misconception: Pretraining means the model was trained on my organisation's documents.
Correction: Usually it does not. Pretraining normally refers to broad training on a very large general dataset before an organisation adapts the model. A vendor may also train or fine-tune later on customer data, but that is a separate stage and should be described separately.
Common misconception: After pretraining, the model already knows how to follow instructions safely for a real workplace task.
Correction: Not necessarily. Pretraining mainly teaches broad data patterns. Helpful instruction-following behaviour, preferred response style and safety constraints are often added or improved in later stages such as instruction tuning or preference-based training, and even then they are not guaranteed.
Common misconception: Pretraining stores a clean copy of all facts from the internet.
Correction: A pretrained model learns statistical patterns distributed across many parameters. It can reflect common facts seen in data, but not as a checked, complete or up-to-date database. That is why models can omit, distort or invent information.
Quick check
Which statement best describes pretraining for a language model?
Pretraining is the large, general learning stage that creates a base model. Uploading internal documents may relate to retrieval or later fine-tuning, safety review is a governance activity rather than pretraining, and examples in a prompt are in-context learning rather than parameter training.
Go deeper
Go deeper
The simple version
Pretraining gives a model its broad starting abilities by exposing it to a huge amount of general data before any task-specific customisation.
The technical version
Pretraining is the initial large-scale optimisation phase in which a model's parameters are learned from a broad dataset, often with a self-supervised objective such as next-token prediction or masked-token reconstruction. Starting from random or transferred initialisation, the system repeatedly computes predictions, measures loss against the original data-derived target and updates weights through gradient-based optimisation. The result is a checkpoint whose representations encode general statistical regularities across the training distribution. Later stages, such as supervised fine-tuning, instruction tuning, preference learning or reinforcement learning from human feedback, modify this pretrained checkpoint for narrower behaviours or objectives.
Reviewed external resources
EncyclopaediaWikipedia search: Pretraining (opens in a new tab)Wikipedia
This Wikipedia search results page helps readers quickly find multiple related articles and suits someone who wants broad orientation rather than one authoritative source.
search results, not one pagelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This Hugging Face course gives structured technical teaching on how large language models are built and suits readers who want a guided learning path.
Deeperlink checked 2026-07-31
CourseShort Courses (opens in a new tab)DeepLearning.AI
These DeepLearning.AI short courses offer practical, lesson-based introductions and suit busy learners who prefer applied explanations.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Pretraining (opens in a new tab)YouTube search
This YouTube search results page helps readers compare different video explanations and suits people who learn best by watching and listening.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Pretraining (opens in a new tab)Reddit search
This Reddit search results page shows community discussions, questions and disagreements and suits readers who want informal explanations and common points of confusion.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Pretraining creates the reusable base model. Fine-tuning, retrieval or prompting may adapt that base for a particular task, but they do not retroactively change what happened during pretraining.