Learn AI /Transformers and Attention /Autoregressive model
Autoregressive model
- Deeper
- Transformers and Attention
- Technical readers
Also called: autoregressive.
Back to the milestone you came from
In five seconds
What it is
A model that assigns each token a probability based on earlier tokens in the same sequence, and generates by predicting and adding one new token at a time.
In plain words
It writes a sequence in order. It looks at what is already there, predicts the next piece, adds it, and repeats.
An everyday example
A team is grouping survey answers that raise the same issue. One colleague writes a short summary card, but after each word they pause, look only at the words already on the card and the source notes on the desk, and choose the next word that best continues the sentence. They keep extending the card word by word until the summary is complete. If they choose an awkward word early, the rest of the sentence often has to work around it.
A person can reflect on the overall meaning, revise earlier words, and use outside judgement. A basic autoregressive generation process does not literally pause to understand in that human sense, and standard left-to-right decoding does not freely rewrite earlier tokens once chosen. Also, some autoregressive models work on data such as audio or images in a fixed order, not just words on a page.
Remember thisAn autoregressive model generates one token at a time. Each selection becomes part of the context for the next prediction, so an early choice can shape the remainder of the output.
Understand it
Understand it
Why it matters
If you use, buy, govern or evaluate language AI, autoregressive behaviour affects speed, cost, reliability and risk. These models generate output sequentially, so long responses take multiple prediction steps. Each step can introduce small errors, and those errors can influence later text. This helps explain common behaviours such as fluent continuation, occasional repetition, drifting off topic, and sensitivity to prompt wording. It also matters for system design: caching earlier computations can improve efficiency, while human review is important in high-stakes uses because the model’s output is generated plausibly step by step, not guaranteed fact by fact.
How autoregressive generation builds a sequence
After viewing this, a reader should be able to explain that the model predicts one token from earlier tokens, appends it, and repeats without looking ahead.
01
Start with context
The process begins with an initial prompt or prefix, such as a few words already present in the sequence.
Draw a row of token boxes labelled “The”, “meeting”, “starts”, “at”. Use a bracket below labelled “current context”.
02
Compute next-token probabilities
The model reads the current context and produces scores for possible next tokens. These scores can be converted into probabilities.
To the right, draw a model box. From it, show a small ranked list: “09:00 0.62”, “10:00 0.21”, “noon 0.09”, “once 0.03”.
03
Choose one token
A decoding rule selects one token, for example the highest-probability option or a sampled option.
Highlight “09:00” in the ranked list and place it in a new token box with a label “selected token”.
04
Append and repeat
The selected token is added to the sequence, becoming part of the context for the next prediction step.
Extend the original token row with a new box “09:00”. Add a loop arrow from the extended row back to the model box, labelled “next step uses updated context”.
Show a left-to-right sequence growing across four steps. Each step displays the current context, the model’s next-token probability output, the chosen token, and the updated context. Read from step 1 to step 4 to see the repeated loop: context in, probabilities out, one token selected, context extended.
Hovering or tapping a step highlights the relevant token boxes and the probability list for that step.
Describe this without the diagram
The diagram explains autoregressive generation as a four-step loop. Step 1 shows an initial context made of tokens: “The”, “meeting”, “starts”, “at”. Step 2 shows a model reading those tokens and producing a ranked list of possible next tokens with probabilities, such as “09:00 0.62”, “10:00 0.21”, “noon 0.09”, and “once 0.03”. Step 3 shows one token being selected from that list, here “09:00”. Step 4 shows that selected token appended to the sequence, making the new context “The meeting starts at 09:00”, and a loop indicates the process repeats. The key idea is that each new token is predicted from earlier tokens only, then added back into the context.
With reduced motion: A static four-panel diagram shows the same loop without animation: initial context, probability list, selected token, and extended context with a visible arrow indicating repetition.
An autoregressive model handles a sequence in order. A sequence can be words in a sentence, sound samples in audio, or pixels arranged in a scan order. The model takes the tokens it has already seen or already generated and uses them to estimate what token should come next.
Input goes in as an ordered context: for example, the words "The meeting starts at". Inside the model, each earlier token is turned into numbers called vectors, and the model computes patterns over that growing context. In modern language systems, a transformer often does this with self-attention, which lets each position weigh earlier positions differently.
The output at each step is not usually a single guaranteed answer. It is a probability distribution: a ranked list of possible next tokens with scores that can be turned into probabilities. A decoding method then picks one token, such as the highest-probability token or a sampled token, appends it to the context, and repeats the process.
This makes autoregressive generation causal: later tokens depend on earlier ones, but not on future tokens that have not been generated yet. That is why these models are strong at continuing text and dialogue. It is also why errors can accumulate: if one early token is poor, later steps build on it.
The term is often confused with "transformer", but they are not the same. Autoregressive describes the prediction pattern over a sequence. Transformer describes one model architecture often used to implement that pattern. It is also different from diffusion models, which generate by iterative denoising rather than next-step sequence prediction.
How “Writing a summary card one line at a time” maps onto the real thing
| In the story | In the AI system |
|---|---|
| The growing summary card | The partial sequence already generated |
| Choosing the next word after reading earlier words | Predicting the next token from previous tokens |
| Pausing after each word and continuing | Repeated step-by-step generation |
| An awkward early word affecting later wording | Error propagation through the sequence |
A worked example
A drafting assistant continues a sentence in a donor progress report from the prefix: “The reporting period concluded with”.
Starting from: Prompt tokens: [The] [reporting] [period] [concluded] [with]
- From the prefix, the model assigns probabilities to possible next tokens. Illustrative leading options are “all” at 0.34, “three” at 0.21 and “the” at 0.15.
- The decoding rule selects “all”. With that token added, the context is now “The reporting period concluded with all”, and the model calculates a new next-token distribution.
- The leading options now include “activities” at 0.46, “planned” at 0.24 and “partners” at 0.09. The decoder selects “planned”.
- Conditioned on the longer prefix, the model selects “activities”, then “completed”, then a full stop through successive prediction steps.
Result: The output is “The reporting period concluded with all planned activities completed.” It was not retrieved as a complete sentence. It was assembled token by token, and each selected token changed the probabilities for the next one.
In a United Nations workplace
At close of business, an ICT officer in an inter-agency secretariat is testing an approved drafting assistant against cleared, non-sensitive bullet points for a donor progress report. The assistant drafts the narrative one token at a time, with each choice conditioned on the report prompt and the wording it has already produced.
An early choice such as “completed” rather than “initiated” changes the probabilities of later words and may lead the paragraph away from the cleared facts. Understanding the autoregressive mechanism helps the officer recognise why fluent text still requires line-by-line verification before clearance.
Before you try thisDo not enter personal data, confidential partner information, procurement-sensitive material or uncleared figures into a tool that has not been approved for them. Such material may be exposed or handled outside required controls. Use only approved systems, minimise the input and verify every claim against the cleared source.
Where you may hear it
- A technical review of a drafting assistant before an ICT governance board
- A vendor demonstration during a procurement meeting
- An inter-agency working group discussing safeguards for text generation
- A briefing with the front office on why generated wording can drift
- A model evaluation meeting before an approved tool is rolled out
Avoid confusion
Avoid confusion
Often confused with
Transformer
A neural network design that processes many parts of a sequence in parallel by weighting which parts matter most to each other.
Encoder and decoder
In an encoder-decoder model, the encoder represents input, while the decoder uses those representations to generate output, often one token at a time.
Diffusion model
A diffusion model creates media by learning how to remove noise step by step until a coherent image, audio clip or other output appears.
Common misconceptions
Common misconception: Autoregressive means the model only copies the previous word.
Correction: It uses the whole available prefix, not just the immediately previous token. In transformer-based systems, self-attention can weigh many earlier positions when estimating the next token. The “auto” part refers to conditioning on the sequence itself, not simple copying.
Common misconception: Autoregressive and transformer mean the same thing.
Correction: They describe different things. Autoregressive is a generation pattern: predict the next item from earlier items. Transformer is an architecture for processing sequences. Many transformers are autoregressive during generation, but not all transformer uses are autoregressive.
Common misconception: Because it predicts one token at a time, the model cannot use long-range context.
Correction: It can use long-range context up to its context window and architecture limits. In practice, performance may weaken over long spans, but the mechanism is not restricted to only the last few words.
Common misconception: If the next-token probabilities are high, the full answer is reliable.
Correction: High local confidence at each step does not guarantee the whole sequence is correct. The model may produce a fluent chain of individually likely tokens that still forms an inaccurate or unsupported statement.
Quick check
Which statement best captures what makes a model autoregressive?
Autoregressive generation is defined by sequential conditioning on the growing prefix. It is not the same as all-at-once prediction, diffusion-style denoising, or a model that only looks one step back.
Go deeper
Go deeper
The simple version
It writes a sequence in order. It looks at what is already there, predicts the next piece, adds it, and repeats.
The technical version
An autoregressive model factorises the probability of a sequence into a product of conditional probabilities over positions, typically left to right: each token is modelled given the preceding tokens. In modern language models, the input prefix is embedded into vectors, position information is added or encoded, and masked self-attention enforces causality so position t can attend only to positions 1 through t. The network outputs logits over the vocabulary for the next token; a decoding procedure such as greedy selection, beam search or stochastic sampling chooses one token, appends it to the prefix, and feeds the longer prefix back for the next step. During training, teacher forcing commonly supplies the true previous tokens; during generation, the model conditions on its own earlier outputs, which creates exposure to error accumulation. The same autoregressive idea can be applied beyond text, but the ordering of the sequence then becomes an important design choice.
Reviewed external resources
EncyclopaediaAutoregressive model (opens in a new tab)Wikipedia
This link gives a broad reference-style overview of the term and suits readers who want a quick starting definition and related terminology.
Intermediatelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
This link gives the original transformer paper context for causal decoding and suits readers comfortable with foundational research writing.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
This link gives an intuitive visual walkthrough of transformer generation behaviour and suits readers who learn best from diagrams and plain-language explanations.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Autoregressive model (opens in a new tab)YouTube search
This link gives YouTube search results for many video explanations of the term, so it suits readers who prefer to compare different teaching styles rather than rely on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Autoregressive model (opens in a new tab)Reddit search
This link gives Reddit search results for community discussions about the term, which suits readers who want to see common questions, confusions and informal explanations.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
An autoregressive model generates one token at a time. Each selection becomes part of the context for the next prediction, so an early choice can shape the remainder of the output.