Learn AI /Transformers and Attention /Self-attention
Self-attention
- Deeper
- Transformers and Attention
- Technical readers
Back to the milestone you came from
In five seconds
What it is
A mechanism that lets each token reweight information from available tokens in the same sequence, including itself, before the next layer processes it.
In plain words
Each token draws more information from the available tokens in the same input that receive higher computed weights.
An everyday example
A clerk in Lima receives a phone photo of a paper form. To identify each blurred field, the clerk compares one field at a time with every field on the standard template: the position of the date, the shape of the ID box, the nearby label and the spacing around it. For the field that looks like a date, the clerk gives more weight to template regions that match those clues and less weight to unrelated regions such as the signature line. The clerk then combines those stronger matches to decide how to label that field in context, not just by its isolated appearance.
The analogy compares an image to a separate template, but self-attention usually works within one sequence attending to itself, not to an external reference. A clerk may use explicit rules such as 'dates look like this', while self-attention uses learned numeric projections. Also, real self-attention updates all positions in parallel and produces continuous vectors, not human-readable labels at each step.
Remember thisSelf-attention lets each token rebuild its representation from a weighted combination of all tokens it is allowed to attend to, including itself, before the next layer processes it.
Understand it
Understand it
Why it matters
Self-attention helps explain why transformers work well on language, code and other sequences. For a non-specialist, this matters because it shows what the model is actually doing: not retrieving one exact rule, but repeatedly reweighting parts of the input to build context-aware representations. That helps you judge strengths and limits. A system using self-attention may capture long-range relationships better than older sequence models, but it can still miss meaning, rely on weak patterns or spread errors through the representation. Knowing this supports better choices about procurement, evaluation, safety checks and when human review is needed.
How one token gathers context from the whole sequence
After using this explainer, a reader should be able to say how one token compares itself with all tokens, turns match strengths into weights, and combines their information into a new representation.
Select a word to see what it attends to
These weights are illustrative. They are hand-written to show the shape of the mechanism. A real model computes them from its parameters, and its numbers would differ.
An interactive sequence of tokens is shown in a row. Selecting a focus token highlights lines from its query to every token's key, with line thickness showing score strength. A second panel shows the normalised weights and the resulting weighted mix of value vectors. The reader can switch examples, move the focus token, and compare before and after representations.
The reader can choose among short example sequences, click any token to make it the focus, and toggle views for scores, normalised weights and the final weighted sum. A reset button returns to the default example.
Describe this without the diagram
Screen-reader version: You hear a sequence of tokens, then choose one as the focus. For that token, the interface reads a list of comparison scores against every token in order, followed by the normalised weights that sum to 1. It then states that the output for the focus token is formed by multiplying each token's value vector by its weight and adding the results. A final summary compares the original token representation with the new context-aware representation and explains which tokens influenced it most.
With reduced motion: Reduced-motion version: no animated lines or transitions. The same example appears as a static table with columns for token, score, weight and weighted value contribution, plus a fixed side-by-side view of the original and updated representation for the selected token.
Self-attention is a way for a model to compute how strongly information from parts of one sequence should contribute at each position. A sequence can be a sentence, code tokens, image patches, or other ordered pieces of input. For every token, the model produces a new representation: a numeric description that mixes information from that token and from other available tokens.
The input is a sequence of token representations, often called embeddings. For each token, the model creates three derived vectors: a query, a key and a value. A vector is an ordered list of numbers. The query from one token is compared with the keys of available tokens to get attention scores. Higher scores give information from those tokens more influence at this step.
Those scores are turned into weights, usually so they add up to 1 across the available tokens. The model then builds a weighted sum of the value vectors. That weighted sum forms the self-attention output for the token being processed. This happens for every token, so each token can gain context from distant as well as nearby tokens.
This is called self-attention because the queries, keys and values come from the same sequence. That differs from cross-attention, where one sequence attends to a different sequence, such as a decoder attending to an encoder output. It is also not the same as attention in the broad sense: self-attention is one specific attention pattern used inside transformers.
How “Matching a photographed form to the right template” maps onto the real thing
| In the story | In the AI system |
|---|---|
| one field in the photographed form being checked | one token whose representation is being updated |
| comparing that field with all template fields | comparing one query with all keys in the sequence |
| stronger and weaker matches across template regions | attention scores turned into weights |
| combining the best-matching template clues | weighted sum of value vectors to form a new representation |
A worked example
In the sentence "The catalogue entry was returned to the focal point because it was incomplete", examine the token "it".
Starting from: Tokens: [The, catalogue, entry, was, returned, to, the, focal, point, because, it, was, incomplete]
- The layer projects every token representation into a query, key and value. The query for "it" is compared with the keys for all allowed tokens, including its own key.
- For one illustrative attention head, the normalised weights might assign 0.30 to "entry", 0.25 to "incomplete", 0.12 to "catalogue", 0.08 to "it", 0.07 to "returned", and 0.18 across the remaining tokens.
- The representation for "it" becomes a weighted sum of the corresponding value vectors. Information from "entry" and "incomplete" therefore contributes more strongly than information from "focal point".
- A residual connection typically combines this attention output with the token's earlier representation before the result passes to subsequent processing.
Result: The position of "it" is unchanged, but its updated representation now carries evidence that it refers more strongly to the catalogue entry than to the focal point.
In a United Nations workplace
At the annual workplan review, an ombudsman case officer in a documentation and library service uses an approved drafting tool to revise a non-sensitive paragraph on records management. Self-attention allows each token to draw on other tokens in the paragraph, so a phrase such as "the revised filing procedure" can influence the later word "it" even when several words intervene.
The mechanism helps explain how a transformer keeps references, qualifications and defined terms connected across a sentence. It does not retrieve the correct interpretation from a file. It reweights token representations within the sequence presented to it.
Before you try thisOmbudsman case files, identities, allegations, correspondence and associated metadata must not be entered into a tool unless it is explicitly approved for that material. Such information is confidential and may expose individuals or compromise a case. Keep the exercise to non-sensitive administrative text and review the output before clearance.
Where you may hear it
- A vendor demo to an ICT governance board
- A technical review of terms of reference for a language-model procurement
- An information management officer's briefing before a drafting-tool pilot
- A research presentation in the margins of a digital cooperation session
Avoid confusion
Avoid confusion
Often confused with
Attention
A method that lets a model give different importance to different parts of the input while producing an output.
Multi-head attention
A transformer mechanism that runs several attention patterns in parallel so the model can combine different kinds of relationships between tokens.
Encoder and decoder
In an encoder-decoder model, the encoder represents input, while the decoder uses those representations to generate output, often one token at a time.
Common misconceptions
Common misconception: Self-attention means the model looks only at the most important word.
Correction: Not usually. Each output token is built from a weighted mixture of all tokens, not a single winner. Some weights may be large and many small, but the mechanism is generally a soft combination across the sequence.
Common misconception: Self-attention and attention are the same term.
Correction: Self-attention is one specific kind of attention. The general term attention covers several patterns. In self-attention, queries, keys and values all come from the same sequence. In cross-attention, they do not.
Common misconception: Self-attention automatically understands word order.
Correction: By itself, self-attention compares token representations without an inherent notion of sequence order. Transformer systems therefore add position information, often through positional encodings or related methods, so the model can distinguish 'A before B' from 'B before A'.
Common misconception: A high attention weight proves a model's reasoning or explanation.
Correction: Attention weights show how information is mixed at one step of computation. They can be useful clues, but they are not a complete or universally accepted explanation of why the full model produced a final output.
Quick check
Which statement best describes self-attention in a transformer?
In self-attention, each position compares its query with keys from available positions in the same sequence, obtains weights, and forms a weighted mixture of their value vectors. The weights can differ by position. Cross-attention uses a separate source sequence, while multi-head attention describes running several heads in parallel.
Go deeper
Go deeper
The simple version
Each token draws more information from the available tokens in the same input that receive higher computed weights.
The technical version
Given input token representations X, the layer learns projections that produce Q, K and V. For each position i, it computes similarity scores between q_i and every allowed k_j, scales and normalises them into weights, and uses those weights to form a weighted sum of the corresponding v_j. The result is a contextualised representation for position i. Doing this for every position gives a full set of updated token representations. In transformers, this is typically wrapped with residual connections, layer normalisation and feed-forward sublayers. Self-attention describes where Q, K and V come from. Multi-head attention describes running several attention heads in parallel, so multi-head attention can perform self-attention.
Reviewed external resources
EncyclopaediaSelf-attention (opens in a new tab)Wikipedia
A broad reference overview of the term, useful for readers who want a general summary and links to related concepts.
Intermediatelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
The original transformer paper, best for readers who want the canonical technical source and are comfortable with research writing.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
An illustrated walkthrough of the transformer architecture, suitable for readers who want an intuitive but still technical explanation.
Intermediatelink checked 2026-07-31
VideoAttention in transformers, step by step (opens in a new tab)3Blue1Brown
A step-by-step visual explanation of how attention works in transformers, well suited to readers who learn best from intuition and animation.
Deeperlink checked 2026-07-31
VideoVideo explanations: Self-attention (opens in a new tab)YouTube search
A YouTube search results page that helps readers find multiple video explanations and compare teaching styles; useful if one explanation has not clicked yet.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Self-attention (opens in a new tab)Reddit search
A Reddit search results page that helps readers see recurring questions, confusions and community explanations; useful for informal discussion, not as an authoritative source.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Self-attention lets each token rebuild its representation from a weighted combination of all tokens it is allowed to attend to, including itself, before the next layer processes it.