Learn AI /Transformers and Attention /Attention
Attention
- Intermediate
- Transformers and Attention
- Technical readers
Also called: attention mechanism.
Back to the milestone you came from
In five seconds
What it is
A method that lets a model give different importance to different parts of the input while producing an output.
In plain words
It is a way for a model to focus more on some parts of the text and less on others when updating each part.
An everyday example
A project officer is writing a summary of a long meeting. For each sentence in the summary, she looks back at her notes and marks some lines as very useful, some as a little useful, and others as not useful for that sentence. She then writes the sentence using a blend of the most relevant notes rather than copying one line exactly. When she moves to the next sentence, she repeats the process with a different pattern of importance.
A person may consciously understand the meeting and the purpose of the summary. An attention mechanism does not have that kind of human understanding. It performs numerical comparisons between vectors, and in many models this happens for all positions in parallel, not one handwritten sentence at a time.
Remember thisAttention creates context-dependent representations by weighting and combining information from different input positions. The weighting is recalculated for each position.
This term is contested
Whether attention weights count as an explanation of model behaviour is debated, so it is safer to treat them as one signal rather than a full account.
Understand it
Understand it
Why it matters
If you use or oversee AI systems, attention helps explain why outputs change with context. That matters when judging reliability, debugging errors and deciding how much human review is needed.
How attention shifts importance across a sentence
After using this explainer, a reader should be able to say that attention lets one token gather more information from some tokens than from others, producing a context-aware representation.
Select a word to see what it attends to
These weights are illustrative. They are hand-written to show the shape of the mechanism. A real model computes them from its parameters, and its numbers would differ.
The explainer shows a short sentence as clickable tokens in one row and the currently selected token in a highlight box. Lines connect the selected token to all tokens, with thicker and darker lines for higher weights. A side panel shows weights as bars and a final mixed representation as a stacked bar.
Click or keyboard-select a token to change the focus; hover or focus on a bar or line to read its exact weight; use previous and next controls to step through the sentence.
Describe this without the diagram
Screen-reader version: 'Sentence: The committee approved the budget because it was urgent. Selected token: it. Relative weights from this token to others are listed from highest to lowest. approved: high. budget: medium. urgent: medium. the, committee, because, was: lower. The mechanism then forms a new representation for it by combining information from all tokens, with larger contributions from the higher-weight tokens. Selecting another token gives a different weighting pattern.'
With reduced motion: A static version shows one selected token at a time with fixed connection lines and a ranked weight table beside it. Changing the token swaps the diagram instantly without animated movement or fading.
Imagine a model processing a sentence made of tokens. A token is a small unit such as a word or part of a word. For each position, attention lets the model compare its representation with a set of other representations and give more weight to information that is more relevant.
What goes in are token representations, often called embeddings or vectors. A vector is a list of numbers that stores features in a form a model can calculate with. The mechanism computes attention scores, converts them into weights and uses those weights to control how strongly information from each token contributes.
What happens next is not just selection but mixing. The model does not usually pick one token and ignore the rest. It combines information from several tokens, giving larger influence to some and smaller influence to others. This can help connect information across distant positions, such as a pronoun and a possible referent.
What comes out is a new set of token representations enriched with context. Attention is a general mechanism whose information may come from the same sequence or another source. Self-attention is the specific case where positions attend to information from the same sequence.
How “Choosing the right notes during a meeting” maps onto the real thing
| In the story | In the AI system |
|---|---|
| the sentence she is currently writing | the current token position being updated or predicted |
| marking some notes as more useful than others | assigning attention weights or scores |
| blending several relevant notes | combining information from multiple token representations |
| repeating with a new pattern for the next sentence | recomputing attention for each position |
A worked example
A language model is processing: “Maria thanked Amina because she helped.” It is computing a context-dependent representation for the token “she”.
Starting from: Tokens: Maria | thanked | Amina | because | she | helped
- The representation of “she” is projected into a query. The token representations across the sentence are projected into keys and values.
- The query is compared with each key to produce relevance scores, which are converted into weights. An illustrative pattern might assign larger weights to “Amina” and “helped” than to the other tokens.
- The model forms a weighted combination of the value vectors. Information associated with “Amina” and “helped” therefore contributes more strongly at this position.
- Later layers process this context-dependent representation. Other attention heads may learn different patterns, such as grammatical structure or longer-range relationships.
Result: The result is a contextual representation of “she”, not a final interpretation of the sentence. It can help later layers resolve the reference, but an attention pattern alone does not prove why the model reached a particular answer.
In a United Nations workplace
An associate expert supporting the programme officer in a UN Country Team working group is asked to turn an implementing partner's progress report into a two-paragraph update before close of business. In an approved internal drafting tool, attention allows the model to weight passages on outputs, delays and corrective action differently for each part of the draft. A sentence on delivery dates may draw heavily on the timeline, while one on results may give more weight to the monitoring table.
For each position, the model compares a query vector with key vectors derived from the report's tokens, then uses the resulting weights to combine the corresponding value vectors. Because the weights change by position, different parts of the report shape different words and sentences in the draft.
Before you try thisThe progress report must not be uploaded if it contains personal, protection-sensitive, security-sensitive, medical or other confidential material unless the tool is approved for that class of data. Use a cleared extract where necessary, and have the programme officer verify the draft against the report before circulation.
Where you may hear it
- In a procurement meeting during a vendor demo for a drafting tool
- At a technical briefing to an information and communications technology governance board
- In an inter-agency working group reviewing machine-translation output
- During a model-assurance review before a pilot is cleared
Avoid confusion
Avoid confusion
Often confused with
Self-attention
A mechanism that lets each token reweight information from available tokens in the same sequence, including itself, before the next layer processes it.
Query, key and value
Three vectors produced by learned projections represent matching needs, matching features, and information that attention may pass onward.
Attention score
A number comparing one token’s query with a candidate token’s key during an attention step.
Common misconceptions
Common misconception: Attention means the model looks at only one word.
Correction: Usually it combines information from many tokens with different weights. The important idea is graded influence, not a single hard choice.
Common misconception: Attention shows what the model truly understood or why it answered.
Correction: Attention weights can be informative, but they are not a complete or universally accepted explanation of model behaviour. Other parts of the model also shape the output.
Common misconception: Attention and self-attention are the same term.
Correction: Attention is the broader mechanism. Self-attention is one case where tokens attend to other tokens in the same sequence.
Quick check
What does attention mainly produce inside a transformer?
Attention calculates relative weights and uses them to combine information from tokens. It does not create a human-written rule, store all training data for the task, or provide a complete explanation of model behaviour.
Go deeper
Go deeper
The simple version
It is a way for a model to focus more on some parts of the text and less on others when updating each part.
The technical version
Given token vectors, the mechanism computes relevance weights between positions and uses those weights to form weighted combinations of value vectors, producing context-dependent representations. In transformers, this is often implemented through query, key and value projections.
Reviewed external resources
EncyclopaediaAttention (machine learning) (opens in a new tab)Wikipedia
This encyclopaedia-style page gives a broad overview of attention in machine learning and suits readers who want definitions and terminology.
Intermediatelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
This research paper introduces the transformer architecture that made attention central and suits technical readers who want the original source.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
This article walks through the transformer and attention with clear diagrams and suits readers who want a practical conceptual explanation.
Intermediatelink checked 2026-07-31
VideoAttention in transformers, step by step (opens in a new tab)3Blue1Brown
This video gives an intuitive step-by-step visual explanation of attention in transformers and suits visual learners and non-specialists.
Deeperlink checked 2026-07-31
VideoVideo explanations: Attention (opens in a new tab)YouTube search
This search results page helps readers find multiple video explanations of attention at different levels and suits people comparing teaching styles.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Attention (opens in a new tab)Reddit search
This search results page lets readers browse community questions and informal explanations about attention and suits readers who want to see common points of confusion.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Attention creates context-dependent representations by weighting and combining information from different input positions. The weighting is recalculated for each position.