Learn AI /Transformers and Attention /Attention score
Attention score
- Deeper
- Transformers and Attention
- Technical readers
Also called: attention weight.
Back to the milestone you came from
In five seconds
What it is
A number comparing one token’s query with a candidate token’s key during an attention step.
In plain words
It is a relevance number. It tells the model how much one token should draw information from another token during one attention step.
An everyday example
A decorator compares a target colour card with the label on each paint tin and gives every tin a match mark. Those marks are like attention scores: they indicate relative compatibility but are not yet mixing proportions. The decorator converts the marks into shares that add up to one, then takes that share of paint from each tin and blends the contents. The labels were used for comparison, while the paint inside the tins supplied what was actually mixed.
A librarian can read meanings and use world knowledge. Attention scores do not work that way. They are computed from learned numerical representations, not from a human-like understanding of books or topics. Also, in a transformer many tokens are scored in parallel, and the score is usually turned into a normalised weight before values are combined.
Remember thisAn attention score is a raw query-key comparison for one candidate position. After masking and normalisation, it helps determine how much that position’s value vector contributes to the head’s output.
Understand it
Understand it
Why it matters
If you read model cards, technical reports or debugging notes, you will often see attention scores mentioned as if they explain what a model is doing. They matter because, after normalisation, they help determine which value vectors contribute more to each output representation. That affects translation, summarisation, retrieval and many other tasks. But people also over-interpret them. Within one query and head, a higher allowed score usually leads to a larger attention weight. It is not proof of meaning, causation or explanation by itself. Knowing this helps you ask better questions about reliability, traceability and limits.
How an attention score is produced and used
After viewing, the reader should be able to say that an attention score compares one token’s query with other tokens’ keys, then helps weight value vectors in the output.
01
1. Choose a focus token
Pick the token whose new representation is being updated in this attention step.
Draw one highlighted token box on the left labelled 'focus token'. Under it, show a small chip labelled 'query q'.
02
2. Compare with other tokens
Match the focus token’s query against the key of each candidate token to get raw relevance scores.
Draw three token boxes on the right, each with a small chip labelled 'key k' and 'value v'. Connect the focus query to each key through a comparison circle. Label outputs 'score 2.1', 'score 0.4', 'score -0.7'.
03
3. Turn scores into usable weights
Convert the raw scores into comparable weights, commonly so they sum to 1 across the candidate tokens.
Add a 'normalise' stage after the three raw scores. Show outputs '0.75', '0.14', '0.11' with a note 'weights sum to 1'.
04
4. Mix the values
Use the weights to scale the candidate value vectors and add them together, producing the output for the focus token.
Multiply each displayed weight by its corresponding value vector, then merge arrows into one final box labelled 'new token representation'.
Show one focus token on the left and three candidate tokens on the right. Arrows carry the focus token’s query to three query–key comparisons, producing three raw scores. A normalisation stage converts them into weights, and a final stage uses those weights to combine the three value vectors into one output vector for the focus token.
Hovering or tapping each candidate token highlights its score, weight and contribution path to the final output vector.
Describe this without the diagram
The diagram explains attention scoring for one focus token. First, one token is selected for update and shown with its query vector. Next, that query is compared with the key vector of each of three other tokens, producing three raw scores: 2.1, 0.4 and minus 0.7. Then a normalisation step turns those into weights of 0.75, 0.14 and 0.11, which sum to 1. Finally, those weights are applied to the three value vectors and combined into one new representation for the focus token. The main lesson is that the score is an intermediate comparison result, and the weighted mixture of values is the output.
With reduced motion: A static flow diagram with four columns: focus token and query, comparisons to three key vectors with raw scores, a normalisation box with three weights summing to 1, and a final merged output vector built from weighted value vectors.
In a transformer, each token such as a word or subword is compared with tokens at allowed positions, which may include its own position. The model computes a score for each pair. That score reflects their compatibility for the current attention calculation.
What goes in are vector representations of tokens. A vector is a list of numbers. For each token, the model creates a query vector, a key vector and a value vector. The query represents what information may be useful at that position. The key represents features available at a candidate position. The attention score comes from comparing a query with keys.
What happens next is important. The raw scores are often converted by softmax into weights that add up to 1 across allowed positions. Those normalised values are called attention weights. They are then used to mix the corresponding value vectors.
What comes out is not the score itself but a new representation for each token. That representation contains a weighted mixture of value vectors from allowed positions. So the attention score is an intermediate compatibility signal that helps determine the mixture.
How “Mixing a target paint colour” maps onto the real thing
| In the story | In the AI system |
|---|---|
| The student’s question | The query vector for the current token |
| Book titles and subject labels | The key vectors of other tokens |
| The quick relevance mark for each book | The attention score |
| Using some books more heavily in the report | Weighting value vectors more strongly in the output |
A worked example
The synthetic test sentence is: “Because the bridge was blocked, the convoy stopped before it.” The focus token is “it”, and the example examines three candidate tokens: “bridge”, “convoy” and “stopped”.
Starting from: Illustrative scaled query-key scores for “it”: bridge 1.8, convoy 0.2, stopped 1.1. Assume that no mask excludes these positions.
- The query vector for “it” is compared with each candidate token’s key vector, producing the raw attention scores 1.8, 0.2 and 1.1.
- Softmax converts those logits into non-negative attention weights of approximately 0.59, 0.12 and 0.29, which sum to 1 across the three allowed positions.
- Each weight multiplies the corresponding value vector for “bridge”, “convoy” or “stopped”.
- The three weighted value vectors are added to form the output for this attention head at the position of “it”.
Result: In this head, “bridge” contributes the largest share of the mixture, but no token is simply selected. The raw score of 1.8 is not itself a probability, and the pattern does not prove that the model has resolved the reference correctly.
In a United Nations workplace
A protection officer joins a specialised agency technical division for the technical evaluation of three bids. Before the evaluation file goes for clearance at close of business, the officer runs the same synthetic sentence through each bidder’s approved demonstration environment. One interface calls the raw query-key comparisons “attention weights”, although they have not yet been normalised.
Distinguishing an attention score from an attention weight lets the officer test whether each bidder is describing the model accurately. The score is a raw logit for one candidate position. Only after masking and normalisation does it determine that position’s share of the weighted value mixture.
Before you try thisBid documents, evaluators’ notes, commercial information and any personal or protection data must not be entered into a vendor demonstration or a tool not approved for that material. Use synthetic text in the authorised environment. Attention scores also show an internal numerical association, not the model’s reasoning or a sufficient explanation of its output.
Where you may hear it
- A vendor demonstration during a technical evaluation
- An information management officer’s review of an attention visualisation
- An ICT governance board discussion on model transparency
- A research briefing in the margins of a technical session
Avoid confusion
Avoid confusion
Often confused with
Weight
A weight is a learned number that scales how strongly one input signal affects the next calculation in a neural network.
Explainability
Explainability means being able to show, in human-usable terms, why an AI system produced a particular output or pattern of outputs.
Common misconceptions
Common misconception: An attention score is the same thing as a model parameter or learned weight.
Correction: Not usually. Model parameters are stored numbers learned during training. An attention score is typically computed fresh for the current input by comparing query and key vectors. After normalisation, it becomes an input-dependent weight used in that moment.
Common misconception: A high attention score proves the model focused on the true cause or meaning.
Correction: It only shows a strong numerical influence in one attention calculation. Models have many layers, many heads and other pathways such as residual connections. So high attention can be informative, but it is not a complete explanation of why an output occurred.
Common misconception: The score itself is the final output of attention.
Correction: The score is an intermediate comparison result. It is usually transformed into attention weights, and those weights are then used to combine value vectors. The final output is a new vector representation, not the score alone.
Quick check
Which statement best describes an attention score in a transformer?
Attention scores are usually computed anew for each input by comparing query and key vectors. They are intermediate values that are often normalised into attention weights, which then shape how value vectors are combined. They are not fixed parameters, final output probabilities or fact-checks.
Go deeper
Go deeper
The simple version
It is a relevance number. It tells the model how much one token should draw information from another token during one attention step.
The technical version
For a given focus token, the model forms a query vector and compares it with key vectors from candidate tokens, commonly using a scaled dot product. The resulting raw attention scores are logits over positions. A normalisation function, usually softmax and sometimes with a mask, converts them into non-negative attention weights that sum to 1 across the allowed positions. Those weights are applied to the corresponding value vectors, and the weighted sum becomes the output for that head. In multi-head attention, separate heads produce separate sets of scores and weighted sums, which are later combined. People often use 'attention score' for the raw compatibility value and 'attention weight' for the normalised value, but usage is not always consistent.
Reviewed external resources
EncyclopaediaWikipedia search: Attention score (opens in a new tab)Wikipedia
The Wikipedia search results page helps readers scan possible articles and related terms, and it suits someone who is still orienting themselves.
search results, not one pagelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
The Vaswani et al. paper gives the original transformer formulation and suits readers who want the canonical technical source.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
The Illustrated Transformer offers an intuitive visual walkthrough of attention mechanisms and suits readers who learn well from diagrams and plain-language explanation.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Attention score (opens in a new tab)YouTube search
The YouTube search results page helps readers find different video styles and levels, and it suits readers who prefer spoken or animated explanations.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Attention score (opens in a new tab)Reddit search
The Reddit search results page shows how practitioners and learners discuss the term informally, and it suits readers who want common questions and points of confusion.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
An attention score is a raw query-key comparison for one candidate position. After masking and normalisation, it helps determine how much that position’s value vector contributes to the head’s output.