Learn AI /Transformers and Attention /Multi-head attention

Multi-head attention

Also called: attention heads.

Detail

In five seconds

What it is

A transformer mechanism that runs several attention patterns in parallel so the model can combine different kinds of relationships between tokens.

In plain words

Instead of looking at context through one attention pattern, the model uses several in parallel and then combines them.

An everyday example

A team is reviewing the sentence: “Amina spoke to Sofia after she finished the report.” One reader looks for the nearest possible person, another checks who was likely to write the report, and another looks at the wider paragraph for clues about the topic. Each reader marks a different likely interpretation and explains which words matter most. The team leader then combines those notes into one final summary of what “she” probably refers to.

Where the story stops working

In the analogy, separate people deliberately choose explicit strategies and can explain them in words. Attention heads are not people, and their patterns come from training rather than conscious reasoning. Also, real heads work with dense vectors and numeric weights, not tidy verbal clues, and many heads do not map neatly to one human-interpretable role.

Remember thisMulti-head attention learns several token-relationship patterns in parallel, combines their outputs, and gives later layers a richer representation to work with.

Understand it

Why it matters

Multi-head attention is one reason transformer models can handle complex language patterns at scale. For a non-specialist, the practical value is this: it helps explain why modern models can connect a word with several relevant parts of a sentence or document at the same time. That matters when you judge system strengths, limits and risks. A model may capture multiple kinds of context in parallel, but that does not guarantee correctness, faithfulness to sources or stable reasoning. Understanding this term helps you ask better questions about model design, performance, compute cost and why a system may succeed on one language task but still fail on another.

How multi-head attention combines several parallel attention patterns

Show that one input sequence is projected into several parallel attention heads, each computes its own weighted mix, and the results are merged into one output per token.

01
Input sequence
Start with token vectors for the same sequence, one vector per token position.
Draw 4 to 6 equal boxes in a horizontal row labelled with sample tokens such as 'Amina', 'spoke', 'to', 'Sofia', 'she'. Under each, show a small stacked bar icon to indicate a vector.
02
Split into heads
Create several parallel learned views of the same inputs. Each head gets its own query, key and value projections.
From the input row, draw lines into three or four parallel lanes labelled 'Head 1', 'Head 2', 'Head 3', 'Head 4'. In each lane, place three small boxes labelled Q, K and V.
03
Attention inside each head
Within each head, queries are compared with keys to produce weights, and those weights are used to mix the values.
In each head lane, show a small matrix or dot grid labelled 'scores → weights', followed by arrows to a box labelled 'weighted sum of V'. Use different accent colours for different heads.
04
Join head outputs
The outputs from all heads are concatenated, meaning placed side by side into one larger vector.
Draw each head output as a narrow vertical block. Merge them into one wider block labelled 'concatenate'.
05
Final projection
A learned output projection mixes the concatenated information into the final output vector for each token.
Draw an arrow from the concatenated block to a box labelled 'output projection', then to a final row of output token vectors aligned with the original token positions.

The diagram reads left to right. A row of input token vectors splits into several parallel heads. Each head shows query, key and value projections, attention weighting, and a head output. The head outputs are then joined and passed through a final projection to form the output sequence.

On hover or tap, highlight one head at a time and show a short caption such as 'may emphasise nearby agreement' or 'may emphasise longer-range reference', while noting that these are examples, not fixed rules.

Describe this without the diagram

A left-to-right flow diagram explains multi-head attention. First, a sequence of token vectors is shown as a row of labelled boxes. Second, the row splits into four parallel lanes called heads. In each lane, the same input is projected into queries, keys and values. Third, each head computes scores from queries and keys, turns them into weights, and uses those weights to combine the values into one head output. Fourth, the four head outputs are placed side by side in a concatenation step. Fifth, a final output projection turns the combined result into the output sequence of vectors. The key idea is that several attention calculations happen in parallel, not one after another, and their results are merged.

With reduced motion: A static five-stage diagram with no animation shows the same sequence: input token vectors, parallel heads with Q K V projections, per-head weighting and value mixing, concatenation of head outputs, and a final output projection.

In a transformer, tokens are represented by vectors, which are lists of numbers representing token features. In later layers, these vectors may already include contextual information. Multi-head attention takes those token vectors as input and creates several parallel versions of the attention calculation. Each parallel part is called a head.

Inside each head, the model builds queries, keys and values. These are learned projections: new numeric views of the same input. A query asks what a token is looking for, a key describes what each token offers, and a value carries the information that may be passed along. The head compares queries with keys to produce attention scores, turns those scores into weights, and then mixes the values accordingly.

Because there are several heads, the model does not rely on one single pattern of relevance. One head may focus on a nearby word, another on a long-range reference, and another on part of a phrase or sentence structure. This does not mean each head always has a clean human meaning. It means the architecture gives the model several parallel channels for relating tokens.

The outputs from all heads are joined together and passed through another learned projection to produce one combined output vector per token. What comes out is still a sequence of vectors, but now each token representation has gathered information from multiple attention patterns at once.

Self-attention and multi-head attention describe different aspects. Self-attention means tokens attend to tokens in the same sequence. Multi-head attention means several attention calculations run in parallel and are combined. It can be used for self-attention or for attention between different inputs.

How “Several readers resolving a pronoun in one sentence” maps onto the real thing

In the story	In the AI system
each reader using a different strategy	each attention head learning a different pattern of relevance
the words each reader highlights	attention weights over tokens
the notes from each reader	the weighted value vectors produced by each head
the team leader combining the notes	concatenating head outputs and applying a final learned projection

A worked example

A reporting officer tests an approved model on the sentence: “The delegation sent the revised text to the secretariat, which circulated it before the informals.”

Starting from: Tokens representing the sentence are passed into one transformer layer.

For each attention head, separate learned projections turn the token representations into head-specific queries, keys and values.
Within each head, query-key similarity produces attention scores. After normalisation, these become weights used to combine the value vectors.
One head may give “which” greater weight towards “secretariat”, while another may connect “it” with “revised text”. These are learned patterns, not fixed grammatical assignments.
The head outputs are concatenated and passed through an output projection, producing updated representations that combine the parallel patterns.
Later layers can use those representations when generating or checking the note for the file.

Result: The layer can represent several relationships at once, including who circulated the text and what was circulated. The attention weights show how information was combined, but do not by themselves establish that either reference was resolved correctly.

In a United Nations workplace

A reporting officer in a documentation and library service is preparing the note for the file after a bilateral, due by close of business. In an approved drafting tool, separate attention heads may connect a pronoun to the relevant delegation, link a shortened title to its earlier full form, and relate a requested follow-up to the paragraph where it was first raised. The heads process these token relationships in parallel, and their outputs are combined before the next layer.

The mechanism helps the model maintain several possible relationships across the note at once. Individual heads are not assigned fixed jobs, and an attention pattern is not proof that the model has interpreted the exchange correctly. The reporting officer still checks every attribution, commitment and follow-up against the record.

Before you try thisA bilateral note may contain confidential, personal or politically sensitive material. It must not be entered into any tool that has not been approved for that information. Where an approved tool is available, apply the office's handling rules and remove details the task does not require.

Where you may hear it

A vendor demonstration to an information management working group
A technical review before an artificial intelligence tool is cleared for office use
A briefing to an information and communications technology governance board
A model evaluation meeting after inconsistent document summaries

Avoid confusion

Often confused with

Common misconceptions

Common misconception: Each head learns one clear linguistic job, such as grammar, pronoun resolution or translation alignment.
Correction: Some heads do show patterns humans can describe, but many do not have one clean role. Heads are learned numeric mechanisms, and several heads may share, split or change functions across layers and tasks.
Common misconception: More heads always mean a better model.
Correction: Extra heads can increase capacity, but only up to a point and with trade-offs in parameters, memory use and speed. Performance depends on the whole design, including head size, number of layers, training data and optimisation.
Common misconception: Multi-head attention and self-attention are the same term.
Correction: Self-attention describes the operation where tokens attend to other tokens in the same sequence. Multi-head attention describes a parallel structure often used inside that operation, where several attention calculations run at once and are then combined.

Quick check

What does multi-head attention mainly add to a single attention calculation?

Go deeper

The simple version

Instead of looking at context through one attention pattern, the model uses several in parallel and then combines them.

The technical version

Given an input sequence of token representations, the model applies separate learned linear projections for each head to produce head-specific queries, keys and values. Within each head, attention scores are computed from query-key similarity, normalised into weights, and used to form a weighted sum of the value vectors. The outputs of all heads are concatenated and passed through an output projection. This lets the layer represent different relational patterns in parallel within a fixed model dimension, though the exact role of any one head may be hard to interpret.

Reviewed external resources

EncyclopaediaWikipedia search: Multi-head attention (opens in a new tab)Wikipedia
This Wikipedia link is a search results page, so it helps readers find related articles and terminology rather than giving one settled explanation; it suits readers who want a broad starting point.
search results, not one pagelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
This paper introduces the transformer architecture that popularised multi-head attention, and it suits readers who want the original research description.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
This article gives an intuitive visual walk-through of transformer parts, including multi-head attention, and it suits readers who learn best from diagrams and plain language.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Multi-head attention (opens in a new tab)YouTube search
This YouTube link is a search results page, so it helps readers compare different video explainers and teaching styles; it suits readers who prefer spoken and visual explanations.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Multi-head attention (opens in a new tab)Reddit search
This Reddit link is a search results page, so it can show common questions and informal explanations from the community; it suits readers who want to see where people get confused, but claims should be checked carefully.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Multi-head attention learns several token-relationship patterns in parallel, combines their outputs, and gives later layers a richer representation to work with.