Learn AI /Transformers and Attention /Transformer

Transformer

Also called: transformer architecture.

Detail

In five seconds

What it is

A neural network design that processes many parts of a sequence in parallel by weighting which parts matter most to each other.

In plain words

A transformer is a model design that lets each input part directly combine clues from other permitted parts, rather than carrying context step by step.

An everyday example

A city office receives a large printed satellite image divided into many square tiles. Four staff members sit around a table and inspect all the tiles at the same time, but each person looks for a different clue, such as cracks, flooding patterns, shadows, or nearby traffic lines. When they judge one tile, they keep glancing at other tiles to see what gives context, such as whether a dark patch is a pothole or just a tree shadow. At the end, the team combines all those notes into one clearer judgement for each tile and then prepares a summary map.

Where the story stops working

People inspect tiles consciously and may use outside knowledge about roads and weather. A transformer does not inspect images like a person and does not literally look around a table; it computes numerical weights from training patterns, often across many more layers and dimensions than the analogy suggests.

Remember thisA transformer builds context by letting each position weight relevant permitted positions in parallel, then refining those representations through stacked attention and feed-forward layers.

Understand it

Why it matters

If you choose, buy, govern, or use AI systems, this term helps you judge what current tools are good at, why they scale well, and where they may still fail without human checking.

How a transformer processes a sequence

After viewing this, a reader should be able to say that a transformer turns inputs into vectors, adds order, uses self-attention across all parts, repeats this in layers, and produces task-specific outputs.

01
1. Input parts
Start with a sequence, such as words in a sentence or image patches.
Draw 5 small rounded rectangles in a row labelled Token 1 to Token 5 or Patch 1 to Patch 5.
02
2. Turn into vectors with order
Each part becomes a numeric representation, and position information is added so order is not lost.
Draw an arrow to a second row of 5 circles or bars labelled vector plus position.
03
3. Self-attention
Each part compares itself with all the others and gives different weights to the most relevant ones.
Inside a large box, place the 5 vectors in a row and draw thin lines connecting every item to every other item, with 2 or 3 darker lines to suggest stronger weights.
04
4. Repeat in layers
Several layers repeat attention and further transformation, gradually refining the representation.
Draw 3 stacked boxes labelled Layer 1, Layer 2, Layer 3, with arrows upward through them.
05
5. Task output
A final component turns the refined representation into a result such as next-word probabilities, a translation, or a class label.
Draw an output box branching to three small example labels: next word, translated text, class label.

Show one horizontal flow from input tokens to output. Each stage is a labelled box with arrows: input parts, embeddings plus positions, self-attention, repeated layers, output head. Include faint cross-links inside the self-attention box to show each part can connect to every other part.

On hover or tap, highlight one input part and emphasise the links it gives high weight to in the self-attention stage; include a toggle to switch labels between text tokens and image patches.

Describe this without the diagram

The diagram is a left-to-right flow with five stages. First, a sequence is split into parts such as words or image patches. Second, each part is converted into a numeric vector and given position information. Third, in self-attention, every part can connect to every other part, and some connections are stronger because they are more relevant. Fourth, this attention-and-transformation process is repeated through several layers to refine the representation. Fifth, a task-specific output head produces a result such as the next word, a translation, or a class label. The main lesson is that the transformer processes all parts together and lets each part use context from the whole sequence.

With reduced motion: Show the same five stages as a static left-to-right diagram. Use darker fixed lines in the self-attention box to indicate stronger connections instead of animated highlighting.

A transformer takes a sequence as input, such as words in a sentence or small patches from an image. Each part is first turned into numbers called vectors, which let the system compare patterns. Because order matters, the model also adds position information so it can tell the start from the end.

Inside the model, each part looks at other parts and gives them different weights based on relevance. This mechanism is called attention. In self-attention, the sequence compares with itself, so one word can focus on earlier or later words that help clarify its meaning.

The transformer repeats this process across several layers. Each layer mixes information, refines it, and passes it on. Many transformers use multi-head attention, which means several attention patterns run at once so the model can track different kinds of relationships, such as grammar, topic, or long-range links.

What comes out depends on the task. The output might be a translated sentence, a prediction of the next word, or a category label for an image. A transformer is the architecture, not the final product: a large language model is one kind of system built with this architecture.

How “A road-damage review table” maps onto the real thing

In the story	In the AI system
image tiles on the table	input pieces in a sequence, such as tokens or image patches
staff glancing at other tiles for context	self-attention weighting other input parts by relevance
different staff looking for different clues	multi-head attention capturing different relationships
combined notes after each review round	layer outputs that refine representations step by step

A worked example

Predicting the next token in: "After the final informals, the draft was adopted by ___".

Starting from: After | the | final | informals | , | the | draft | was | adopted | by

Each token is mapped to an embedding, with positional information added so the model can distinguish both the words and their order.
Through masked self-attention, the representation at the final input position can draw on permitted earlier tokens. Different attention heads may emphasise "adopted by", "draft" and "final informals".
Stacked attention and feed-forward layers refine these context-aware representations in parallel.
The output layer scores possible next tokens. Given this context, "consensus" may rank above less fitting alternatives.

Result: The model selects or samples from the ranked candidates, producing a continuation such as: "After the final informals, the draft was adopted by consensus."

In a United Nations workplace

A reporting officer in a liaison office is finishing a handover note for an incoming colleague before close of business. Using an internal tool approved for non-sensitive material, the officer submits a list of recurring deadlines and asks it to group related items under reporting, clearance and meetings.

The tool's transformer can process the list in parallel and use self-attention to weight connections between words in different entries. This helps it recognise, for example, that several references to circulation, comments and sign-off belong under clearance even when they are phrased differently.

Before you try thisA handover note may contain personal, confidential or security-sensitive details. Such material must not be entered into a tool unless that tool is explicitly approved for that category of information, because prompts or outputs may be retained or exposed outside authorised controls. The officer must also check every grouping and omission.

Where you may hear it

At a vendor demo during a procurement meeting
In an ICT governance board review of an AI proposal
During a Headquarters briefing on machine translation
At an inter-agency working group discussing document classification

Avoid confusion

Often confused with

Common misconceptions

Common misconception: A transformer is the same thing as a large language model.
Correction: A transformer is an architecture, or design pattern. A large language model is a specific system trained on large amounts of text, often using that architecture.
Common misconception: It reads one word at a time like older sequence models.
Correction: Transformers were notable because they can process many input parts in parallel during training, while still linking them through attention.
Common misconception: Attention means the system truly understands which words matter.
Correction: Attention is a computed weighting mechanism inside the model. It can be useful for context, but it is not the same as human understanding or a full explanation of model behaviour.
Common misconception: Transformers are only for text.
Correction: They became famous in language tasks, but related designs are also used for images, audio, and multimodal systems because the core idea works on many kinds of sequences.

Quick check

Which statement best describes why transformers became so important?

Go deeper

The simple version

A transformer is a model design that lets each input part directly combine clues from other permitted parts, rather than carrying context step by step.

The technical version

A transformer maps input tokens or patches to embeddings with positional information, then applies stacked layers of self-attention and feed-forward transformations. Each position builds a context-aware representation from permitted sequence positions, often using multi-head attention and parallel computation.

Reviewed external resources

EncyclopaediaTransformer (opens in a new tab)Wikipedia
This Wikipedia page gives a broad reference overview of the architecture, terminology, and history, and suits readers who want a general starting point.
Intermediatelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
This paper presents the original transformer proposal and suits readers who want the source text and are comfortable with formal research writing.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
This article explains the components with diagrams and plain language, and suits readers who want a step-by-step walkthrough.
Intermediatelink checked 2026-07-31
VideoTransformers, the tech behind LLMs (opens in a new tab)3Blue1Brown
This video gives an intuitive visual explanation of how transformers work and suits visual learners who want the big picture before technical details.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Transformer (opens in a new tab)YouTube search
This is a search results page for many video explanations, not one single source, and it suits readers who want to compare teaching styles and difficulty levels.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Transformer (opens in a new tab)Reddit search
This is a search results page for community discussions, not one single source, and it suits readers who want informal questions, answers, and practical confusion points.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

A transformer builds context by letting each position weight relevant permitted positions in parallel, then refining those representations through stacked attention and feed-forward layers.