Learn AI /Transformers and Attention /Vector and vector space
Vector and vector space
- Intermediate
- Transformers and Attention
- Technical readers
Also called: vector, vector space, vectors.
Back to the milestone you came from
In five seconds
What it is
A vector is an ordered list of numbers, and a vector space is a set of same-length vectors that is closed under addition and scaling.
In plain words
A vector is one item written as many numbers. A vector space is the shared numeric landscape where many such items can be compared, combined and moved around by the model.
An everyday example
Imagine each movement is written as an ordered pair: [3, 2] means three tiles to the right and two forward. Combining [3, 2] with [-1, 4] gives [2, 6], still an ordered pair. Doubling [3, 2] gives [6, 4], also an ordered pair. One instruction is a vector; the set of all such compatible pairs, including fractional and negative movements, is the vector space.
A real noticeboard usually has two visible dimensions, while model vectors often have hundreds or thousands. Also, vectors in a transformer are not manually pinned by people and do not carry fixed human-readable meanings per coordinate. Their numbers are learned and then repeatedly transformed, so the space changes through the network rather than staying like one static board.
Remember thisA vector is one ordered numeric representation. A vector space supplies the compatible structure in which models can add, scale, compare and transform such representations.
Understand it
Understand it
Why it matters
If you hear that a model “represents” words or documents numerically, vectors are usually the mechanism. Understanding this helps you ask better questions about similarity, bias, retrieval, clustering and attention. It also helps you separate three different ideas: one vector, the larger vector space, and a learned embedding. For decision-making, this matters because many system behaviours depend on how items are positioned and transformed in that space. Similar-looking items may be grouped together; ambiguous items may shift with context; and errors can come from poor representations, not only from the final output stage. You do not need the full mathematics to recognise what the model is doing and where caution is needed.
From token to vector, then through a shared vector space
Show the difference between one vector and the vector space it lives in, and explain how transformers compare and transform vectors during processing.
01
1. Input token
Start with a token such as “bank” or “they”. On its own, it is text, not yet usable for model calculations.
Draw a rounded rectangle labelled “token: they” on the left, with a right-pointing arrow.
02
2. Convert to a vector
The token is mapped to an ordered list of numbers. That list is one vector.
Show the token leading to a small card labelled “[0.4, -0.2, 1.1, …]” with a caption “one vector”.
03
3. Place in a vector space
The vector belongs to a shared coordinate system where other token vectors also live. That makes comparison and combination possible.
Place several dots inside a large panel labelled “vector space”; highlight one dot for the current token and a few nearby dots for related tokens.
04
4. Compare and mix
The model compares vectors and uses weighted combinations to bring in relevant context from other tokens.
Add arrows from several dots toward the highlighted dot, with varying line thicknesses to suggest stronger or weaker influence.
05
5. Output a new vector
After transformation, the model produces an updated vector that carries contextual information for the next layer or for prediction.
Show a new vector card on the right labelled “[−0.1, 0.8, 0.3, …]” with a note “updated by context”.
A left-to-right flow diagram. Tokens enter, become vectors, are shown as points inside one labelled vector space, then pass through comparison and transformation stages to produce new vectors. Read it as: input text becomes numeric coordinates, shared space makes them comparable, model operations create updated vectors for the next step.
Hover or tap each stage to reveal a short caption and highlight how one token’s vector changes after receiving context from nearby or strongly related vectors.
Describe this without the diagram
The diagram is a five-step flow from left to right. First, a token labelled “they” appears as plain text. Second, it is converted into one vector, shown as a list of numbers. Third, that vector is placed inside a large shared area labelled vector space, where several other token vectors appear as nearby or distant points. Fourth, arrows from other points flow toward the selected point, showing that the model compares vectors and mixes information from relevant tokens with different strengths. Fifth, a new list of numbers appears on the right as an updated vector shaped by context. The main lesson is that a vector is one numeric representation, while a vector space is the common system that lets many vectors be compared and transformed.
With reduced motion: A static five-panel diagram with the same stages: token, one vector, vector space with several points, comparison arrows of different thickness, and an updated vector on the right. No animation; emphasis is shown by colour contrast and labels.
In modern language models, many things are turned into vectors: words, parts of words, positions, or intermediate states inside the model. A vector is an ordered list of numbers, such as [0.2, -1.1, 0.7]. Each number can be treated like one coordinate. The whole list acts as one compact description that a computer can compare or transform.
A vector space is the larger set those vectors belong to. It gives the rules for combining them. In a vector space, you can add two vectors and multiply a vector by a number, and the result still stays in the same space. In plain terms, vectors of the same dimension can be combined using consistent operations.
What goes in? Usually text tokens or other inputs are first converted into vectors. What happens next? The model moves those vectors through many calculations, such as comparing, mixing, projecting or otherwise transforming them with matrix operations. A matrix is a table of numbers used to transform one vector into another. What comes out? New vectors that capture useful patterns for the next step, such as attention or prediction.
This is often confused with an embedding. An embedding is one particular kind of vector representation learned so that related items often end up near each other. But not every vector in a transformer is an embedding, and a vector space is not the same thing as one vector. It is also not the same as a physical arrow in school geometry, though that image can help at first.
How “Instructions on a tiled floor” maps onto the real thing
| In the story | In the AI system |
|---|---|
| one card’s many coordinates on the noticeboard | one vector made of many numbers |
| the whole noticeboard layout | the vector space |
| cards placed near similar cards | similar vectors being close in the space |
| checking nearby cards to resolve “they” | using vector comparisons to help attention link related words |
A worked example
A simplified transformer represents the token “clearance” and its position in a briefing note using three-number vectors. Real systems use many more dimensions, but the arithmetic is the same.
Starting from: Token vector: [0.4, -0.2, 0.7]. Position vector: [0.1, 0.3, -0.1].
- Both vectors have three components, so the model can add them component by component: [0.4, -0.2, 0.7] + [0.1, 0.3, -0.1] = [0.5, 0.1, 0.6].
- The result is another three-component vector in the same vector space. This illustrates closure under addition.
- Suppose attention combines value vectors [1, 0, 2] and [0, 2, -1] with weights 0.25 and 0.75. Scaling gives [0.25, 0, 0.5] and [0, 1.5, -0.75].
- Adding the scaled vectors produces [0.25, 1.5, -0.25], again a three-component vector that later layers can process.
Result: Because addition and scaling preserve the vector length, embeddings, positional information and attention outputs remain compatible with subsequent model operations. The numbers are illustrative and should not be read as named semantic features.
In a United Nations workplace
A chief of section in an evaluation office is finalising a handover note for an incoming colleague before close of business. The note records how a proposed search tool represents each paragraph of a cleared evaluation report as a same-length vector, then compares vectors to retrieve passages with similar meaning even when they use different wording.
The vector is the numeric representation of one paragraph. The shared vector space allows the tool to compare, add and transform compatible representations. The same principle underlies token embeddings and hidden states inside a transformer, although those vectors usually have far more dimensions.
Before you try thisOnly cleared reports should enter this workflow. Confidential interview notes, personal data and material identifying participants must not be uploaded to a tool that has not been approved for them. Access controls and human review remain necessary, particularly where retrieved passages could be mistaken for evaluation findings.
Where you may hear it
- A vendor demo during a procurement unit meeting
- An ICT governance board review of a language-model pilot
- A technical debrief in an evaluation office
- A briefing for an inter-agency working group considering semantic search
Avoid confusion
Avoid confusion
Often confused with
Common misconceptions
Common misconception: A vector is just a direction arrow like in school physics.
Correction: That picture can help initially, but in AI a vector is usually treated as an ordered list of numbers in many dimensions, not a literal arrow moving through physical space. The important idea is numeric representation and comparison, not motion.
Common misconception: Vector space means the model stores concepts in neat human-readable dimensions such as gender, place or topic.
Correction: Some dimensions may correlate with patterns people can describe, but most coordinates in a learned space do not have a single clear human label. Meaning is usually distributed across many numbers and changes after transformations in each layer.
Common misconception: Vector and embedding mean exactly the same thing.
Correction: An embedding is a specific learned vector representation, often used near the input or for retrieval tasks. But transformers also create many other vectors internally, such as queries, keys, values and hidden states. The vector space is the broader coordinate system, not one embedding table.
Common misconception: If two vectors are close, the model has found a true real-world similarity.
Correction: Closeness reflects how the system has represented the data for a task, not an objective fact about the world. Similarity can be useful, but it can also reflect training data biases, missing context or design choices.
Quick check
Which statement best distinguishes a vector from a vector space in a transformer?
A vector is one numeric representation, such as a token’s current state. A vector space is the larger set closed under addition and scaling. Models may compare compatible vectors using dot products. Embeddings are one kind of vector, but not the whole story.
Go deeper
Go deeper
The simple version
A vector is one item written as many numbers. A vector space is the shared numeric landscape where many such items can be compared, combined and moved around by the model.
The technical version
In transformer systems, tokens and intermediate states are represented as vectors in a high-dimensional vector space, often with dimension d_model. A vector is an element of that space: an ordered numeric array. The space matters because model operations assume closure under addition and scalar multiplication, which allows weighted sums and linear transformations to stay in compatible form. Input embeddings, positional information and hidden states are vectors; matrices project them into new vectors such as queries, keys and values. Attention then compares vectors, often through dot products, and uses the resulting weights to build weighted combinations of value vectors. Distances or angles between vectors can also support similarity measures such as cosine similarity, but those measures depend on the chosen representation and task. In short, a vector is one point or direction in the space; a vector space is the structured set that makes those operations mathematically and computationally consistent.
Reviewed external resources
EncyclopaediaVector space (opens in a new tab)Wikipedia
Wikipedia gives a general reference overview of vector spaces and suits readers who want the formal concept with broad background.
Intermediatelink checked 2026-07-31
ArticleAttention Is All You Need (opens in a new tab)Vaswani et al., 2017
The Vaswani et al. paper shows where vectors are used inside the original transformer design and suits technical readers who want the source architecture.
Deeperlink checked 2026-07-31
ArticleThe Illustrated Transformer (opens in a new tab)Jay Alammar
The Illustrated Transformer gives an intuitive visual walkthrough of how vectors move through transformer components and suits readers who prefer diagrams over formal maths.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Vector and vector space (opens in a new tab)YouTube search
This YouTube search results page helps readers find different video explanations of vectors and vector spaces, and it suits people who learn best by listening and watching.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Vector and vector space (opens in a new tab)Reddit search
This Reddit search results page helps readers see common questions, confusions and informal explanations about vectors and vector spaces, and it suits readers who want community discussion rather than a single article.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
A vector is one ordered numeric representation. A vector space supplies the compatible structure in which models can add, scale, compare and transform such representations.