presentsUNAIVERSE
The UN AI timeline

    Learn AI /Transformers and Attention /Embedding

    Embedding

    Also called: embeddings, vector representation.

    Detail

    In five seconds

    What it is

    A list of numbers that places a token or other item in a mathematical space so a model can compare patterns, relationships and context.

    In plain words

    An embedding is a way of turning a token or other item into numbers so a computer system can compare it with other items and use it inside a model.

    An everyday example

    Imagine arranging thousands of short messages in a space with many dimensions. Each message receives a set of coordinates based on patterns in how its words are used. Messages used in similar ways tend to receive nearby coordinates, even when they share few exact words. A new query also receives coordinates, so nearby messages can be found. The coordinates are not a plain-language summary. They are a practical mathematical address for comparison.

    Where the story stops working

    Real embeddings are not drawn by a person on a wall map, and the space usually has many dimensions, not two. Also, closeness in embedding space does not mean two items are identical or interchangeable. In language models, the embedding for a token is only the starting representation; later layers change it based on context, so a single fixed pin does not capture the full process.

    Remember thisAn embedding is a fixed-size list of numbers representing a token or other item. Token embeddings feed transformer layers, which produce contextual representations. Document embeddings can support similarity search.

    Understand it

    Understand it

    Why it matters

    Embeddings are one of the main reasons modern AI systems can compare text, search by meaning-like patterns, cluster similar items and feed language into transformers. If you use AI tools for search, classification, recommendations or document matching, embeddings often sit underneath. For a non-specialist, the key decision point is this: a system may treat items as similar because of learned statistical patterns in data, not because a human checked that they truly mean the same thing. That affects quality, bias, retrieval results and whether human review is needed.

    How embeddings place tokens in a space

    After using this explainer, a reader should be able to say that an embedding converts a token into numbers so the model can place it near or far from other tokens and then use those positions for comparison and attention.

    A scatter plot of short phrases. Phrases with similar meaning are drawn close together, even when they share no words.

      The explainer shows a small set of tokens, their embedding vectors, and a 2D projection of their positions. Readers select tokens, switch between isolated tokens and short phrases, and see how closeness changes what looks similar. A side panel shows the numeric vector and reminds readers that the true space has many dimensions, while the chart is a simplified projection.

      Readers can click tokens, hover points for labels, toggle between token-only and in-phrase views, and switch on an option that displays nearest neighbours and approximate distances.

      Describe this without the diagram

      This alternative presents the same lesson in text. First, the reader chooses a token from a list. The interface announces that the token has been converted into a vector, which is a list of numbers. Next, it reports several nearby tokens and explains that nearby means the model has learned similar usage patterns, not necessarily the same dictionary meaning. Then it describes a simple pipeline: text is split into tokens, tokens become embeddings, and transformer layers further modify those representations using context. Finally, in phrase mode, it explains that the starting embedding for a token is updated by surrounding words, so later representations are context-sensitive.

      With reduced motion: Instead of animated movement, the reader sees two static panels. The first shows selected tokens with their vectors and a fixed projected scatter plot with nearest neighbours outlined. The second shows a static before-and-after comparison: tokenisation, initial embedding, and a later contextual representation inside a phrase. All changes appear instantly without motion.

      An embedding turns something discrete, such as a word piece, sentence or image label, into a list of numbers. That list is called a vector. The input might be the token for “rain”, a whole sentence, or another item the system needs to represent. The output is not a definition in words. It is a numeric representation the model can work with.

      Some embedding systems are trained so that a chosen similarity measure places certain related items closer together. This is not guaranteed for every embedding or every kind of similarity. For example, tokens used in similar contexts may have similar embeddings. In modern language models, each token first gets an input embedding, then the transformer processes those vectors using attention.

      What happens in between depends on the system. Usually, the model learns these numbers during training by adjusting them so the overall task works better, such as predicting the next token. The embedding does not contain a perfect human meaning of a word. It captures patterns from the training data and the model’s objective.

      This term is often confused with tokenisation and vector databases. Tokenisation breaks text into units such as words or word pieces. Embedding comes after that and maps those units to vectors. A vector database stores and searches vectors. An embedding is the representation itself, not the database.

      How “Coordinates on a map with no street names” maps onto the real thing

      In the storyIn the AI system
      A pin for each villageAn embedding vector for a token or other item
      Pins placed near villages with similar rainfall behaviourSimilar embeddings placed close in vector space
      Recent rainfall pattern used to position a new villageTraining or an encoder producing a representation from data
      Using nearby pins to find similar planting adviceUsing embeddings for similarity, retrieval or later model processing

      A worked example

      Before first reading of a background note on a text-classification prototype, a reporting officer at a regional bureau checks the technical example. To avoid using operational material, the note uses the fictional sentence: “Heavy rain damaged roads.”

      Starting from: Illustrative tokens: [Heavy] [rain] [damaged] [roads]. An actual tokeniser may divide the text differently.

      1. The tokeniser assigns each token an id. Each id selects one row from a learned embedding matrix.
      2. For illustration, “rain” might select a vector shortened here to [0.42, -0.18, 1.07, 0.09]. Real embedding vectors usually contain many more values.
      3. The lookup produces one fixed-size vector for each token. Positional information is added or otherwise incorporated so that the model can account for order.
      4. The vectors enter the transformer layers. Attention combines information across the sentence, producing a contextual representation of “rain” that differs from its initial lookup embedding.

      Result: The embedding lookup supplies numeric starting representations. Transformer layers then contextualise them for tasks such as predicting another token or classifying the sentence. The initial token embedding is therefore an input to the model’s processing, not its final output.

      In a United Nations workplace

      After a bilateral on a proposed document-search tool, an ICT officer in a Resident Coordinator's Office drafts the note for the file before close of business. The practical question is whether staff could retrieve a cleared briefing note when their search uses different wording from the note itself.

      The tool can convert the query and approved documents into embeddings, then compare their positions in a mathematical space. This can retrieve passages with similar patterns of meaning despite limited keyword overlap. The officer records that similarity supports retrieval, but does not establish that a result is correct, complete or suitable for the front office.

      Before you try thisThe officer must not upload records of the bilateral, personal data, security-sensitive material or other restricted content to a service not approved for it. Embeddings, vector stores and logs may retain or reveal information, so deleting the source file alone may not remove the risk. Retrieval rankings also require testing and human review.

      Where you may hear it

      • At a vendor demonstration to an ICT governance board, while testing semantic search results
      • During first reading of a concept note for a document-search pilot
      • In a procurement unit meeting reviewing claims about vector search and document clustering
      • During clearance of a background note on how a retrieval tool ranks passages

      Avoid confusion

      Avoid confusion

      Often confused with

      Common misconceptions

      • An embedding is just another word for tokenisation.

        They are different stages. Tokenisation splits text into units such as words or word pieces. Embedding converts each unit into a numeric vector so the model can process it. You usually need tokenisation before token embeddings, but they are not the same thing.

      • An embedding stores the exact meaning of a word.

        It is better to say it captures learned usage patterns relevant to training. A vector may place “bank” near finance-related terms in some contexts, but it does not guarantee a complete or stable human meaning. Meaning-like behaviour depends on training data, objective and later context processing.

      • If two items are close in embedding space, they mean the same thing.

        Closeness only suggests the model has learned similar patterns of use or related features. Two items can be close because they appear in similar contexts, belong to the same topic, or share style. Human judgement is still needed for high-stakes interpretation.

      • A vector database creates embeddings.

        Usually it stores and searches vectors that were created elsewhere by an embedding model. Some tools bundle both steps together, which causes confusion, but the database and the embedding are different components.

      Quick check

      Which statement best describes an embedding in a language model?

      Go deeper

      Go deeper

      The simple version

      An embedding is a way of turning a token or other item into numbers so a computer system can compare it with other items and use it inside a model.

      The technical version

      In transformer-based systems, an embedding is typically a dense vector representation assigned to each token id, often via a learned embedding matrix. The token index selects a row from that matrix, producing a continuous-valued vector of fixed size. Positional information is then added or otherwise incorporated so the model can handle order. These vectors become the input to attention layers. More broadly, embeddings can also represent sentences, documents or images, produced either by lookup tables or by encoder networks. Their geometry supports operations such as nearest-neighbour search, clustering and similarity scoring, but the geometry reflects training data and objectives rather than a guaranteed human semantic theory.

      Reviewed external resources

      • EncyclopaediaWord embeddingWikipedia

        This overview gives a broad reference-style introduction to word embeddings, which suits readers who want a quick general background.

        Intermediatelink checked 2026-07-31

      • ArticleAttention Is All You NeedVaswani et al., 2017

        This paper introduces the transformer architecture that uses embeddings as model inputs, which suits technical readers who want the original source.

        Deeperlink checked 2026-07-31

      • ArticleThe Illustrated TransformerJay Alammar

        This article explains how embeddings fit into transformers with diagrams and plain language, which suits readers who want intuition before formal detail.

        Intermediatelink checked 2026-07-31

      • InteractiveEmbedding ProjectorTensorFlow

        This interactive tool lets readers explore how points cluster in embedding space, which suits visual learners and anyone who wants hands-on intuition.

        Starterlink checked 2026-07-31

      • VideoVideo explanations: EmbeddingYouTube search

        This is a search results page for many video explanations on embeddings, which suits readers who prefer to compare different teaching styles.

        search results, not one pagelink checked 2026-07-31

      • DiscussionCommunity discussion: EmbeddingReddit search

        This is a search results page for community discussions about embeddings, which suits readers who want to see common questions, confusions and informal explanations.

        search results, not one pagelink checked 2026-07-31

      Where this came from

      Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

      The one thing to remember

      An embedding is a fixed-size list of numbers representing a token or other item. Token embeddings feed transformer layers, which produce contextual representations. Document embeddings can support similarity search.