presentsUNAIVERSE
The UN AI timeline

    Learn AI /Transformers and Attention /Positional encoding

    Positional encoding

    Also called: position embedding.

    Detail

    In five seconds

    What it is

    A method that supplies sequence-position information so attention can use word order, not just token content.

    In plain words

    Transformers need a way to keep track of order. Positional encoding gives each token a signal for where it appears, so the model can tell the difference between the same words in different positions.

    An everyday example

    A team is summarising meeting notes from Nairobi, Bangkok and Bogotá. They print each note sentence on a separate card, but the cards only show the sentence text. Before sorting the cards into a summary, they stamp each card with a line number so they know what came earlier and what came later. Now two cards that say similar things can still be treated differently if one was the opening problem statement and the other was the final decision.

    Where the story stops working

    Real positional encoding does not literally attach human-readable numbers like 1, 2, 3. It adds numeric patterns inside vectors, and the model learns how to use those patterns. Also, some transformer variants encode relative distance between tokens, not only absolute place in the sequence.

    Remember thisPositional encoding supplies the order information that token content alone does not contain, allowing attention to distinguish sequences made from the same words.

    This term is contested

    Terminology varies. Some authors use 'positional encoding' broadly for any position signal, while others distinguish fixed encodings from learned position embeddings or separate absolute from relative methods.

    Understand it

    Understand it

    Why it matters

    If you assess, buy, govern or use language AI tools, positional encoding helps explain a basic strength and limitation of transformers. These systems do not get word order for free. They need an explicit way to represent sequence position. That affects translation, summarisation, question answering and long-document processing. It also affects design choices: some models work better on long contexts, repeated patterns or reordered text because of how they encode position. Knowing this helps you ask better questions about performance, failure modes and whether a model may struggle when meaning depends heavily on order, distance or chronology.

    How token vectors gain order information before attention

    After viewing, the reader should be able to explain that transformers need an extra position signal because token embeddings alone do not preserve sequence order.

    1. 01

      1. Tokens become content vectors

      Split a short sentence into tokens. Each token gets an embedding vector that represents content, but these vectors do not by themselves say where the token appeared.

      Draw four token boxes in a vertical list, for example 'report', 'was', 'not', 'approved'. Next to each, show a small coloured horizontal bar icon labelled 'content vector'. Use the same style for repeated tokens.

    2. 02

      2. Add a position vector

      Create a different position vector for each slot in the sequence: first, second, third and so on. This can be fixed or learned, depending on the model design.

      Add a second column with one position marker per row labelled p1, p2, p3, p4. Use a distinct colour family from the content vectors. Include a note: 'different positions, even if token text repeats'.

    3. 03

      3. Combine content and position

      For each row, combine the token vector and the position vector, often by addition. The result is a new vector that carries both what the token is and where it is.

      Draw plus signs between the two columns and a third output column of merged bars. Label each merged bar 'input to attention'. Show one repeated token example with different output because position differs.

    4. 04

      4. Attention can now use order

      Self-attention receives vectors that include position information, so it can treat 'dog bites man' differently from 'man bites dog' and can recognise nearby or earlier tokens differently.

      From the merged output column, draw arrows into a simplified attention block. Add a small side note with two mini-sequences in opposite order to show that different positions lead to different attention behaviour.

    Show one sentence moving through three stages: token embeddings, position vectors and combined vectors used by self-attention. Read left to right. Each row represents one token position, so the viewer can compare what changes when position information is added.

    Hovering over a row highlights the token embedding, its position vector and the merged vector together; a toggle switches between 'fixed positions' and 'learned positions' labels without changing the overall layout.

    Describe this without the diagram

    The diagram is a left-to-right flow with four rows, one row per token in a short sentence. In the first column, each token has a content vector only. In the second column, each row has a distinct position vector labelled by sequence slot. In the third column, the content vector and position vector for the same row are combined into one merged vector. All merged vectors feed into a final attention block. The main point is that token embeddings alone represent content, while the added position vectors supply order, allowing attention to distinguish the same words in different places.

    With reduced motion: A static four-column diagram shows tokens, their content vectors, their position vectors and the merged vectors that feed into attention. Fixed colour coding and labels replace hover highlights and toggles.

    A transformer processes text as a sequence of tokens, which are small pieces such as words or word fragments. Each token first gets an embedding, meaning a numeric vector that represents features learned from data. But an embedding by itself does not tell the model whether the token came first, fifth or last.

    Positional encoding introduces location information into token representations or the attention calculation. Some methods add a fixed or learned position vector to each token vector. Other methods represent relative distance or modify the attention calculation directly.

    With a position signal, self-attention can compare token content while also taking order or distance into account. Without such a signal, basic self-attention does not inherently encode sequence order and responds to reordered inputs with correspondingly reordered outputs.

    You may also hear position embedding. People sometimes use that as a broad synonym, but it can refer more specifically to learned position vectors, while positional encoding can also include fixed patterns such as sinusoidal signals. The terminology is not always used consistently across papers and tools.

    How “Meeting notes with line numbers” maps onto the real thing

    In the storyIn the AI system
    sentence cardstoken embeddings
    line numbers stamped on each cardposition information added to each token
    sorting cards into a coherent summaryself-attention combining information across the sequence
    same sentence text appearing in different placessame token content having different roles because of position

    A worked example

    A monitoring and evaluation officer answers a town hall question by comparing two sentences that contain the same tokens in different orders.

    Starting from: Sequence A: “assessment preceded training”. Sequence B: “training preceded assessment”.

    1. Tokenise Sequence A as “assessment”, “preceded”, “training”, and Sequence B as “training”, “preceded”, “assessment”. Both contain the same three token identities.
    2. Assign illustrative content vectors: e(assessment)=[2,1], e(preceded)=[1,3] and e(training)=[0,2]. A token receives the same content vector wherever it appears.
    3. Assign illustrative position vectors to the three slots: p1=[10,0], p2=[0,10] and p3=[-10,0].
    4. Add the vectors. “Assessment” becomes [12,1] in Sequence A because it is in position 1, but [-8,1] in Sequence B because it is in position 3.
    5. Attention now receives different position-sensitive representations for the two sequences, so it can learn that the reported order of assessment and training has been reversed.

    Result: Although the token set is identical, positional encoding lets the transformer distinguish which activity came first.

    In a United Nations workplace

    At a shared services centre town hall, a monitoring and evaluation officer is asked why the summarisation pilot treats “assessment preceded training” differently from “training preceded assessment”. A consolidated progress report is due to the front office by close of business, naturally. Positional encoding gives the transformer information about where each word occurs, allowing attention to use order as well as word content.

    The two sentences contain the same words but report opposite sequences of events. Without a positional signal, attention cannot reliably recover that distinction from token content alone. That can affect summaries of milestones, dependencies and results.

    Before you try thisRaw monitoring comments containing names, identifiable participant information or other sensitive details must not be entered into a tool that has not been approved for them. Use synthetic or properly minimised text for testing, follow the office’s data protection guidance, and retain human review of the progress report.

    Where you may hear it

    • During a vendor demonstration to an information technology governance board
    • In a model validation session before a reporting tool is cleared
    • At a procurement meeting comparing transformer architectures
    • In a technical briefing on long-context models

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • Attention already handles order, so no extra position signal is needed.

      Basic self-attention compares tokens to each other based on their vectors, but it does not inherently mark first, second or last. If token vectors contain only content, many reorderings look too similar. Positional encoding supplies the missing order information.

    • Positional encoding and embedding mean exactly the same thing.

      An embedding usually means a learned vector representing an item such as a token. Position information can also be represented as vectors, but 'positional encoding' is a broader label. It may refer to fixed patterns, such as sinusoidal functions, or to learned position embeddings. In practice, people often blur the terms.

    • Position vectors just label each token with an index, so they are trivial.

      The position signal must work inside the model's vector computations. Good designs help attention compare nearby and distant tokens and sometimes generalise to sequence lengths not seen in training. That is more subtle than attaching a simple visible number.

    Quick check

    Why is positional encoding added to token representations in a transformer?

    Go deeper

    Go deeper

    The simple version

    Transformers need a way to keep track of order. Positional encoding gives each token a signal for where it appears, so the model can tell the difference between the same words in different positions.

    The technical version

    In a transformer, token embeddings alone are permutation-insensitive with respect to order unless some positional signal is introduced. Positional encoding injects sequence-position information into the token representations before, or within, attention layers. A common early method adds a fixed sinusoidal vector to each token embedding, with different frequencies across dimensions so positions can be distinguished and relative offsets can be inferred by linear operations. Other systems use learned absolute position embeddings, learned relative position biases, rotary position methods or other schemes. The exact method differs, but the role is the same: modify the representations or attention computation so self-attention can model order and distance, not only token identity.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Positional encoding supplies the order information that token content alone does not contain, allowing attention to distinguish sequences made from the same words.