presentsUNAIVERSE
The UN AI timeline

    Learn AI /Large Language Models /Tokenisation

    Tokenisation

    Also called: tokenization, tokenizer.

    Detail

    In five seconds

    What it is

    The process of breaking text into smaller units a language model can process, such as words, parts of words, punctuation marks or special symbols.

    In plain words

    Before a language model can work with text, the text is chopped into small pieces it recognises. Those pieces are counted and turned into numbers.

    An everyday example

    Imagine composing a sentence using a box of magnetic pieces. The box contains some complete words, many common word fragments, punctuation marks and symbols. If the word you need is present, you use one piece. If it is not, you assemble it from several smaller pieces. Each magnet also has a number on the back. Tokenisation is the step that chooses the pieces in order and records their numbers. A different box may divide the same sentence differently.

    Where the story stops working

    A radio receives a physical signal, while tokenisation is a text-splitting procedure applied to characters. The tokenizer does not search the airwaves or recover meaning from noise; it deterministically maps text to predefined token pieces.

    Remember thisModels process token sequences, not ordinary word counts. Context limits, input size and output length therefore depend on the tokenizer and its fixed vocabulary.

    Understand it

    Understand it

    Why it matters

    If you use or procure AI systems, tokenisation affects cost, speed, context limits and reliability. A message that looks short to a person may be long to the model, which can change whether important text fits, how much processing it needs and how results are interpreted.

    How a tokenizer splits text into model-sized pieces

    After using this explainer, the reader should be able to say that text is converted into token pieces and IDs, and that token count can differ from word count.

      • whole word
      • part of a word
      • punctuation or number

      This is a teaching approximation, not a real tokeniser. Every model splits text differently, and the same sentence can be a different number of tokens in two models. Use a model's own tokeniser when the count matters.

      An interactive panel shows a text input area at the top, a token count badge, and two linked outputs below. The first output displays the input text segmented into coloured token chips in reading order. The second output shows the corresponding token IDs in a horizontal list. A side note updates live to explain why some words stay whole while others are split into smaller pieces, especially for uncommon words, punctuation and mixed-language text.

      Readers can type their own text, choose preset examples, click or tab through token chips to inspect IDs, and toggle a small text change to compare token counts before and after.

      Describe this without the diagram

      Screen-reader users get a text-first simulation. After entering text or selecting an example, the page announces the token count, then reads an ordered list where each item states the token number, the exact token text and its token ID. A summary then states the character count, approximate word count and token count, followed by a short explanation of whether any words were split into multiple tokens. A comparison toggle can be activated to hear how a small text change alters the token sequence and total count.

      With reduced motion: Users who prefer reduced motion see the same explainer without animated transitions. Token chips and IDs update instantly as static content, and selection changes use colour, outline and text labels rather than movement.

      A language model does not read text as whole sentences in the way people usually describe reading. First, the input text is split into smaller pieces called tokens. A token may be a whole word, part of a word, a punctuation mark, a space pattern or a special control symbol.

      What goes in is ordinary text, such as a sentence or prompt. A tokenizer applies a fixed set of rules and a predefined vocabulary, meaning the list of token pieces the model was built to use. What comes out is a sequence of token IDs, which are numbers standing for those pieces.

      This matters because the model works on tokens, not directly on characters or ideas. Length limits are usually measured in tokens. Cost, speed and truncation often depend on token count rather than word count.

      Tokenisation is often confused with a token itself. Tokenisation is the splitting process. A token is one resulting piece. It is also different from embedding: tokenisation turns text into discrete pieces, while embedding turns those pieces into numerical vectors the model can process further.

      How “Writing with a fixed box of word magnets” maps onto the real thing

      In the storyIn the AI system
      radio that only locks onto certain stationstokenizer with a fixed vocabulary of recognised pieces
      spoken message arriving in partsinput text being split into tokens
      whole words versus short fragmentssome tokens represent full words, others represent subword pieces
      unusual name needing several channelsrare or unfamiliar text often becoming multiple tokens

      A worked example

      A toy tokenizer processes the sentence "The evaluation is multi-year." Its fixed vocabulary contains pieces for whole words, subwords, punctuation and spaces attached to some pieces.

      Starting from: The evaluation is multi-year.

      1. The tokenizer matches the character string against pieces in its fixed vocabulary.
      2. For illustration, it produces the ordered pieces ["The", " evaluation", " is", " multi", "-", "year", "."]. Another tokenizer could split the same sentence differently.
      3. Each piece is mapped to its vocabulary ID, producing an ordered sequence of seven token IDs.
      4. The model processes those IDs. It does not receive the sentence as a sequence of five ordinary words.

      Result: In this toy example, five written words become seven tokens because "multi-year" and the full stop are split into separate vocabulary pieces. The exact count depends on the tokenizer used.

      In a United Nations workplace

      In an evaluation office, a chief of section is clearing terms of reference before Friday. The proposed AI tool has a stated context limit, but the annexes contain tables, acronyms and multilingual text, so the page count is a poor guide to whether the full file will fit.

      Tokenisation determines how the text is divided before processing. The chief asks for a token count using the tool's actual tokenizer, rather than accepting an estimate based on words or pages. Different languages, symbols and formatting may consume markedly different numbers of tokens.

      Before you try thisDo not upload confidential evaluation material, identifying information or procurement-sensitive content merely to obtain a token count. Use an approved tool suitable for the material, or test with a sanitised substitute that preserves the relevant formatting and language patterns.

      Where you may hear it

      • At a vendor demo during a procurement meeting
      • In an ICT governance board discussion on context limits
      • During clearance of a long briefing note for an AI-assisted workflow
      • While troubleshooting a prompt that fits by word count but is rejected by the tool

      Avoid confusion

      Avoid confusion

      Often confused with

      Common misconceptions

      • One word always equals one token.

        Not usually. Many tokenizers split words into parts, and punctuation, spaces or special symbols can also count as separate tokens.

      • Tokenisation gives the model the meaning of the text.

        No. Tokenisation only converts text into manageable pieces and IDs. Later stages process those pieces, and even then output is not guaranteed to be factually correct.

      • All models split text in the same way.

        Different models can use different tokenizers and vocabularies, so the same sentence may produce different token counts across systems.

      • Tokenisation and embedding are the same step.

        They are separate. Tokenisation makes discrete pieces; embedding converts those pieces into numerical vectors used inside the model.

      Quick check

      Why can two sentences with the same number of words use different amounts of model context?

      Go deeper

      Go deeper

      The simple version

      Before a language model can work with text, the text is chopped into small pieces it recognises. Those pieces are counted and turned into numbers.

      The technical version

      A tokenizer maps an input character string into an ordered sequence of tokens drawn from a fixed vocabulary, then into token IDs. Most modern LLM tokenizers use subword schemes, so boundaries often do not align with words, and downstream limits and inference operate over token sequences.

      Reviewed external resources

      Where this came from

      Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

      The one thing to remember

      Models process token sequences, not ordinary word counts. Context limits, input size and output length therefore depend on the tokenizer and its fixed vocabulary.