Learn AI /Large Language Models /Token
Token
- Starter
- Large Language Models
- Technical readers
Also called: tokens.
Back to the milestone you came from
In five seconds
What it is
A token is one unit from a model’s vocabulary used to represent input or output, often smaller than a word.
In plain words
A token is a model input or output unit, such as a word piece, punctuation mark or special symbol.
An everyday example
An admissions office receives application forms from schools in Nairobi, Lima, Dhaka and Belgrade. Before staff can check them quickly, one worker cuts each form into standard pieces such as surname, given name, date, postcode and signature box. Some pieces match a whole field, while others split awkwardly because different schools format forms differently. The office system then processes those standard pieces one by one instead of looking at each page as a whole. Later, the pieces are arranged back into a readable application record.
Real tokenisation does not understand form fields the way office staff do. It usually splits by patterns learned for efficient processing, so the pieces may look odd to people and do not carry meaning as neatly as application sections.
Remember thisModels process and generate sequences of tokens, which may be whole words, parts of words, punctuation marks or other vocabulary units.
Understand it
Understand it
Why it matters
If you use or buy AI tools, token counts affect what fits, how fast replies arrive, and what the service may cost. Understanding tokens helps you set realistic expectations and compare systems more sensibly.
How one line of text becomes tokens
Show that a model processes text as tokens, not as whole visible words, and that one sentence can split differently depending on the tokeniser.
- whole word
- part of a word
- punctuation or number
This is a teaching approximation, not a real tokeniser. Every model splits text differently, and the same sentence can be a different number of tokens in two models. Use a model's own tokeniser when the count matters.
An interactive tokenizer lets the reader type or paste text and watch it split into coloured token blocks with counts and IDs. It shows that tokens can be words, word parts, punctuation or spaces, and that token totals affect limits and cost.
Readers can type their own text, paste examples in different languages, select token blocks to inspect IDs, and compare character, word and token counts as the display updates.
Describe this without the diagram
Screen-reader version: You enter text in an edit field. The page then reports the text split into an ordered list of tokens. For each token, it announces the token number, the exact text piece, and its token ID. It also reads summary counts for characters, words and tokens, followed by a short note that tokens may be whole words, parts of words, punctuation or spaces, so token count may differ from word count.
With reduced motion: Reduced-motion version: token blocks appear instantly in a static row with no animated splitting or counting effects. Selecting a token updates the side panel without movement.
People often imagine a language model processing one letter at a time or one whole word at a time. Usually, it does neither. A tokenizer first breaks text into units called tokens. A token might be a whole word, part of a word, punctuation, a number, a space pattern or a special symbol.
When you type a prompt, the system converts its tokens into token IDs. A token ID is a number linked to one item in the model’s vocabulary, meaning the full set of tokens it can represent and choose from. The model processes the resulting sequence rather than relying on visible word boundaries.
When the model replies, it predicts output tokens in sequence. Software converts them back into readable text and may display the text gradually, piece by piece.
Token is often confused with tokenisation. A token is one unit after splitting. Tokenisation is the process that does the splitting. It is also often confused with a word, but tokens and words do not line up neatly across languages, writing systems, or even within one sentence.
Token counts matter because they affect cost, speed and limits. Many systems charge by token and set token limits for the context and output in one run.
How “Sorting application forms into standard pieces” maps onto the real thing
| In the story | In the AI system |
|---|---|
| standard pieces cut from each form | tokens |
| the worker dividing forms into pieces | tokenisation |
| the office system processing pieces one by one | the model handling a sequence of token IDs |
| rebuilding the readable application record | turning output tokens back into text |
A worked example
Before sending the side-event concept note to the front office, the information management officer tests a heading instruction in the approved assistant.
Starting from: Draft a heading: Digital inclusion side event at 13:15.
- The tokeniser divides the instruction into pieces from its vocabulary. An illustrative split might include "Draft", " a", " heading", ":", " Digital", " inclusion", " side", " event", " at", " 13", ":", "15" and "."
- Each piece is represented by a token ID. These IDs, rather than the visible words themselves, are the sequence processed by the model.
- The model predicts the next token ID, adds it to the sequence and repeats the process until the response is complete or an output limit is reached.
- The software converts the generated token IDs back into readable text.
Result: The officer might receive: "Digital Inclusion Side Event, 13:15". The number of tokens will not necessarily match the number of words, and another model may split the same instruction differently because token vocabularies and tokenisers are model-specific.
In a United Nations workplace
At a duty station where the front office works in three languages, an information management officer is consolidating the language versions of a side-event concept note for circulation by close of business. The officer checks whether the approved assistant can take the drafting instruction and all three versions without losing the final paragraphs.
The context window is measured in tokens, not pages or words. Similar-length versions may use different numbers of tokens because each language is divided according to the model's vocabulary. The prompt, earlier exchanges and generated reply may all count towards the limit. Token counts can also affect output limits, response time and, where applicable, usage charges.
Before you try thisA larger token allowance is not permission to include more sensitive material. Do not paste participant contact details, confidential delegation comments, tracked comments or other protected information into an external tool. Use only a tool approved for the material concerned, and include only what the drafting task requires.
Where you may hear it
- In a vendor demo during a procurement meeting
- At an ICT governance board review of context-window limits
- During a technical handover on prompt and output limits
- When an approved assistant cuts off a draft just before clearance
Avoid confusion
Avoid confusion
Often confused with
Tokenisation
The process of breaking text into smaller units a language model can process, such as words, parts of words, punctuation marks or special symbols.
Vocabulary
The full set of tokens a language model can represent and choose from during text processing and generation.
Parameter
A parameter is a learned numerical setting inside a model that helps shape which token it is likely to produce next.
Side-by-side
Common misconceptions
Common misconception: One token always equals one word.
Correction: Many tokens are only part of a word, punctuation mark or space pattern. This is why short-looking text can still use several tokens.
Common misconception: Tokens are the same as letters or characters.
Correction: Characters are the visible symbols in writing. Tokens are processing units chosen by a tokenizer, so one token can contain several characters or only part of a visible word.
Common misconception: If two models see the same sentence, they must use the same tokens.
Correction: Different models can use different tokenizers and vocabularies, so the same text may split into different token sequences.
Quick check
Which statement about tokens is most accurate?
A token is one unit the model handles. It does not always match a word, and it is different from tokenisation, which is the splitting process. It is also different from parameters, which are part of the model itself.
Go deeper
Go deeper
The simple version
A token is a model input or output unit, such as a word piece, punctuation mark or special symbol.
The technical version
A token is a discrete symbol from a model-specific vocabulary, typically represented by a token ID after tokenisation; the model consumes and predicts sequences of these IDs during next-token generation.
Reviewed external resources
EncyclopaediaLarge language model (opens in a new tab)Wikipedia
This overview article gives broad background on large language models, which suits readers who want context around where tokens fit.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This course explains core LLM concepts including tokenisation in a structured way, which suits readers who want guided technical learning.
Deeperlink checked 2026-07-31
InteractiveTokenizer Playground (opens in a new tab)Hugging Face
This interactive playground lets readers paste text and see how a tokenizer splits it, which suits hands-on learners and developers.
Starterlink checked 2026-07-31
VideoLarge language models, explained visually (opens in a new tab)3Blue1Brown
This video gives a visual intuition for how large language models work, which suits visual learners who want the bigger picture around tokens.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Token (opens in a new tab)YouTube search
This is a search results page for videos about tokens, which suits readers who want to compare several explanations rather than rely on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Token (opens in a new tab)Reddit search
This is a search results page for community discussions about tokens, which suits readers who want informal questions, examples and points of confusion from other users.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Models process and generate sequences of tokens, which may be whole words, parts of words, punctuation marks or other vocabulary units.