Learn AI /Large Language Models /Vocabulary
Vocabulary
- Deeper
- Large Language Models
- Technical readers
Also called: token vocabulary.
Back to the milestone you came from
In five seconds
What it is
The full set of tokens a language model can represent and choose from during text processing and generation.
In plain words
Think of it as the model’s box of text pieces. When text goes in, it gets broken into pieces from that box. When text comes out, the model chooses the next piece from the same box.
An everyday example
In a water utility control room, an analyst watches a pump sensor dashboard. Beside the screen is a drawer of pre-printed labels such as “pressure”, “drop”, “alarm”, “12”, “.5”, and “kPa”. When an unusual reading appears, the analyst can describe it only by pulling labels from that drawer, sometimes using one label and sometimes combining several smaller ones to express a rare term or number. If there is no single label for a specialised code, the analyst spells it out from smaller pieces that are available.
A real analyst understands the equipment and can invent new descriptions freely. A language model does not have a literal drawer or human situational understanding. Its vocabulary is a fixed token inventory tied to a tokenisation scheme, and generation works by scoring those tokens mathematically, not by consciously choosing labels.
Remember thisA model's vocabulary is the finite token set its tokeniser maps text into and its output layer scores at every generation step.
Understand it
Understand it
Why it matters
Vocabulary affects practical decisions. It influences how efficiently a model handles names, codes, numbers, spelling variants, and different languages or scripts. A term that matches one token may be cheaper and easier to keep inside the context window than the same term split into many tokens. It also affects system design: model interoperability, prompt length, latency, and output control all depend partly on token inventories. For non-specialists, this helps explain why one model handles a domain term smoothly while another breaks it into awkward pieces, and why text length in characters is not the same as text length in tokens.
How vocabulary constrains what a model can read and write
Show that text is converted into token IDs from a fixed vocabulary, and that generation selects the next token only from that same inventory.
01
1. Input text
Start with raw text such as a common word, a rare code, or a number with punctuation.
Draw a text box on the left containing three short examples: a common word, a rare technical string, and a decimal number.
02
2. Tokenisation
The text is split into tokens. Some items become one token; others are broken into smaller pieces.
Show arrows from each example to token chips. For one example, use one chip; for another, use several chips; include punctuation as its own chip.
03
3. Vocabulary mapping
Each token chip must exist in the model’s fixed vocabulary and is represented by an ID.
Place a central panel labelled 'fixed vocabulary' with a grid of token chips and ID numbers. Highlight the chips used by the examples.
04
4. Next-token choice
During generation, the model scores all tokens in the vocabulary and selects one token from that inventory.
On the right, show a bar list of candidate tokens from the vocabulary with different scores, then highlight one selected token and append it to the output line.
A left-to-right flow with four boxes: input text, tokenisation, vocabulary lookup, and next-token choice. Read it as one cycle. The same vocabulary supports both reading text into tokens and generating the next token from a fixed menu of options.
Hovering over an example highlights which vocabulary items are used, and a toggle switches between a common term and a rare term to show different token splits.
Describe this without the diagram
This diagram explains vocabulary as a fixed inventory of tokens. First, raw text appears on the left. Second, tokenisation breaks that text into one or more token pieces; a common item may become one token, while a rare technical string may become several smaller tokens. Third, each token piece is matched to an item in a central fixed vocabulary and shown with an ID number. Fourth, on the right, generation is shown as scoring many possible tokens from that same vocabulary and selecting one of them as the next token. The main point is that the vocabulary is the model’s allowed token set for both reading and writing, while tokenisation is the splitting procedure.
With reduced motion: A static four-column diagram shows text examples, their token splits, a highlighted fixed vocabulary panel with token IDs, and a final column where one candidate token is chosen from a scored list.
A language model does not handle raw text one character at a time in the main prediction step. First, text is split into tokens: units such as whole words, parts of words, punctuation marks, numbers, or special markers. The vocabulary is the catalogue of all token types the model can use.
What goes in is text. That text is tokenised, meaning converted into token IDs from the model’s vocabulary. If a piece of text is unusual, it may be split into several smaller tokens rather than one neat token. What comes out at this stage is a sequence of token IDs the model can process.
During generation, the model does not choose any possible string of characters. At each step, it assigns scores to every token in its vocabulary and then one token is selected by the decoding method. That selected token is appended, and the process repeats. So the vocabulary directly limits the menu of next-step choices.
This term is often confused with tokenisation and with corpus. Tokenisation is the procedure that breaks text into tokens. Corpus is the training text collection. The vocabulary is neither the procedure nor the data source; it is the fixed inventory of token units the model was built to use.
How “A control room with a fixed label drawer” maps onto the real thing
| In the story | In the AI system |
|---|---|
| drawer of pre-printed labels | the model’s token vocabulary |
| pulling labels to describe the reading | selecting tokens during generation |
| combining smaller labels for a rare term | representing uncommon text with multiple tokens |
| labels like numbers, units, and punctuation | tokens are not only words; they can be subwords and symbols |
A worked example
A reporting officer at a regional bureau investigates why an approved drafting tool sometimes reproduces the term “co-financing” inconsistently in a progress report.
Starting from: Co-financing remains subject to clearance.
- The tokeniser searches its fixed vocabulary for token pieces that represent the character string.
- Depending on that vocabulary, “Co-financing” might be one token or several, such as “Co”, “-”, “financing”. Each selected token has a corresponding vocabulary ID.
- For every next-token position, the model's output layer assigns a score to every token in the same vocabulary.
- The decoding rule selects one token ID, converts it back to its token string and appends it. The process then repeats.
Result: The model never reads or writes the term outside its vocabulary. If the term requires several tokens, generation takes more steps and offers more points at which a different token may be selected.
In a United Nations workplace
A budget officer in a Resident Coordinator's Office uses an approved language tool to check terminology in the quarterly report to a donor before it goes for clearance. Repeated output labels, abbreviations and hyphenated budget terms do not always correspond to single tokens in the model's vocabulary.
If a term is absent as a single vocabulary item, the tokeniser represents it with smaller pieces. This can increase token use and make specialised labels, unusual abbreviations or number formats harder for the model to reproduce consistently. At each generation step, the model can select only from its fixed vocabulary.
Before you try thisDo not enter an uncleared donor report, implementing-partner financial details, banking information or personal data into a tool that is not approved for that material. Tool approval does not automatically cover every data category. Follow the office's access controls and standing instruction on external tools.
Where you may hear it
- A vendor demo before an ICT governance board
- A procurement unit's technical evaluation of language-model bids
- A multilingual testing session at Headquarters
- A model review before an approved tool is opened to a country office
Avoid confusion
Avoid confusion
Often confused with
Token
A token is one unit from a model’s vocabulary used to represent input or output, often smaller than a word.
Tokenisation
The process of breaking text into smaller units a language model can process, such as words, parts of words, punctuation marks or special symbols.
Corpus
An organised collection of text or recorded speech used to study language or train and test language-processing systems.
Common misconceptions
Common misconception: The vocabulary is just a list of dictionary words the model knows.
Correction: Not quite. Vocabulary items are tokens, not necessarily full words. They can be word pieces, punctuation marks, numbers, whitespace-prefixed fragments, control markers, or special symbols. That is why an unfamiliar word can still be represented: it is often split into smaller known tokens.
Common misconception: A bigger vocabulary always makes a model better.
Correction: A larger vocabulary can reduce token counts for some text, but it also increases the number of token options the model scores at each step and can complicate design trade-offs. Quality depends on many factors, including training data, tokenisation scheme, model size, and optimisation, not vocabulary size alone.
Common misconception: If a term is not in the vocabulary, the model cannot process it.
Correction: Usually it still can, by splitting the term into smaller tokens that are in the vocabulary. The issue is often efficiency and convenience, not total impossibility. The representation may become longer or less neat, which can affect context usage and sometimes behaviour.
Quick check
Which statement best describes token vocabulary in a language model?
The vocabulary is the model’s token inventory. It is not the corpus, not the current context, and not a guarantee of subject competence. Text is tokenised into this inventory, and generation chooses from it step by step.
Go deeper
Go deeper
The simple version
Think of it as the model’s box of text pieces. When text goes in, it gets broken into pieces from that box. When text comes out, the model chooses the next piece from the same box.
The technical version
In most LLMs, the vocabulary is the finite set V of token types defined by the tokenizer and tied to the model’s input and output layers. Tokenisation maps a character string to a sequence of token IDs in V. During generation, the model produces a logit or score for each token in V at every step; sampling or another decoding rule then selects one token ID. The selected ID is converted back to its token string and appended to the sequence. Because the output distribution is defined over the vocabulary, the model does not directly choose arbitrary character strings at each step. Vocabulary design interacts with tokenisation strategy, multilingual coverage, byte handling, special tokens, embedding matrices, output projection, memory use, and token efficiency.
Reviewed external resources
EncyclopaediaVocabulary (opens in a new tab)Wikipedia
Wikipedia gives a broad reference overview of vocabulary in language and computing, suitable for readers who want a quick orientation.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
The Hugging Face LLM Course helps technically minded readers connect vocabulary to tokenisers, IDs and model pipelines.
Deeperlink checked 2026-07-31
VideoLarge language models, explained visually (opens in a new tab)3Blue1Brown
The 3Blue1Brown video gives an intuitive visual explanation of how language models process text, useful for readers who learn best from diagrams and animation.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Vocabulary (opens in a new tab)YouTube search
The YouTube search results page helps readers find multiple video explanations of vocabulary from different teachers and difficulty levels.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Vocabulary (opens in a new tab)Reddit search
The Reddit search results page helps readers see common questions, confusions and practitioner discussions about vocabulary, with the usual need for careful fact-checking.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
A model's vocabulary is the finite token set its tokeniser maps text into and its output layer scores at every generation step.