presentsUNAIVERSE
The UN AI timeline

    Learn AI /Large Language Models /Sampling (choosing the next token)

    Sampling (choosing the next token)

    Also called: decoding, greedy decoding.

    Detail

    In five seconds

    What it is

    A stochastic procedure that draws one next token from the model’s probability distribution, appends it to the output, and repeats until a stop condition.

    In plain words

    The model suggests possible next pieces of text, and sampling randomly picks one using their probabilities.

    An everyday example

    An office worker receives a photograph of a paper form and must match it to one template in a cabinet. A screen shows likely matches in order: travel claim, vendor registration, incident report, and equipment request. If the worker always picks the top suggestion, the process is fast and repeatable. If the worker instead chooses from the top few plausible matches, the result can vary, which may help in borderline cases but can also introduce a wrong choice.

    Where the story stops working

    A worker can inspect the full form and apply explicit judgement, while a language model produces token probabilities from learned patterns in text. Also, template matching is a classification task with a fixed target, whereas token sampling is a repeated generation process in which each chosen token changes the next step.

    Remember thisSampling draws each next token from a probability distribution. Because every selected token changes the context for the next draw, small early differences can produce substantially different text.

    Understand it

    Understand it

    Why it matters

    If you use or procure language-model systems, sampling helps explain why the same prompt can produce different outputs, why some settings make answers more predictable, and why others increase variety. This affects quality control, auditability, reproducibility, safety review, latency, and user trust. A team choosing settings for drafting, coding help, translation support, or brainstorming should know that decoding is a controllable part of the system. It is separate from training, and it can make a model seem more careful or more creative without changing what the model has learned.

    How next-token sampling turns probabilities into text

    After viewing this, a reader should be able to explain that generation happens one token at a time and that decoding rules change which token is selected from the model’s predicted probabilities.

    1. 01

      1. Current context

      Start with the prompt and any tokens already generated.

      Draw a text box labelled 'Context' containing a short prompt and partial reply, for example 'Summarise: Climate talks... The meeting'.

    2. 02

      2. Candidate tokens scored

      The model assigns probabilities to many possible next tokens.

      Show a vertical list of candidate tokens with bars and percentages, such as 'continued 40%', 'ended 25%', 'focused 15%', 'was 10%', 'on 10%'.

    3. 03

      3. Decoding rule applied

      A rule such as greedy, temperature-adjusted, or top-p determines how the final choice is made from the scored list.

      Place a filter/selector panel over the list with toggles labelled 'greedy', 'temperature', and 'top-p'; highlight one active rule.

    4. 04

      4. One token chosen and appended

      The chosen token is added to the text, and the process repeats until a stop condition is reached.

      Animate or indicate the selected token moving into the context box, updating the partial reply; add a loop arrow back to step 1 and a small stop icon labelled 'end token / limit'.

    A left-to-right sequence shows one generation cycle repeated: context enters, candidate tokens receive probabilities, a decoding rule chooses one token, and that token is appended before the next cycle. Read the arrows from step 1 to step 4, then note that step 4 loops back to step 1.

    Hovering or tapping each decoding rule updates the highlighted candidate and shows a short note on likely effects: more deterministic, more varied, or more constrained.

    Describe this without the diagram

    The diagram explains text generation as a four-step loop. First, the current context contains the prompt and tokens already produced. Second, the model scores many possible next tokens and shows them as a probability list. Third, a decoding rule, such as greedy selection, temperature adjustment, or top-p filtering, is applied to that list. Fourth, one token is chosen and appended to the text. The updated text becomes the new context, and the loop repeats until a stopping condition such as an end token or length limit is reached. The key message is that the model predicts probabilities, while decoding turns those probabilities into one actual next token.

    With reduced motion: A static four-panel diagram replaces animation. Panel 1 shows the current context. Panel 2 shows candidate next tokens with probabilities. Panel 3 shows a highlighted decoding rule selecting from those candidates. Panel 4 shows the chosen token appended to the text, with a visible loop arrow back to panel 1 and a stop condition note.

    A language model does not write a whole reply in one step. At each step, it produces scores for possible next tokens. These scores are used to form a probability distribution. A token is a small unit of text, such as a word, part of a word, punctuation, or a space pattern.

    Decoding is the broader procedure for choosing the next token. Greedy decoding picks the highest-probability token. Sampling instead draws a token using the probabilities, often after controls such as temperature, top-p, or top-k have reshaped or restricted them.

    The input is the current context: the prompt plus all tokens already generated. The model scores possible next tokens, any configured controls are applied, and the selection procedure chooses one token. That token is added to the context before the next step.

    These choices can make outputs more stable, more varied, more concise, or more risky. They do not change the model’s weights. They change how its scores are turned into text.

    How “Picking the next form template” maps onto the real thing

    In the storyIn the AI system
    list of likely templates on the screenprobability distribution over possible next tokens
    always choosing the top templategreedy decoding
    choosing from the top few plausible templatessampling from a restricted set such as top-p or top-k
    repeating the match for each new page in a packetselecting one token at a time through the whole completion

    A worked example

    Drafting the opening sentence of a side-event concept note

    Starting from: Prompt: "Complete this sentence: The side event will"

    1. The model assigns probabilities to possible next tokens. Suppose "explore" receives 0.40, "examine" 0.30, "highlight" 0.15, "consider" 0.10 and all others together 0.05.
    2. Greedy decoding always selects "explore", the highest-probability token. Sampling may instead draw "examine" or another token according to the distribution.
    3. With top-p set to 0.85, sampling is restricted here to "explore", "examine" and "highlight". Temperature could separately make their probabilities sharper or flatter before the draw.
    4. After the selected token is appended, the model calculates a new distribution for the next token. This draw-and-append cycle continues until an end token, a stop sequence or another limit is reached.

    Result: Greedy decoding might repeatedly begin, "The side event will explore..." A sampled run could begin, "The side event will examine..." That first draw changes the context, so the rest of the sentence may follow a different path.

    In a United Nations workplace

    At a Addis Ababa liaison office, a legal officer is clearing the concept note for a side event before the front office circulates it to permanent missions at close of business. The drafting assistant produces slightly different formulations each time the same instruction is run.

    Sampling explains the variation. At every step, the assistant draws a token from the model’s adjusted probability distribution, appends it and calculates a new distribution from the expanded context. Lower randomness can support more consistent drafting, while broader sampling can offer alternative wording.

    Before you try thisDo not put uncleared concept notes, internal legal advice, bracketed negotiating text or personal data into a tool not approved for that material. Sampling controls variation, not confidentiality or accuracy. Every generated formulation still requires legal review before clearance.

    Where you may hear it

    • During a vendor demo to an ICT governance board
    • In a technical review before a chat assistant is cleared for pilot use
    • While a programme officer reviews generation settings with a supplier
    • In a reproducibility discussion at an evaluation office

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • Sampling means the model is pulling examples directly from its training data.

      Here, sampling means choosing one next token from the model’s predicted distribution during generation. It is different from sampling data for training or evaluation. Confusing the two can hide the fact that output variation often comes from decoding settings, not from retrieving stored examples.

    • Greedy decoding is always best because it picks the most likely token.

      Picking the top token at each step can produce stable output, but local best choices do not always give the best full sentence or paragraph. Greedy decoding can become repetitive, bland, or trapped in a poor path because each token choice changes the next probabilities.

    • Temperature is the same thing as sampling.

      Temperature is one control that reshapes the probability distribution before selection. Sampling or decoding is the broader process of turning probabilities into an actual token. You can sample with different temperatures, or use deterministic greedy decoding with no random draw at all.

    Quick check

    What does greedy decoding do during text generation?

    Go deeper

    Go deeper

    The simple version

    The model suggests possible next pieces of text, and sampling randomly picks one using their probabilities.

    The technical version

    At each generation step, the model outputs logits over the vocabulary, conditioned on the current context. Decoding may transform these logits or their probability distribution. Greedy decoding chooses the argmax token deterministically. Sampling instead draws a token from the distribution, which may first be adjusted by temperature or restricted by top-p. The selected token is appended to the sequence, updating the context for the next forward pass. This repeats until an end token, stop sequence, or length limit is reached. Sampling is an inference-time decoding method that affects variability and reproducibility.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Sampling draws each next token from a probability distribution. Because every selected token changes the context for the next draw, small early differences can produce substantially different text.