Learn AI /Large Language Models /Throughput
Throughput
- Intermediate
- Large Language Models
- Technical readers
Also called: tokens per second.
Back to the milestone you came from
In five seconds
What it is
Throughput measures how much text a model can process or generate in a given time, usually counted in tokens per second.
In plain words
Throughput is how fast a language model gets through text, usually measured as tokens per second.
An everyday example
At an international conference, a captioner listens to a speaker and types subtitles onto a large screen. If the captioner can type 12 words each second, the subtitles keep up well; if the speaker talks faster than that, the display falls behind. The audience notices both the initial pause before text appears and the speed once the text starts flowing. The organiser cares about the steady typing rate because it determines whether long speeches stay readable in real time.
A human captioner listens to speech and may paraphrase, while a language model processes tokenised text and computes probabilities over its vocabulary. Also, words and tokens are not the same size, so 'words per second' only roughly matches 'tokens per second'. The analogy helps with speed and flow, but not with the actual mechanism.
Remember thisThroughput is the volume of tokens processed or generated per unit time. Input and output throughput may differ, and neither should be confused with the delay before a response begins.
Understand it
Understand it
Why it matters
Throughput affects cost, capacity and user experience, but the scope of the measure matters. Aggregate service throughput shows how much work a deployment completes across users. Per-response output throughput shows how quickly one response generates tokens. Batching can raise aggregate throughput without making each response arrive faster. If you ignore throughput, a system that performs well in a demo may struggle under daily demand. Technical teams can use clearly defined measures to compare deployments and estimate infrastructure needs. Non-specialists can ask: 'Can this service handle enough work, and do individual responses arrive fast enough to be useful?'
Latency and throughput in one response
Show that throughput is the sustained token rate during processing or generation, and that it is not the same thing as latency.
01
1. Input arrives
The user sends a prompt, which is split into tokens before model computation begins.
Draw a box labelled 'Prompt' feeding into a row of small token rectangles labelled t1, t2, t3, t4.
02
2. Initial delay
The system spends some time preparing and computing before the first visible output token appears. That delay is latency, not throughput.
Place a short gap with a clock icon between input tokens and the first output token. Label the gap 'initial wait'.
03
3. Output starts
After the first token appears, the model continues generating tokens one step at a time.
Show output token rectangles appearing sequentially: o1, o2, o3, o4, connected by arrows.
04
4. Measure the rate
Count how many output tokens appear in one second. That count is output throughput. Input processing can have its own separate rate.
Add a one-second bracket under several output tokens and label it '8 tokens in 1 second = 8 tok/s'. Include a note: 'input and output rates may differ'.
A left-to-right flow diagram with one request moving through stages: input tokens arrive, the model starts work, output tokens appear one by one. A small clock marks the initial wait, and a rate label marks the ongoing token stream. Read from left to right to compare waiting time with steady speed.
On hover or tap, highlight either the latency gap or the one-second output bracket to show which metric each label refers to.
Describe this without the diagram
The diagram is a left-to-right flow showing a prompt becoming tokens, then a pause, then a sequence of generated output tokens. The pause before the first output token is labelled as latency. The later section groups several output tokens inside a one-second bracket and labels that count as throughput, for example 8 tokens per second. A note says input processing speed and output generation speed can be measured separately. The point is that waiting time and sustained rate are different measurements.
With reduced motion: A static flowchart shows prompt tokens on the left, a labelled pause in the middle, and a row of output tokens on the right. The pause is marked as latency, and a bracket under several output tokens marks throughput as tokens per second.
Throughput tells you the rate of work. In large language models, it usually means how many tokens the system handles each second. A token is a small unit of text, often a word piece rather than a full word.
What goes in is text that has been split into tokens. The model then runs inference, meaning it computes what token is likely to come next, one step at a time for generation, or processes many input tokens while reading the prompt. What comes out is a stream of generated tokens, and throughput tells you how fast that stream moves.
This is different from latency. Latency is the waiting time before you see a response or before the next token appears. Throughput is the sustained rate once work is under way. A system can have low latency but modest throughput, or high throughput but a slow initial delay.
Throughput depends on several things: model size, hardware, batch size, prompt length, context window use, and whether the system is reading input tokens or generating output tokens. People often quote one number, but in practice there may be separate rates for prompt processing and token generation.
How “Subtitles typed during a live event” maps onto the real thing
| In the story | In the AI system |
|---|---|
| captioner typing words each second | model generating tokens per second |
| speaker's stream of speech | incoming prompt or context to be processed |
| initial pause before subtitles appear | latency before output starts |
| subtitles continuing at a steady pace | sustained throughput during generation |
A worked example
A country office compares two approved deployments for producing a 180-token draft from a synthetic 1,200-token prompt. Both are tested under the same serving load, and the reported rates cover output generation rather than prompt ingestion.
Starting from: Deployment A starts output after 0.8 seconds and generates 40 tokens per second. Deployment B starts after 0.3 seconds and generates 18 tokens per second.
- Deployment A takes about 180 ÷ 40 = 4.5 seconds to generate the output after generation begins.
- Including the 0.8-second wait for the first token, Deployment A finishes in about 5.3 seconds.
- Deployment B takes about 180 ÷ 18 = 10 seconds to generate the output after generation begins.
- Including the 0.3-second wait for the first token, Deployment B finishes in about 10.3 seconds.
- Prompt-ingestion throughput cannot be derived from these figures. It would require a separate measurement of how quickly each deployment processes the 1,200-token input.
Result: Deployment B has lower time to first token, so it appears more responsive initially. Deployment A has higher output throughput and completes the full draft sooner.
In a United Nations workplace
A protection officer in a country office is finalising a delegation briefing while comments from capitals are still outstanding. The office tests two approved deployments using a synthetic prompt with the same length and structure as the expected task, measuring how quickly each produces the full draft under the anticipated workload.
Higher output throughput can finish a long draft sooner and allow more colleagues to use the service concurrently. It does not necessarily mean that the first words appear sooner. Input throughput should be measured separately if ingesting lengthy background material is a significant part of the task.
Before you try thisThe protection officer must not use the actual briefing, case details, names, witness accounts or other protection-sensitive material in a benchmark unless the environment is specifically approved for that information. Use synthetic or properly sanitised text, because prompts and outputs may be logged, retained or accessible beyond the intended group.
Where you may hear it
- A vendor demonstration to a country office procurement unit
- An ICT governance board reviewing capacity for an internal drafting tool
- A service review after response times slow near close of business
Avoid confusion
Avoid confusion
Often confused with
Latency
The delay between sending a request to a model and receiving its reply, or the next part of that reply.
Token
A token is one unit from a model’s vocabulary used to represent input or output, often smaller than a word.
Completion
The text a language model produces after a prompt, built token by token from patterns learned during training.
Inference
Using a trained model on new input to produce an output such as a score, category, translation or generated text.
Common misconceptions
Common misconception: Higher throughput means the model is more accurate.
Correction: Not necessarily. Throughput measures speed, not factual reliability or suitability for a task. A faster model may produce weaker answers, and a slower model may produce better ones. You need separate evaluation for quality.
Common misconception: Throughput and latency are the same thing.
Correction: They describe different parts of performance. Latency is how long you wait before output starts, or before the next token appears. Throughput is the sustained rate of token processing or generation over time. A system can be good at one and weak at the other.
Common misconception: Tokens per second tells you exactly how many words per second users will see.
Correction: Tokens are not the same as words. One word may map to one token, several tokens, or part of a token depending on the tokeniser and language. So tokens per second is a useful engineering measure, but it does not translate cleanly into words per second.
Quick check
Which statement best describes throughput for a language model service?
Throughput is a rate, usually tokens per second. The first-token wait is latency, correctness is a quality question, and maximum prompt length relates to the context window rather than speed.
Go deeper
Go deeper
The simple version
Throughput is how fast a language model gets through text, usually measured as tokens per second.
The technical version
In LLM inference, throughput is the volume of token processing completed per unit time. People often report output throughput as generated tokens per second during decoding, but input throughput for prompt ingestion can be measured separately and may be much higher. Throughput is shaped by model architecture, parameter count, hardware bandwidth, parallelism, batch size, sequence length, memory limits and serving strategy. It differs from latency: latency captures delay to first token or per-token response delay, while throughput captures sustained work rate. Reported numbers also depend on what exactly is counted, such as prompt tokens only, completion tokens only, or both combined, so comparisons need the measurement method to be clear.
Reviewed external resources
EncyclopaediaThroughput (business) (opens in a new tab)Wikipedia
This link gives a broad, non-AI introduction to throughput as a rate of work, which suits readers who want the basic idea first.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This course helps technical readers place throughput within the wider mechanics of language model inference and deployment.
Deeperlink checked 2026-07-31
VideoLarge language models, explained visually (opens in a new tab)3Blue1Brown
This video gives an intuitive visual picture of how large language models process and generate text, which helps readers relate speed metrics to the mechanism.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Throughput (opens in a new tab)YouTube search
This search page helps readers find multiple video explanations of throughput and suits people who want different teaching styles rather than one fixed source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Throughput (opens in a new tab)Reddit search
This search page helps readers see how practitioners discuss throughput in real-world use, which suits readers looking for informal examples and recurring questions.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Throughput is the volume of tokens processed or generated per unit time. Input and output throughput may differ, and neither should be confused with the delay before a response begins.