Learn AI /Large Language Models /Latency
Latency
- Starter
- Large Language Models
- Technical readers
Also called: response time.
Back to the milestone you came from
In five seconds
What it is
The delay between sending a request to a model and receiving its reply, or the next part of that reply.
In plain words
It is how long you wait after asking the model something before you start seeing the answer, or before the whole answer is finished.
An everyday example
You ask a journey desk in Santiago how to get from the airport to a meeting venue during heavy traffic. The staff member first reads your note, checks the map, looks at the time of day, and compares two routes. A few seconds later, they begin speaking with the first step, then continue with the rest of the directions. Your waiting time before hearing the answer, and the pace of the directions after that, are the parts that match latency.
A human journey clerk may reason with real-world knowledge, ask clarifying questions on their own, or choose to stay silent. A language model does not work like a person giving directions. It generates text token by token based on patterns from training and current input. Also, real travel time and response latency are different things: latency is only the waiting time for the answer, not the quality or usefulness of the route.
Remember thisLatency may mean the wait until the first words appear or the wait until the reply is complete. Neither measure tells you whether the answer is good.
Understand it
Understand it
Why it matters
Latency affects whether a tool feels usable in practice. If replies arrive quickly, people can keep their place in a task such as drafting, searching or comparing options. If replies are slow, users may rush, repeat prompts, or stop trusting the workflow. For non-specialists, latency helps with practical choices. A short prompt may return faster than a long one. Turning on extra tools, adding many documents or using a larger model may increase waiting time. That can be acceptable for a careful background task, but less suitable in a live meeting or urgent support setting. It also helps to ask the right question. If a team says a system is "slow", the issue might be network delay, time to first token, full-answer time, or server load. Those are related, but not identical.
Where waiting time comes from in a model reply
Show that response time is the sum of several delays before and during generation, and that long prompts or extra tools can increase it.
01
1. Send request
The user's prompt and conversation history travel to the service. Network distance and congestion can add a small delay before processing starts.
Draw a message bubble entering the first box. Add a thin clock icon below with the note 'network'.
02
2. Prepare input
The system formats instructions, checks safety rules, and breaks text into tokens. Longer context usually means more work before generation begins.
Second box shows stacked text lines turning into small token blocks. Add note 'prompt length, safety checks'.
03
3. Generate first token
The model computes the first piece of output. This often shapes the delay users notice most, because it is the moment the reply first appears.
Third box shows a model chip icon and one token appearing. Add emphasis ring and note 'time to first token'.
04
4. Generate the rest
The model continues producing tokens one by one until the reply ends. The speed here affects how quickly the full answer arrives.
Fourth box shows a row of tokens extending rightward. Add note 'tokens per second, tools may pause output'.
A left-to-right step diagram with four boxes connected by arrows. Each box names one stage that adds delay: send request, prepare input, generate first token, generate remaining tokens. Small notes under boxes show common causes such as network delay, long context, model size and tool use. Read from left to right as a timeline.
Hover or tap each box to reveal one short note about what can increase delay: long context, large model, server load, tool calls, or slow network.
Describe this without the diagram
This diagram explains response time as a timeline with four stages. First, the request travels to the service, where network delay can add waiting time. Second, the service prepares the input by formatting instructions, applying checks and turning text into tokens; longer conversation history increases this work. Third, the model generates the first token, which determines when the reply first becomes visible. Fourth, it generates the remaining tokens one by one, which affects how long the full answer takes. The main lesson is that waiting time comes from several stages, not one single cause.
With reduced motion: A static four-step horizontal timeline shows the same stages without animation: send request, prepare input, generate first token, generate remaining tokens. Short notes under each step list common causes of delay.
Latency is the waiting time. In a chat system, it usually means how long it takes from your prompt being sent until the first word appears, or until the full reply arrives.
What goes in is a request: your prompt, the earlier conversation, and system instructions. The service then prepares that text, runs the model, and starts generating output one token at a time. A token is a small piece of text, such as a word, part of a word, or punctuation.
What happens inside can take longer for several reasons. A longer prompt gives the system more text to process. A larger model may take more computation per token. Network delay, server load, safety checks, and tools such as web search can also add waiting time.
What comes out is the model's response, often streamed gradually. Latency is about delay, not quality. A fast answer can still be wrong, and a slower answer is not automatically better.
Latency is often confused with throughput. Latency asks, "How long do I wait for one reply?" Throughput asks, "How many replies or tokens can the system produce in a period of time?"
How “Planning a journey across a city” maps onto the real thing
| In the story | In the AI system |
|---|---|
| Your written route question | The prompt sent to the model |
| The staff member reading the note and checking the map | Preparing the request and running inference |
| The pause before the first direction is spoken | Time to first token |
| The pace of the rest of the directions | Rate at which output tokens are generated |
A worked example
The officer asks an approved assistant to draft three bullets from cleared, aggregate staffing updates for the donor report. The service log records when the request arrives, when the first words appear and when the final bullet is complete.
Starting from: A prompt plus the cleared staffing narrative and aggregate figures.
- The request travels to the service and may wait in a queue.
- The service processes and tokenises the prompt, applies required checks and retrieves any authorised supporting material.
- The model generates the first visible words. The elapsed time to this point is the time to first token.
- The model continues generating until the final bullet is complete. The total elapsed time is the time to last token.
Result: Both measurements are latency. One describes the wait before the reply starts; the other describes the wait for the complete reply. They can change independently, for example when retrieval delays the start or a longer answer extends generation.
In a United Nations workplace
At a shared services centre, a human resources officer is finalising the staffing narrative for a quarterly progress report to a donor. The front office wants it by close of business. An approved internal assistant is asked to turn cleared, aggregate recruitment updates into three concise points. The officer sees how long the tool takes to begin replying and how long it takes to finish.
The first delay determines whether the tool feels responsive. The time until the complete answer arrives determines whether it actually saves time before clearance. A longer prompt, retrieval from approved files, policy checks, server load and the length of the requested answer can affect either measure.
Before you try thisHuman resources material may contain personal, medical or performance information. Do not enter such material into any tool that has not been approved for it. Even in an approved system, use only authorised content, remove unnecessary identifiers and follow the applicable access and retention controls.
Where you may hear it
- During a vendor demonstration to an ICT governance board
- In a procurement unit’s technical evaluation of bids
- At a service review after an internal assistant slows near close of business
- In a performance discussion between a country office and Headquarters
Avoid confusion
Avoid confusion
Often confused with
Throughput
Throughput measures how much text a model can process or generate in a given time, usually counted in tokens per second.
Inference
Using a trained model on new input to produce an output such as a score, category, translation or generated text.
Completion
The text a language model produces after a prompt, built token by token from patterns learned during training.
Common misconceptions
Common misconception: Lower latency means the model is more accurate.
Correction: Not necessarily. Latency measures waiting time, not truthfulness or suitability. A system can answer quickly and still produce an incorrect or misleading completion. Accuracy depends on many other factors, including the prompt, context, model and whether tools are used well.
Common misconception: Latency is only caused by a slow internet connection.
Correction: Network delay is only one part. Time is also spent preparing the input, applying checks, running the model, generating tokens and sometimes calling external tools. A fast connection cannot remove those steps.
Common misconception: One latency number tells the whole story.
Correction: Often there are at least two useful numbers: time until the first token appears and time until the full answer finishes. A service may feel responsive on the first measure but still take a long time to complete a long reply.
Quick check
Which statement best describes latency in a chat assistant?
Latency is about delay. In chat systems, people often care about both time to first token and time to full response. Correctness, user capacity and training data are different questions.
Go deeper
Go deeper
The simple version
It is how long you wait after asking the model something before you start seeing the answer, or before the whole answer is finished.
The technical version
In large language model systems, latency usually refers to elapsed time between request submission and one or more output milestones, commonly time to first token and time to last token. It can include network transit, request queuing, prompt processing, tokenisation, safety or policy checks, retrieval or tool calls, and autoregressive generation during inference. Prompt length, context window use, model size, hardware, batching and server load can all change it. This differs from throughput, which measures how many requests or tokens a system can handle per unit time.
Reviewed external resources
EncyclopaediaLatency (opens in a new tab)Wikipedia
This link gives a broad general-purpose definition of latency and suits readers who want a quick orientation before diving into AI-specific use.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This link gives course-style background on how large language model systems run, which helps technical learners place latency within the wider inference pipeline.
Deeperlink checked 2026-07-31
VideoLarge language models, explained visually (opens in a new tab)3Blue1Brown
This link gives a visual explanation of large language models and suits readers who learn best from animations and intuition before technical detail.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Latency (opens in a new tab)YouTube search
This is a YouTube search results page, so it helps readers find different video explanations of latency for different levels rather than one guaranteed source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Latency (opens in a new tab)Reddit search
This is a Reddit search results page, so it helps readers browse community questions and practical experiences about latency, but posts vary in quality and should be checked carefully.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Latency may mean the wait until the first words appear or the wait until the reply is complete. Neither measure tells you whether the answer is good.