Learn AI /Data and Infrastructure /Compute

Compute

Starter
Data and Infrastructure
Policy officers
Managers & front offices

Also called: computing power, training compute, inference compute.

Detail

In five seconds

What it is

Compute is the processing capacity used to train and run AI systems, with different needs before deployment and during everyday use.

In plain words

Compute is the amount of machine processing needed to build an AI system and to run it afterwards. Training compute is the big effort to create or adapt the model; inference compute is the smaller but repeated effort each time someone uses it.

An everyday example

A youth club makes posters with a screen-printing press. Preparing the stencil takes concentrated effort: the design is tested, adjusted and fixed before printing begins. Once the stencil is ready, each poster still requires press time. Ten posters take little time, but thousands keep the press occupied, and a queue forms if everyone needs one at once. Preparing the stencil resembles training compute. Running each sheet through the press resembles inference compute. More presses can increase capacity, but they also cost more to operate.

Where the story stops working

A library index is built by people using explicit rules, while AI training usually adjusts many numerical parameters through repeated calculations. Also, a librarian can deliberately judge context in ways a model may not. The analogy helps show the difference between building and using a system, but it does not explain the mathematics or the limits of model output quality.

Remember thisSeparate the compute used to train or adapt a model from the compute used for every output. One is usually concentrated before deployment; the other accumulates with use.

This term is contested

The broad idea is stable, but measurement is not fully standardised. People may use 'compute' to mean hardware capacity, processor time, floating-point operations, service throughput, or the money spent to obtain them, so the intended metric should be stated.

Understand it

Why it matters

Non-specialists hear claims about powerful models, but those claims often hide a practical question: how much processing work is required, when, and at what cost. Compute affects whether an AI project is affordable, how quickly it responds, how many people can use it at once, and whether an organisation depends on scarce infrastructure or external providers. It also matters for procurement, environmental impact, continuity planning and equity, because systems that require very large amounts of compute may be harder for smaller organisations or lower-resource regions to access. Understanding the difference between training compute and inference compute helps leaders ask better questions before approving or scaling a system.

Training compute and inference compute

Show the difference between compute used to build a model and compute used each time the model is used, and explain how both affect cost and speed.

01
Training inputs
Examples, model settings and processing resources are prepared for a training run.
Left column top: icons for data files, model settings sliders and compute chips feeding into a box labelled 'training run'.
02
Repeated training work
The system performs many rounds of calculation to adjust the model. This often uses large amounts of compute over a limited period.
Inside the left column box, show looping arrows around small numbered passes, with labels 'many repetitions', 'high compute', 'hours to months'.
03
Trained model output
The result of training is an updated model that can later be deployed.
Left column bottom: arrow to a sealed model box labelled 'trained model'.
04
Inference inputs
After deployment, a real user request is sent to the trained model.
Right column top: prompt bubble and trained model box entering a box labelled 'inference'.
05
One response per request
Inference uses compute every time the system produces an answer, image label or prediction. Small per request costs can add up at scale.
Right column middle and bottom: one arrow to one output bubble, then duplicate request-output pairs fading behind it to show repetition across many users.
06
Decision view
Training compute is usually a big, occasional investment. Inference compute is an ongoing operational demand that affects speed, cost and scalability.
Bottom strip spanning both columns with three labelled markers: 'build cost', 'run cost', 'user speed'.

A two-column comparison. Left column shows training: data enters, many repeated processing rounds happen, and an updated model comes out. Right column shows inference: a user input enters the trained model, one response comes out, and this repeats for every request. Read top to bottom in each column, then compare the cost and timing labels at the bottom.

Hover or tap each column to highlight its inputs, processing pattern and output; a toggle switches the bottom strip between 'single request' and 'many requests'.

Describe this without the diagram

This diagram compares two uses of compute. On the left, training starts with data, model settings and processing resources. These enter a training run that repeats many calculations, shown as loops, and produces a trained model. Labels emphasise high compute use over a limited period. On the right, inference starts with a user request plus the trained model. These enter one inference step and produce one output. The pattern repeats for every new request. A final summary line says training compute is mainly a build-stage cost, while inference compute is a continuing run-stage cost that affects speed and scale.

With reduced motion: A static two-column chart replaces loops and repeated animations with labelled arrows and repeated icons. The left column shows training inputs, repeated passes and a trained model. The right column shows a request, the trained model and one output, plus a row of repeated request-output pairs to indicate ongoing use.

Compute means the processing capacity available to an AI system or the processing work it uses. In practice, it is often described through processor time or numbers of operations. Chips, memory and system configuration affect how much compute can be delivered. Electricity and money are related requirements and costs, not compute itself. More compute can let a system handle larger models or more requests, but it does not guarantee better results.

Two related terms are often mixed together. Training compute is the compute used while building a model from examples. This stage can take days or months and usually happens only sometimes. Inference compute is the compute used after deployment, when the model produces an output for a real prompt, image or other input. This stage happens every time someone uses the system.

During training, the inputs include training data, model settings and an initial or existing model. Repeated calculations adjust the model parameters. During inference, the inputs include a user request and the trained model. Specialised hardware performs many mathematical operations. The result is an updated model after training or a prediction, classification or generated response during inference.

Compute is related to, but different from, the hardware itself. A GPU or CPU is a type of processor; compute is the processing capacity available or work consumed on that hardware. It is also different from cloud computing, which is one way to rent and manage those resources, and from cost per token, which is one pricing measure for inference use.

For non-specialists, the key point is simple: compute affects cost, speed, scale and who can access advanced AI. It also has environmental implications because heavy compute use can require significant energy and cooling.

How “Cutting a stencil and printing posters” maps onto the real thing

In the story	In the AI system
Weeks spent building and correcting the library index	Training compute used to build or update a model
The finished index used at the desk	A trained model ready for inference
A visitor asking for one book	One user request or input at inference time
More desks or staff for busy periods	More inference compute to handle many requests quickly

A worked example

For the annual workplan review, a desk officer compares the initial and recurring compute requirements of a translation assistant for public, non-sensitive statements.

Starting from: Fine-tuning uses 200 accelerator-hours. After deployment, each request uses about 2 seconds of accelerator time. The section expects 15,000 requests per month.

Fine-tuning consumes 200 accelerator-hours through repeated forward and backward passes that update the model parameters.
Once deployed, the fixed model produces each translation through a forward pass, using about 2 seconds of inference compute.
At 15,000 requests, monthly inference uses about 30,000 seconds, or 8.3 accelerator-hours.
At 300,000 requests, monthly inference rises to about 166.7 accelerator-hours. The original training requirement does not rise unless the model is trained again.

Result: At pilot scale, training dominates the compute estimate. At wider use, recurring inference becomes the larger operational consideration, affecting the workplan line, hosted capacity and response times.

In a United Nations workplace

During the annual workplan review, a desk officer in a Headquarters department examines a proposed translation assistant for cleared statements for delivery. The workplan covers initial adaptation and evaluation, then routine use by the section before close of business deadlines.

The desk officer needs separate estimates for training compute and inference compute. The first affects the implementation line and timetable. The second determines recurring hosting costs, response time and whether the service can absorb a sudden queue of statements without turning clearance into a waiting room.

Before you try thisOnly public, non-sensitive text may be used in this example. Drafts under clearance, personal data and confidential or security-sensitive material must not be entered into a service that has not been approved for them, because the provider may retain, process or expose the content outside authorised arrangements.

Where you may hear it

An annual workplan review with the front office
A vendor demo during a procurement meeting
An ICT governance board reviewing hosting options
A budget discussion before a pilot is cleared
A service review after response times begin to slip

Avoid confusion

Often confused with

Common misconceptions

Common misconception: More compute always means a better AI system.
Correction: More compute can support larger training runs or faster service, but quality also depends on data quality, model design, evaluation and the task itself. A poorly designed system can consume large compute and still perform badly or unsafely.
Common misconception: Compute just means having a powerful chip.
Correction: A chip is hardware. Compute refers to the processing capacity available or used on that hardware over time, together with related resources such as memory, electricity and system configuration. The same hardware can deliver very different practical compute depending on setup and workload.
Common misconception: Once a model is trained, compute is no longer a major issue.
Correction: After training, inference compute continues every time the model is used. For popular services, the ongoing compute for serving users can become a major operational cost and can limit response speed during peak demand.

Quick check

Which statement best distinguishes training compute from inference compute?

Go deeper

The simple version

The technical version

Compute refers to the processing resources consumed to execute numerical operations, usually on accelerators or general processors, over time. In training, compute is used for repeated forward and backward passes that update model parameters using training data. In inference, compute is used for forward passes through a fixed model to produce outputs for new inputs. People sometimes measure compute indirectly through processor time, floating-point operations, throughput or service capacity, but the exact metric depends on context and is not fully standardised across systems. Compute is related to, but distinct from, hardware type, cloud deployment, latency, cost and energy use.

Reviewed external resources

Primary sourceAI Risk Management Framework (opens in a new tab)NIST
The NIST AI Risk Management Framework gives a governance lens for judging AI systems, useful for policy, leadership and risk teams considering compute-related trade-offs and controls.
Intermediatelink checked 2026-07-31
Primary sourceOECD AI Principles (opens in a new tab)OECD
The OECD AI Principles provide high-level guidance on trustworthy AI, helpful for leaders and public-sector readers connecting infrastructure choices to accountability and broader impacts.
Intermediatelink checked 2026-07-31
EncyclopaediaWikipedia search: Compute (opens in a new tab)Wikipedia
This Wikipedia search results page helps readers scan several possible meanings of the word 'compute', so it suits beginners who need orientation rather than a single definitive article.
search results, not one pagelink checked 2026-07-31
VideoVideo explanations: Compute (opens in a new tab)YouTube search
This YouTube search results page lets readers compare different video explanations of compute, which suits visual learners who want introductory explanations from multiple presenters.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Compute (opens in a new tab)Reddit search
This Reddit search results page shows community discussions and practical questions about compute, useful for readers who want to see common confusions and informal practitioner language.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Separate the compute used to train or adapt a model from the compute used for every output. One is usually concentrated before deployment; the other accumulates with use.