Learn AI /Data and Infrastructure /Rate limit
Rate limit
- Starter
- Data and Infrastructure
- Technical readers
Also called: quota, throttling.
Back to the milestone you came from
In five seconds
What it is
A control that caps how many requests or how much use a service accepts within a set time, to keep systems available, fair and affordable.
In plain words
It is a speed cap for using a digital service: only so much can be sent in a certain time.
An everyday example
At an international conference centre, visitors enter through a security gate. The organisers allow only 30 people every 10 minutes through one entrance so the screening team does not get overwhelmed. If 50 people arrive at once, some pass through and the rest wait in a marked queue until the next time window. The rule is not about judging who is important; it is about keeping the entrance safe and orderly for everyone.
A real rate limit is usually enforced automatically by software, not by staff watching a door. It may count tokens, bytes or requests separately, and different users may have different limits at the same time. Also, some services reject excess requests immediately instead of making them wait in a queue.
Remember thisA rate limit controls how much use a service accepts within a set period. It helps manage availability, fair use and cost, but says nothing about answer quality.
Understand it
Understand it
Why it matters
For non-specialists, rate limits affect reliability, speed, fairness and cost. If you use AI tools at work, a limit can explain why a tool suddenly slows down or says 'too many requests' even though it was working a moment ago. If you buy or manage a service, limits shape user experience, budgeting and planning. They also help reduce misuse, such as automated scraping or denial-of-service behaviour. Knowing about rate limits helps you ask practical questions: What exactly is limited, per user or per organisation? Per minute or per day? Are excess requests queued, slowed or rejected? What is the fallback plan?
How a rate limit handles incoming requests
Show what happens when requests arrive below, near and above a service limit, and distinguish normal processing from delay or rejection.
01
1. Requests arrive
Users or apps send requests to a service during a set period, such as one minute.
Draw several small request icons moving toward a service box, with a clock icon labelled '1 minute' above them.
02
2. Usage is counted
The service tracks how many requests came from the same account, app or address in that period.
Show a counter beside the service box increasing from 1 up to a visible number, such as 60.
03
3. Rule is checked
The counter is compared with the allowed maximum, for example 'limit: 60 per minute'.
Place a decision diamond or comparison panel showing 'count' versus 'limit 60/min'.
04
4. Outcome
If the count is within the limit, requests are processed. If it is over the limit, extra requests are delayed or returned with a 'too many requests' response.
Split into two paths: a green path to 'processed' and an amber/red path to 'wait' or '429 too many requests'.
A left-to-right step diagram shows requests arriving, being counted within a time window, compared with a limit, and then either processed or blocked/delayed. Read from step 1 to step 4 to see the cause-and-effect chain.
Hover or tap each step to highlight the matching part of the request path and reveal one short example, such as '58 of 60: processed' and '61 of 60: delayed or rejected'.
Describe this without the diagram
This diagram explains rate limiting as a four-step process. First, requests from users or apps reach a service during a defined time period, shown as one minute. Second, the service counts requests from the same source, such as an account or app. Third, it compares that count with a rule like 60 requests per minute. Fourth, if the count is at or below the rule, the requests are processed. If the count is above the rule, the extra requests are delayed or rejected with a message such as 'too many requests'. The main idea is that the system measures usage over time and then controls access to protect capacity.
With reduced motion: A static four-panel diagram shows the same process without animation: requests arrive, the system counts them within a time window, compares the count with a fixed limit, and then sends them either to normal processing or to delay/rejection.
A rate limit sets a pace. It tells a service how many requests, messages or tokens it will accept from one user, app or organisation in a given time period, such as 60 requests per minute.
What goes in is incoming use: for example, people clicking a website, an app calling an AI service, or a script sending many API requests. The service counts this activity against a rule. The rule might be based on time, account type, Internet address, or a specific endpoint, which means a specific service address for one function.
What happens next depends on the count. If use stays under the limit, requests are processed normally. If use goes over the limit, extra requests may be delayed, slowed or rejected with a message such as “too many requests”. Rate limiting relates to throttling, which usually slows requests.
What comes out is not better answers. It is controlled access. The goal is to protect capacity, prevent one user from crowding out others, reduce abuse, and help manage cost. Throughput instead measures how much text a model processes or generates over time. A system can have high throughput and still impose a low rate limit per user.
How “Security gate at a busy conference” maps onto the real thing
| In the story | In the AI system |
|---|---|
| security gate | service or endpoint receiving requests |
| 30 people every 10 minutes | maximum allowed requests in a time window |
| queue or waiting line | delayed or throttled requests |
| screening team capacity | available compute and system resources |
A worked example
An approved internal translation tool calls a vendor endpoint capped at 120 requests per fixed one-minute window for its account.
Starting from: The tool sends 135 requests during the same one-minute window after several documents are submitted at once.
- The service counts requests associated with the tool's account during the fixed window.
- It accepts the first 120 requests, which remain within the cap.
- It rejects requests 121 to 135 and tells the tool when to retry.
- The tool queues the 15 rejected requests and sends them after the next window begins, rather than retrying immediately.
Result: The service accepts 120 requests in the first window. The remaining 15 are delayed, not lost, provided the tool handles the rejection and retry instruction correctly.
In a United Nations workplace
A procurement unit asks a protection officer to clear the operational section of a note for the file responding to an audit observation. Test logs show that an approved vendor service stopped accepting batch requests against public guidance documents shortly before close of business.
The response records that the service applied a per-minute cap to the organisation's account and rejected excess requests. It also sets out the corrective action: queue requests, follow the vendor's retry instruction and avoid sending every delayed request again at once.
Before you try thisDo not upload case files, names, contact details, witness accounts or bid material to a service that is not explicitly approved for those data. A rate limit is only a traffic control. It provides no confidentiality, privacy or data-protection assurance.
Where you may hear it
- During a vendor demo for a procurement evaluation panel
- At an ICT governance board review of service capacity and cost
- In an error log checked before a briefing deadline
- During an incident review after a portal rejects a burst of requests
Avoid confusion
Avoid confusion
Often confused with
Throughput
Throughput measures how much text a model can process or generate in a given time, usually counted in tokens per second.
Compute
Compute is the processing capacity used to train and run AI systems, with different needs before deployment and during everyday use.
Scalability
The ability of a system to handle more users, data or tasks without a serious drop in speed, quality, reliability or cost control.
Common misconceptions
Common misconception: A rate limit means the whole system is slow.
Correction: Not necessarily. A service may have strong overall capacity but still set per-user or per-app limits so one client does not consume too much at once. That is why rate limit and throughput are related but not the same.
Common misconception: If I pay more, there is no rate limit at all.
Correction: Higher-priced plans often raise limits, but many services still keep some cap to protect stability, fairness and cost control. The limit may be large enough that most users rarely notice it.
Common misconception: Rate limiting is only for security attacks.
Correction: It is also a normal operations tool. Organisations use it for predictable performance, fair sharing of compute, and budget control, even when traffic is legitimate.
Quick check
Which statement best describes a rate limit?
A rate limit is a usage rule over time, such as requests per minute. It is not a promise about response speed, not a data quality method, and not the same as overall system throughput.
Go deeper
Go deeper
The simple version
It is a speed cap for using a digital service: only so much can be sent in a certain time.
The technical version
A rate limit is a policy that measures incoming usage over time and enforces a maximum, such as requests per second, tokens per minute, or bytes per hour. The system identifies a subject to limit, such as an API key, user account, IP address, organisation or endpoint, then applies a method like fixed windows, sliding windows or token buckets. When usage exceeds the threshold, the service may reject the request, delay it, or ask the client to retry later. This differs from throughput, which measures text processed or generated over time, not a policy cap applied to a particular caller.
Reviewed external resources
Primary sourceAI Risk Management Framework (opens in a new tab)NIST
This NIST AI Risk Management Framework link helps readers place controls such as rate limits within wider risk management and governance, and suits policy, assurance and technical teams.
Intermediatelink checked 2026-07-31
Primary sourceOECD AI Principles (opens in a new tab)OECD
This OECD AI Principles link gives high-level principles for trustworthy AI that provide governance context for operational controls, and suits general readers and decision-makers.
Intermediatelink checked 2026-07-31
EncyclopaediaRate limiting (opens in a new tab)Wikipedia
This Wikipedia link gives a broad overview of rate limiting concepts and common methods, and suits readers who want a quick general introduction.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Rate limit (opens in a new tab)YouTube search
This YouTube search results page helps readers find different video explanations of rate limits, and suits people who prefer visual or spoken introductions.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Rate limit (opens in a new tab)Reddit search
This Reddit search results page helps readers see community questions and practical experiences about rate limits, and suits readers looking for informal discussion and troubleshooting examples.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
A rate limit controls how much use a service accepts within a set period. It helps manage availability, fair use and cost, but says nothing about answer quality.