Glossary · Crawlers and access
rate limit
A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.
What rate limit means
A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file. A limit keyed on address rather than identity throttles a verified crawler and a forger alike.
What rate limit can and cannot support
| It can support | It cannot support |
|---|---|
| A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. | For robots.txt, engines treat 429 as a server error rather than a missing file. A limit keyed on address rather than identity throttles a verified crawler and a forger alike. |
Where rate limit comes up on this site
- robots.txt 404, 403, 429 and 503: what each tells a crawlerA rate limit that catches robots.txt therefore has the opposite effect on Google from the one a 403 has.
- How to find broken internal links and dead anchor linksRun requests in parallel if the site can take it, but keep the count modest: a link audit that trips your own rate limit produces a list of 429s that are about the audit, not the links.
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · delivery comparison · honeypot
Questions about rate limit
What is rate limit?
A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.
What does rate limit not show or guarantee?
For robots.txt, engines treat 429 as a server error rather than a missing file. A limit keyed on address rather than identity throttles a verified crawler and a forger alike.
Which area of the glossary does rate limit belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← vantage point · delivery comparison →
See it in the full glossary · 668 terms across 19 areas.