rate limit
A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.
What rate limit does not show
A limit keyed on address rather than identity throttles a verified crawler and a forger alike.
Where rate limit comes up on this site
- robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlersA rate limit that catches robots.txt therefore has the opposite effect on Google from the one a 403 has.
- How to find broken internal links and dead anchor links on your siteRun requests in parallel if the site can take it, but keep the count modest: a link audit that trips your own rate limit produces a list of 429s that are about the audit, not the links.
- Which AI crawlers can't be verified by IP range?A rate limit or allow rule keyed on a name is lifted by changing the name, and cost or abuse attributed to a name is attributed to whoever typed it.
Terms this definition uses
Definitions that name rate limit
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · delivery comparison · honeypot
Questions about rate limit
What is rate limit?
A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.
What does rate limit not show or guarantee?
For robots.txt, engines treat 429 as a server error rather than a missing file. A limit keyed on address rather than identity throttles a verified crawler and a forger alike.
Which area of the glossary does rate limit belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← vantage point · delivery comparison →
See it in the full glossary · 792 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.