[Glossary](/glossary) · [Crawlers and access](/glossary/area/crawlers-and-access)

# rate limit

A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.

## What rate limit does not show

A limit keyed on address rather than identity throttles a verified crawler and a forger alike.

## Where rate limit comes up on this site

- [robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler)A rate limit that catches robots.txt therefore has the opposite effect on Google from the one a 403 has.
- [How to find broken internal links and dead anchor links on your site](/guides/how-to-audit-internal-links-and-dead-anchors)Run requests in parallel if the site can take it, but keep the count modest: a link audit that trips your own rate limit produces a list of 429s that are about the audit, not the links.
- [Which AI crawlers can't be verified by IP range?](/blog/the-crawlers-nobody-can-check)A rate limit or allow rule keyed on a name is lifted by changing the name, and cost or abuse attributed to a name is attributed to whoever typed it.

## Terms this definition uses

[robots.txt](/glossary/robots-txt) · [verified crawler](/glossary/verified-crawler)

## Definitions that name rate limit

[agent conduct record](/glossary/agent-conduct-record)

## Related terms in Crawlers and access

Who is fetching, whether they are who they claim, and what your rules actually permit.

[visitor resolve](/glossary/visitor-resolve) · [published crawler ranges](/glossary/published-crawler-ranges) · [operator origin](/glossary/operator-origin) · [signed agent](/glossary/signed-agent) · [agent arrival](/glossary/agent-arrival) · [agent conduct record](/glossary/agent-conduct-record) · [agent traffic index](/glossary/agent-traffic-index) · [user-triggered fetch exemption](/glossary/user-triggered-fetch-exemption) · [retrieval crawler](/glossary/retrieval-crawler) · [user-triggered fetch](/glossary/user-triggered-fetch) · [verified crawler](/glossary/verified-crawler) · [forged crawler identity](/glossary/forged-crawler-identity) · [unverifiable](/glossary/unverifiable) · [robots.txt](/glossary/robots-txt) · [user-agent group](/glossary/user-agent-group) · [AI opt-out](/glossary/ai-opt-out) · [Content-Signal](/glossary/content-signal) · [crawl budget](/glossary/crawl-budget) · [cloaking](/glossary/cloaking) · [challenge page at 200](/glossary/challenge-page-at-200) · [uniform refusal](/glossary/uniform-refusal) · [nonexistent-path control](/glossary/nonexistent-path-control) · [blocked render resource](/glossary/blocked-render-resource) · [off-host redirect](/glossary/off-host-redirect) · [homepage refused](/glossary/homepage-refused) · [FCrDNS](/glossary/fcrdns) · [Crawler trap](/glossary/crawler-trap) · [Conditional request](/glossary/conditional-request) · [ASN blocking](/glossary/asn-blocking) · [operator feed](/glossary/operator-feed) · [residential proxy](/glossary/residential-proxy) · [Google-Extended](/glossary/google-extended) · [Google-Agent](/glossary/google-agent) · [GPTBot vs OAI-SearchBot](/glossary/gptbot-oai-searchbot) · [ClaudeBot vs Claude-User](/glossary/claudebot-claude-user) · [PerplexityBot vs Perplexity-User](/glossary/perplexitybot-perplexity-user) · [CCBot](/glossary/ccbot) · [Bytespider](/glossary/bytespider) · [agentic traffic](/glossary/agentic-traffic) · [crawler classification](/glossary/crawler-classification) · [user agent](/glossary/user-agent) · [datacenter IP](/glossary/datacenter-ip) · [vantage point](/glossary/vantage-point) · [delivery comparison](/glossary/delivery-comparison) · [honeypot](/glossary/honeypot)

## Questions about rate limit

### What is rate limit?

A cap on how many requests a client may make in a window, answered with HTTP 429 when exceeded. For robots.txt, engines treat 429 as a server error rather than a missing file.

### What does rate limit not show or guarantee?

For robots.txt, engines treat 429 as a server error rather than a missing file. A limit keyed on address rather than identity throttles a verified crawler and a forger alike.

### Which area of the glossary does rate limit belong to?

Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.

← [vantage point](/glossary/vantage-point) · [delivery comparison](/glossary/delivery-comparison) →

[See it in the full glossary](/glossary/rate-limit) · 792 terms across 19 areas.

Free browser extension

See this on any site you visit, **in one click.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=glossary)
[What it does](/extension)

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [ARD trust](/spec/ard-trust) · [Resolve DNS](/spec/resolve-dns) | **Registry:** [MCP: Operator](/registry/mcp/operator) | **Datasets:** [Problems: Sitemap URL off host](/datasets/problems/sitemap-url-off-host) | **Blog:** [Release notes: 7 to 13 November 2026](/blog/release-notes-7-to-13-november-2026) | **Glossary:** [ClaudeBot vs Claude-User](/glossary/claudebot-claude-user) · [Sitemap hreflang](/glossary/sitemap-hreflang) · [Content hub](/glossary/content-hub) · [Claim signature](/glossary/claim-signature)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
