Glossary · Crawlers and access
published crawler ranges
The list of IP address ranges an operator publishes for a crawler, so that a request claiming the crawler's name can be checked against where it came from. OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish such lists; many operators do not.
What published crawler ranges means
The list of IP address ranges an operator publishes for a crawler, so that a request claiming the crawler's name can be checked against where it came from. OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish such lists; many operators do not. It can prove a request came from the operator's own network; it cannot prove anything about an agent whose operator publishes nothing, which is a limit of the evidence rather than a judgement.
What published crawler ranges can and cannot support
| It can support | It cannot support |
|---|---|
| The list of IP address ranges an operator publishes for a crawler, so that a request claiming the crawler's name can be checked against where it came from. | OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish such lists; many operators do not. It can prove a request came from the operator's own network; it cannot prove anything about an agent whose operator publishes nothing, which is a limit of the evidence rather than a judgement. |
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about published crawler ranges
What is published crawler ranges?
The list of IP address ranges an operator publishes for a crawler, so that a request claiming the crawler's name can be checked against where it came from. OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish such lists; many operators do not.
What does published crawler ranges not show or guarantee?
OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish such lists; many operators do not. It can prove a request came from the operator's own network; it cannot prove anything about an agent whose operator publishes nothing, which is a limit of the evidence rather than a judgement.
Which area of the glossary does published crawler ranges belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← visitor resolve · operator origin →
See it in the full glossary · 688 terms across 19 areas.