Glossary · Crawlers and access
ClaudeBot vs Claude-User
The same split for Anthropic. ClaudeBot crawls autonomously to improve models; Claude-User fetches a page because a person asked Claude about it, and Claude-SearchBot fetches to build and refresh search results. All three verify against Anthropic's published IP ranges. Blocking ClaudeBot does not stop a user-triggered fetch, and counting a Claude-User hit as a crawl inflates a crawl figure with visits that were, functionally, referrals.
Terms this definition uses
Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification
← GPTBot vs OAI-SearchBot · PerplexityBot vs Perplexity-User →
See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.