crawler classification
The model edge providers adopted in 2026, Cloudflare from July with new-domain defaults changing on 15 September: verification confirms only who a crawler is, and access is then decided per behaviour class, Search, Agent, or Training, rather than per operator. A multi-purpose crawler is judged under every class it belongs to, which is how a rule that blocks Training can block a search crawler that also trains.
What crawler classification means
It is the industry name for the identity-versus-permission distinction that verified crawler already implies.
What crawler classification does not show
The classes are the provider's judgment of a crawler's purpose, and that judgment is not published in a form a site can audit.
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about crawler classification
What is crawler classification?
The model edge providers adopted in 2026, Cloudflare from July with new-domain defaults changing on 15 September: verification confirms only who a crawler is, and access is then decided per behaviour class, Search, Agent, or Training, rather than per operator. A multi-purpose crawler is judged under every class it belongs to, which is how a rule that blocks Training can block a search crawler that also trains.
What does crawler classification not show or guarantee?
The model edge providers adopted in 2026, Cloudflare from July with new-domain defaults changing on 15 September: verification confirms only who a crawler is, and access is then decided per behaviour class, Search, Agent, or Training, rather than per operator. The classes are the provider's judgment of a crawler's purpose, and that judgment is not published in a form a site can audit.
Which area of the glossary does crawler classification belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← agentic traffic · user agent →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.