Glossary · Crawlers and access
agentic traffic
Requests from agentic browsers, Comet, Atlas and their successors, in which a model drives a real browser session on a person's behalf. They arrive with an ordinary browser user-agent, cookies, and human-looking session patterns, from consumer addresses. No user-agent match, operator feed or reverse-DNS check can classify them, so they are invisible to every verification method in this glossary. This is a definable limit rather than a solved problem: a verified count is a count of what identified itself, and this traffic does not.
Terms this definition uses
Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · crawler classification
← Bytespider · crawler classification →
See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.