Glossary · Crawlers and access
user-agent group
A block of robots.txt rules addressed to named agents. Most parsers apply only the most specific matching group, so naming an agent to allow it can silently remove every rule it would otherwise have followed.
Terms this definition uses
Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification
← robots.txt · AI opt-out →
See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.