user-agent group
A block of robots.txt rules addressed to named agents.
What user-agent group does not show
Most parsers apply only the most specific matching group, so naming an agent to allow it can silently remove every rule it would otherwise have followed.
Where user-agent group comes up on this site
- AI crawler list: what each bot does and how to verify itA crawler obeys only the User-agent group that matches it best and ignores every other group, including * .
- Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexingThen check that every named user-agent group in robots.txt carries the rules you think it does, because a named group replaces the default group rather than adding to it.
- Naming a crawler cancels your * rules: Semrush's robots.txtA crawler reads the User-agent group that matches it most specifically and ignores every other group, including * .
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about user-agent group
What is user-agent group?
A block of robots.txt rules addressed to named agents.
What does user-agent group not show or guarantee?
Most parsers apply only the most specific matching group, so naming an agent to allow it can silently remove every rule it would otherwise have followed.
Which area of the glossary does user-agent group belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← robots.txt · AI opt-out →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.