Glossary · Crawlers and access
user agent
The string a client sends naming itself, in the User-Agent request header. It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
What user agent means
The string a client sends naming itself, in the User-Agent request header. It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic. The name is what a site filters on; the source address is what verifies it.
What user agent can and cannot support
| It can support | It cannot support |
|---|---|
| The string a client sends naming itself, in the User-Agent request header. The name is what a site filters on; the source address is what verifies it. | It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic. |
Where user agent comes up on this site
- 1,582 requests arrived under a name Google says it never sendsGoogle's crawler documentation is explicit about it: Google-Extended does not have a separate HTTP request user agent string, crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity.
- Which AI crawlers publish IP ranges, and which cannot be verifiedAn identity with no operator is not necessarily illegitimate — plenty of real tooling ships a user agent nobody bothered to write a page about.
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about user agent
What is user agent?
The string a client sends naming itself, in the User-Agent request header. It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
What does user agent not show or guarantee?
It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
Which area of the glossary does user agent belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← crawler classification · datacenter IP →
See it in the full glossary · 668 terms across 19 areas.