user agent
The string a client sends naming itself, in the User-Agent request header. It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
What user agent means
The name is what a site filters on; the source address is what verifies it.
Where user agent comes up on this site
- Google-Extended has no user agent. 1,582 requests used itGoogle's crawler documentation is explicit about it: Google-Extended does not have a separate HTTP request user agent string, crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity.
- How to rank in ChatGPT: the part that is in your controlCheck it: request your homepage once with a browser user agent and once with GPTBot's, and compare status and size.
- How to block AI crawlers like GPTBot and ClaudeBotThe AI crawler list has every user agent and its purpose. robots.txt a request honoured by operators that choose to Network rule a refusal enforced on verified identity, not the name Content-use signal a statement of intent what may be done after the fetch…
- HTTP status codes for crawlers: what a 200 can hideSet ua to your own browser's user-agent string first (search "what is my user agent" and paste it).
- How an AI agent should verify a website before it actsWhen the domain sends its own agents, signed requests under Web Bot Auth (built on RFC 9421 HTTP Message Signatures ) identify them better than a user agent string anyone can type.
Definitions that name user agent
visitor resolve · operator origin · signed agent
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about user agent
What is user agent?
The string a client sends naming itself, in the User-Agent request header. It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
What does user agent not show or guarantee?
It is a claim, not an identity: anyone can send any string, so treating a crawler's name as proof of the crawler is the mistake behind most forged-crawler traffic.
Which area of the glossary does user agent belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← crawler classification · datacenter IP →
See it in the full glossary · 792 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.