Google-Extended
The robots.txt token that controls whether Google may use a site's content to train and ground its Gemini models. It is not a crawler; Googlebot still fetches the page.
What Google-Extended does not show
Blocking it does not remove a site from Google Search, from Discover, or from the retrieval path behind AI Overviews and AI Mode, which run on the Search index. It is the most commonly misread token on the web: sites block it expecting to leave AI answers and instead leave only the training set.
Where Google-Extended comes up on this site
- The AI crawler and agent list: what each one is for, and which can be verifiedFour identities in the wider dataset came back at 100% forged: Google-Extended, Claude-SearchBot, anthropic-ai and Applebot-Extended.
- Content Signals in robots.txt: search, ai-input, ai-trainIf you refuse training, back it with a Disallow: / group for the crawlers whose operators state they are used for training, such as GPTBot and CCBot, plus Google's Google-Extended token, and make sure those groups do not drop your other rules, as explained…
- 1,582 requests arrived under a name Google says it never sendsThe name that is not a crawler Google-Extended is the robots.txt token that governs whether content Google has already crawled may be used to train and ground Gemini.
- Is your host blocking ClaudeBot? Group the 403s by IP firstThat address presented seven different crawler identities in a single day: Amazonbot, ChatGPT-User, ClaudeBot, OAI-SearchBot, GPTBot, Google-Extended and PerplexityBot.
- Fake GPTBot and ClaudeBot requests are scanning for .env and .ssh filesEvery request claiming Google-Extended failed verification: 10 claims, none from Google's published ranges.
Terms this definition uses
robots.txt · token · retrieval · AI Overview · AI Mode
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Google-Extended
What is Google-Extended?
The robots.txt token that controls whether Google may use a site's content to train and ground its Gemini models. It is not a crawler; Googlebot still fetches the page.
What does Google-Extended not show or guarantee?
It is not a crawler; Googlebot still fetches the page. Blocking it does not remove a site from Google Search, from Discover, or from the retrieval path behind AI Overviews and AI Mode, which run on the Search index. It is the most commonly misread token on the web: sites block it expecting to leave AI answers and instead leave only the training set.
Which area of the glossary does Google-Extended belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← residential proxy · Google-Agent →
See it in the full glossary · 731 terms across 19 areas.