Google-Extended
The robots.txt token that controls whether Google may use a site's content to train and ground its Gemini models. It is not a crawler; Googlebot still fetches the page.
What Google-Extended does not show
Blocking it does not remove a site from Google Search, from Discover, or from the retrieval path behind AI Overviews and AI Mode, which run on the Search index. It is the most commonly misread token on the web: sites block it expecting to leave AI answers and instead leave only the training set.
Where Google-Extended comes up on this site
- Google-Extended and Applebot-Extended: block AI trainingGoogle-Extended and Applebot-Extended tell each company how it may use pages its ordinary crawler already fetched.
- Google-Extended has no user agent. 1,582 requests used itThe name that is not a crawler Google-Extended is the robots.txt token that governs whether content Google has already crawled may be used to train and ground Gemini.
- Do AI crawlers and agents respect robots.txt?In our October 2026 census, 516 of 4,501 top sites with a robots.txt disallowed Google-Extended and 498 disallowed Applebot-Extended, two different companies' tokens refused at almost the same rate.
- AI crawler list: what each bot does and how to verify itGoogle-Extended and Applebot-Extended are never sent by their operators, so any request carrying one of those names is, by definition, somebody else.
- Content-Signal in robots.txt: search, ai-input, ai-trainIf you refuse training, back it with a Disallow: / group for the crawlers whose operators say they train, such as GPTBot and CCBot, plus Google's Google-Extended token, and make sure those groups do not drop your other rules, as explained in how named…
Terms this definition uses
robots.txt · token · retrieval · AI Overview · AI Mode
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Google-Extended
What is Google-Extended?
The robots.txt token that controls whether Google may use a site's content to train and ground its Gemini models. It is not a crawler; Googlebot still fetches the page.
What does Google-Extended not show or guarantee?
It is not a crawler; Googlebot still fetches the page. Blocking it does not remove a site from Google Search, from Discover, or from the retrieval path behind AI Overviews and AI Mode, which run on the Search index. It is the most commonly misread token on the web: sites block it expecting to leave AI answers and instead leave only the training set.
Which area of the glossary does Google-Extended belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← residential proxy · Google-Agent →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.