AI crawlers · Training crawlers

`Google-Extended`

Google-Extended is a robots.txt control token run by Google: Gemini training. It is a robots.txt control token, not a crawler: no request ever carries this name, so any request that claims it is not from Google.

User-agent token	`Google-Extended`
Operator	Google
What it does	Gemini training
Kind	robots.txt control token (never sent)
Can a request be verified?	No — none control token
In a CrawlCheck scan	Read from robots.txt; reported, never scored

Allow or block it in robots.txt

User-agent: Google-Extended
Disallow: /

User-agent: Google-Extended
Allow: /

Add a group that names it: "User-agent: Google-Extended" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Google-Extended, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want. Because it is a control token, the rule changes how pages may be used, not whether they are fetched.

What blocking it changes

No. It does not change whether pages are fetched; it tells Google whether pages fetched by its other crawlers may be used for gemini training.

What it actually did on crawlcheck.io

Verified requests	0
Requests using its name from outside its operator's ranges	49 (100% of claims)
robots.txt fetched	0 times
Fetches of paths robots.txt disallows	0
Observed	2026-08-15 to 2026-09-14

Measured on this site’s own traffic, verified against the operator’s published ranges. Every agent’s record: agent conduct.

Other agents from Google

Google-CloudVertexBot — Vertex AI grounding
Googlebot — Google Search / AI Overviews
Googlebot-Image — Google image crawler
Googlebot-News — Google News
Googlebot-Video — Google video crawler
Storebot-Google — Google Shopping
Mediapartners-Google — Google AdSense
APIs-Google — Google push delivery
GoogleOther — Google non-search fetch
AdsBot-Google — Google Ads quality
Gemini-Deep-Research — Gemini research agent
Google-Agent — Google user-triggered agent, acts on the web
Google-GeminiNotebook — Gemini Notebook user-supplied URL fetch
GoogleOther-Image — Google common crawler, public image URLs
GoogleOther-Video — Google common crawler, public video URLs

See what Google-Extended and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Google-Extended

QHow do I block Google-Extended in robots.txt?

QHow can I tell if a request from Google-Extended is real?

It is a robots.txt control token, not a crawler: no request ever carries this name, so any request that claims it is not from Google.

QDoes blocking Google-Extended affect whether my site appears in AI answers?

No. It does not change whether pages are fetched; it tells Google whether pages fetched by its other crawlers may be used for gemini training.

QDoes CrawlCheck test my site as Google-Extended?

Not as a request. CrawlCheck reads the rules your robots.txt sets for Google-Extended and reports them. Rules for it are reported and never scored, because refusing it is a policy choice.