Guides · 2026-10-26 · By VSNARY | Emmanuel Orta
Google-Extended and Applebot-Extended: opt out of AI training and stay in search
Neither is a crawler. Both are robots.txt names that decide how pages Google and Apple already crawl may be used for AI, and neither affects search ranking. How to use them, and how many top sites already do.
Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers. Disallowing Google-Extended stops Google from using pages it crawls for training Gemini models and for grounding, and Google states it does not affect inclusion or ranking in Google Search. Disallowing Applebot-Extended opts content out of training Apple's foundation models; Apple states such pages can still appear in search results and Spotlight, Siri and Safari. In October 2026, 516 of the 4,501 top sites with a robots.txt disallowed Google-Extended and 498 disallowed Applebot-Extended.
Blocking Googlebot to keep content out of AI would also remove a site from Google Search. Google and Apple each offer a narrower switch: a name you write in robots.txt that no crawler ever sends. Google-Extended and Applebot-Extended tell each company how it may use pages its ordinary crawler already fetched.
What each one controls #
| Google-Extended | Applebot-Extended | |
|---|---|---|
| Is it a crawler? | No. Google says it has no separate user-agent string; crawling is done by existing Google crawlers | No. Apple says it does not crawl webpages |
| What disallowing it changes | Content may not be used to train future Gemini models or for grounding in Gemini Apps, the Vertex AI API for Gemini and Grounding with Google Search on Vertex AI | Content may not be used to train Apple’s general-purpose foundation models |
| Effect on search | Google: it does not affect inclusion in Google Search and is not a ranking signal | Apple: pages can still appear in search results, and rules for it are not considered in ranking |
| Disallowed by, top 10,000 | 516 of 4,501 sites with a robots.txt (11.5%) | 498 of 4,501 (11.1%) |
Sources: Google’s crawler documentation and About Applebot, read on 26 October 2026. Census figures: CrawlCheck’s run of 24 October 2026.
The robots.txt lines #
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Leave Googlebot and Applebot as they are. A control token in its own group does not touch the rules other crawlers read. One caution: naming a crawler in robots.txt drops the * group’s rules for that crawler, but that does not matter here, because no request ever arrives under these two names.
What it does not do #
- It does not stop crawling. Googlebot and Applebot still fetch every page they are allowed to, so server logs will not change.
- It does not cover other companies. GPTBot, ClaudeBot, CCBot and the rest each need their own group; how often top sites block each one.
- It cannot be verified in logs. There is no request to check, only the policy each company states it follows.
To see what your robots.txt currently says to every AI crawler and control token, and what your server actually serves each crawler, run a free scan. The two side by side: Applebot-Extended vs Google-Extended and Google-Extended vs Googlebot.
The main product
Found this on your own site? We fix it for $500.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
Does blocking Google-Extended remove my site from Google Search?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal. It only controls whether crawled content may be used to train Gemini models and for grounding.
Is Applebot-Extended a crawler?
No. Apple states that Applebot-Extended does not crawl webpages; it only determines how data crawled by Applebot may be used, specifically for training Apple's foundation models.
How many sites block Google-Extended?
In CrawlCheck's October 2026 census, 516 of the 4,501 top-10,000 sites that serve a robots.txt (11.5%) disallowed Google-Extended, and 498 (11.1%) disallowed Applebot-Extended.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.