Applebot-Extended vs Google-Extended
In the census run that finished 2026-10-24, robots.txt closed the homepage to Applebot-Extended on 498 of the 4,501 top sites that serve a robots.txt (11.1%), and to Google-Extended on 516 (11.5%).
| Applebot-Extended | Google-Extended | |
|---|---|---|
| Operator | Apple | |
| What it is | robots.txt control token (never sent) | robots.txt control token (never sent) |
| What it does | Apple training | Gemini training |
| How a request can be verified | It is a robots.txt control token, not a crawler: no request ever carries this name, so any request that claims it is not from Apple. | It is a robots.txt control token, not a crawler: no request ever carries this name, so any request that claims it is not from Google. |
| robots.txt closes the homepage (of sites with a robots.txt) | 498 (11.1%) | 516 (11.5%) |
| Refused at the door (of sites that answered a browser) | not requested as this identity | not requested as this identity |
Top 10,000 sites, census of 2026-10-24. “Refused at the door” is a non-2xx answer to a client declaring that name, from CrawlCheck’s own addresses, where the browser got 2xx. Every crawler on the radar.
Let Applebot-Extended in and keep Google-Extended out
User-agent: Applebot-Extended
Allow: /
User-agent: Google-Extended
Disallow: /A crawler obeys only the most specific group that names it and ignores * entirely. If your User-agent: * group has Disallow rules, repeat them under Applebot-Extended, or naming it drops them. Swap the two names to do the reverse.
Check what your own site tells both: scan it free. Catalogue data CC BY 4.0 at /ai-crawlers.json.