Meta-ExternalAgent vs GPTBot
In the census run that finished 2026-11-08, robots.txt closed the homepage to Meta-ExternalAgent on 645 of the 4,651 top sites that serve a robots.txt (13.9%), and to GPTBot on 756 (16.3%).
| meta-externalagent | GPTBot | |
|---|---|---|
| Operator | Meta | OpenAI |
| What it is | training crawler | training crawler |
| What it does | Meta AI | OpenAI training + index |
| How a request can be verified | Its operator publishes its crawler ranges through its autonomous system: no JSON feed — Meta publishes its crawler ranges as ASN AS32934 (whois -h whois.radb.net -- '-i origin AS32934'). A request that does not come from those ranges is not from it. | Its operator publishes the IP ranges it crawls from (https://openai.com/gptbot.json), so a request carrying the name can be checked against that list. |
| robots.txt closes the homepage (of sites with a robots.txt) | 645 (13.9%) | 756 (16.3%) |
| Refused at the door (of sites that answered a browser) | not requested as this identity | 1,093 (19.2%) |
Top 10,000 sites, census of 2026-11-08. “Refused at the door” is a non-2xx answer to a client declaring that name, from CrawlCheck’s own addresses, where the browser got 2xx. Every crawler on the radar.
Let Meta-ExternalAgent in and keep GPTBot out
User-agent: Meta-ExternalAgent
Allow: /
User-agent: GPTBot
Disallow: /A crawler obeys only the most specific group that names it and ignores * entirely. If your User-agent: * group has Disallow rules, repeat them under Meta-ExternalAgent, or naming it drops them. Swap the two names to do the reverse.
Check what your own site tells both: scan it free. Catalogue data CC BY 4.0 at /ai-crawlers.json.