AI crawlers · Training crawlers

`meta-externalagent`

meta-externalagent is the training crawler run by Meta: Meta AI. Its operator publishes its crawler ranges through its autonomous system: no JSON feed — Meta publishes its crawler ranges as ASN AS32934 (whois -h whois.radb.net -- '-i origin AS32934'). A request that does not come from those ranges is not from it.

User-agent token	`meta-externalagent`
Operator	Meta
What it does	Meta AI
Kind	training crawler
Can a request be verified?	Yes — asn range
Verification source	no JSON feed — Meta publishes its crawler ranges as ASN AS32934 (whois -h whois.radb.net -- '-i origin AS32934')
In a CrawlCheck scan	Sent as a request on every scan; reported, never scored

Allow or block it in robots.txt

User-agent: meta-externalagent
Disallow: /

User-agent: meta-externalagent
Allow: /

Add a group that names it: "User-agent: meta-externalagent" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names meta-externalagent, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

Other agents from Meta

Meta-ExternalFetcher — Meta AI live fetch
Facebot — Meta link crawler
meta-webindexer — Meta web index
facebookexternalhit — Facebook link preview
FacebookBot — Meta legacy

See what meta-externalagent and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about meta-externalagent

QHow do I block meta-externalagent in robots.txt?

QHow can I tell if a request from meta-externalagent is real?

Its operator publishes its crawler ranges through its autonomous system: no JSON feed — Meta publishes its crawler ranges as ASN AS32934 (whois -h whois.radb.net -- '-i origin AS32934'). A request that does not come from those ranges is not from it.

QDoes blocking meta-externalagent affect whether my site appears in AI answers?

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

QDoes CrawlCheck test my site as meta-externalagent?

Yes. Every CrawlCheck scan fetches the homepage as meta-externalagent and compares the response with what a browser receives. Rules for it are reported and never scored, because refusing it is a policy choice.