CrawlCheck

AI crawlers · Answer engines

Meta-ExternalFetcher

Meta-ExternalFetcher is the user-triggered fetch run by Meta: Meta AI live fetch. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.

User-agent tokenMeta-ExternalFetcher
OperatorMeta
What it doesMeta AI live fetch
Kinduser-triggered fetch
Can a request be verified?No — user agent string only
In a CrawlCheck scanRead from robots.txt; a refusal is scored

Allow or block it in robots.txt

User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: Meta-ExternalFetcher
Allow: /

Add a group that names it: "User-agent: Meta-ExternalFetcher" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Meta-ExternalFetcher, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

Yes. It is how Meta reaches a page to answer a question at the moment a person asks, so refusing it stops that engine reading the site.

Other agents from Meta

See what Meta-ExternalFetcher and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Meta-ExternalFetcher

QHow do I block Meta-ExternalFetcher in robots.txt?
Add a group that names it: "User-agent: Meta-ExternalFetcher" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Meta-ExternalFetcher, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
QHow can I tell if a request from Meta-ExternalFetcher is real?
Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.
QDoes blocking Meta-ExternalFetcher affect whether my site appears in AI answers?
Yes. It is how Meta reaches a page to answer a question at the moment a person asks, so refusing it stops that engine reading the site.
QDoes CrawlCheck test my site as Meta-ExternalFetcher?
Not as a request. CrawlCheck reads the rules your robots.txt sets for Meta-ExternalFetcher and reports them. A refusal of it counts toward the grade, because it costs the site visibility.