Guides · 2026-11-13 · By VSNARY | Emmanuel Orta · 8 views
robots.txt and AI fetchers: what seven operators say
A robots.txt rule stops a crawler that chooses to obey it. Five AI operators say their on-demand fetchers may not. Here is what each one publishes, and what to do about it.
A robots.txt rule does not reliably stop an AI fetcher that acts for a person who asked for one page. OpenAI, Perplexity, Amazon, Meta and Google each say so on their own crawler pages for ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, Google-Agent and Google-GeminiNotebook. Anthropic and Mistral say Claude-User and MistralAI-User do follow it. To keep the first group out, refuse them at the server as well.
Most advice about AI crawlers ends with a robots.txt snippet. For the crawlers that build an index or collect training data, that is the right tool: their operators say they obey it. A second kind of visitor is different. It fetches a single page because a person typed a question or pasted a link, and several operators say plainly that it may not follow robots.txt at all.
We read each operator's own crawler page on 13 November 2026. This guide reports what those pages say, in the operators' terms, and what follows for a site owner.
Two kinds of AI visitor #
The operators' reasoning is the same in each case: the request is treated as the person's own visit, made through the assistant, so the rules written for automated crawling are not applied to it.
What each operator publishes #
| Fetcher | Operator | What the operator's page says about robots.txt | Source |
|---|---|---|---|
| ChatGPT-User | OpenAI | robots.txt rules may not apply, because the visit is started by a user | OpenAI |
| Perplexity-User | Perplexity | The fetcher "generally ignores robots.txt rules" | Perplexity |
| Amzn-User | Amazon | May not follow every robots.txt directive | Amazon |
| Meta-ExternalFetcher | Meta | "may bypass robots.txt rules" because a user started the fetch | Meta |
| Google-Agent, Google-GeminiNotebook | Listed with the user-triggered fetchers, which Google says generally ignore robots.txt | ||
| Claude-User | Anthropic | Site owners can keep it out with a robots.txt rule | Anthropic |
| MistralAI-User | Mistral | Its robots.txt tag governs which sites these requests can be made to | Mistral |
Five operators, six fetchers, one message: do not rely on robots.txt for these. Two operators say the opposite about theirs. The difference is not visible from the name, which is why it has to be read from the source.
What a Disallow rule still does #
Writing the rule is still worth doing. It records your intent, it is honoured by the operators who say they honour it, and an operator's practice can be stricter than its wording. What it cannot be is your only control. If a page must not reach one of these fetchers, the refusal has to come from the server.
Refuse it at the server #
A server rule matches the user agent and answers 403 before any page is sent. We publish paste-ready rules for nginx, Apache, Caddy and Cloudflare on the ready-made policies page, generated from the crawler catalogue, and the steps are in Block AI crawlers in nginx, Apache and Caddy. Each robots.txt file there lists the fetchers above at the top, so nobody pastes it believing it is a lock.
A user agent is only a name, and anyone can send one. Where the operator publishes its address ranges, check the address before trusting the name. The ranges for the operators that publish them are mirrored in one file.
Should you block them at all? #
Think about who is on the other end. A user-triggered fetch is a person asking an assistant about your page, often with your link in the answer. Refusing it shows that person an error. Blocking a training crawler changes what a future model learned; blocking a fetcher changes what a reader sees today. Most sites that block training crawlers should leave fetchers alone.
What we changed in our own catalogue #
Until 13 November our catalogue treated a robots.txt rule against these fetchers the same way as a rule against an index crawler. That was wrong on the operators' own word, and we have corrected it: each fetcher's page now carries the operator's caveat and the source it was read from. The full account is in 14 of 43 crawlers in our catalogue were classed wrong.
CrawlCheck’s dataset when this page loaded: 570,959 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Does robots.txt block ChatGPT-User?
Not reliably. OpenAI's crawler page says robots.txt rules may not apply to ChatGPT-User because the visit is started by a person. To keep it out, refuse its user agent at the server or firewall as well.
Which AI fetchers may ignore robots.txt?
On their operators' own pages, read 13 November 2026: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Amzn-User (Amazon), Meta-ExternalFetcher (Meta), and Google-Agent and Google-GeminiNotebook (Google).
Which user-triggered AI fetchers do obey robots.txt?
Anthropic says site owners can keep Claude-User out with a robots.txt rule, and Mistral says its robots.txt tag governs which sites MistralAI-User requests can be made to.
How do I actually block a user-triggered AI fetcher?
Match its user agent in your web server or firewall and answer 403. Where the operator publishes address ranges, check the address too, because a user agent can be forged.
Will blocking a fetcher remove my site from AI answers?
It stops that assistant from reading your page when a person asks about it, so the person may see an error or an answer built without your page. It does not change what an index crawler or a training crawler has already collected.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.