Agent view · Guides · 2026-11-13
robots.txt and AI fetchers: what seven operators say
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 13,827 |
| words | 1,852 |
| sha-256 | b8252b7ff94fa2c0ece04790bcbcf05f4dbf8ce6c9622814faa270c2cf6d81fd |
| repr-digest | sha-256=:uCUrf/lPosDs4EeQvLzwX02/jObJYigU+qJwws9tgf0=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795181428;expires=1795185028;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="0VbYn5jyr4TBqk-5C059qQ";tag="crawlcheck-page" |
| signature | sig1=:DG6BmUQ0As6l66O1eyV0zxSzAthg75MDPZfiZCAGlDRI9cRME7O0mhazcCgJmHuk+1n3ywQvxLAthQ7FZkvSAA==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author) · 8 views # robots.txt and AI fetchers: what seven operators say A robots.txt rule stops a crawler that chooses to obey it. Five AI operators say their on-demand fetchers may not. Here is what each one publishes, and what to do about it. A robots.txt rule does not reliably stop an AI fetcher that acts for a person who asked for one page. OpenAI, Perplexity, Amazon, Meta and Google each say so on their own crawler pages for ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, Google-Agent and Google-GeminiNotebook. Anthropic and Mistral say Claude-User and MistralAI-User do follow it. To keep the first group out, refuse them at the server as well. Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - Two kinds of AI visitor - What each operator publishes - What a Disallow rule still does - Refuse it at the server - Should you block them at all? - What we changed in our own catalogue Most advice about AI crawlers ends with a [robots.txt](/glossary/robots-txt) snippet. For the crawlers that build an index or collect [training data](/glossary/training-data), that is the right tool: their operators say they obey it. A second kind of visitor is different. It fetches a single page because a person typed a question or pasted a link, and several operators say plainly that it may not follow robots.txt at all. We read each operator's own crawler page on 13 November 2026. This guide reports what those pages say, in the operators' terms, and what follows for a site owner. ## Two kinds of AI visitor # The operators' reasoning is the same in each case: the request is treated as the person's own visit, made through the assistant, so the rules written for automated crawling are not applied to it. ## What each operator publishes # | Fetcher | Operator | What the operator's page says about robots.txt | Source | | --- | --- | --- | --- | | ChatGPT-User | OpenAI | robots.txt rules may not apply, because the visit is started by a user | [OpenAI](https://developers.openai.com/api/docs/bots) | | Perplexity-User | Perplexity | The fetcher "generally ignores robots.txt rules" | [Perplexity](https://docs.perplexity.ai/guides/bots) | | Amzn-User | Amazon | May not follow every robots.txt directive | [Amazon](https://developer.amazon.com/en/amazonbot) | | Meta-ExternalFetcher | Meta | "may bypass robots.txt rules" because a user started the fetch | [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) | | [Google-Agent](/glossary/google-agent), Google-GeminiNotebook | Google | Listed with the user-triggered fetchers, which Google says generally ignore robots.txt | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) | | Claude-User | Anthropic | Site owners can keep it out with a robots.txt rule | [Anthropic](https://support.claude.com/en/articles/8896518) | | MistralAI-User | Mistral | Its robots.txt tag governs which sites these requests can be made to | [Mistral](https://docs.mistral.ai/robots) | Five operators, six fetchers, one message: do not rely on robots.txt for these. Two operators say the opposite about theirs. The difference is not visible from the name, which is why it has to be read from the source. ## What a Disallow rule still does # Writing the rule is still worth doing. It records your intent, it is honoured by the operators who say they honour it, and an operator's practice can be stricter than its wording. What it cannot be is your only control. If a page must not reach one of these fetchers, the [refusal](/glossary/refusal) has to come from the server. ## Refuse it at the server # A server rule matches the [user agent](/glossary/user-agent) and answers 403 before any page is sent. We publish paste-ready rules for nginx, Apache, Caddy and Cloudflare on the [ready-made policies page](/robots-policies), generated from the [crawler catalogue](/ai-crawlers), and the steps are in [Block AI crawlers in nginx, Apache and Caddy](/guides/block-ai-crawlers-nginx-apache-caddy). Each robots.txt file there lists the fetchers above at the top, so nobody pastes it believing it is a lock. A user agent is only a name, and anyone can send one. Where the operator publishes its address ranges, check the address before trusting the name. The ranges for the operators that publish them are mirrored in [one file](/ai-crawlers/ip-ranges.json). ## Should you block them at all? # Think about who is on the other end. A [user-triggered fetch](/glossary/user-triggered-fetch) is a person asking an assistant about your page, often with your link in the answer. Refusing it shows that person an error. Blocking a training crawler changes what a future model learned; blocking a fetcher changes what a reader sees today. Most sites that block training crawlers should leave fetchers alone. ## What we changed in our own catalogue # Until 13 November our catalogue treated a robots.txt rule against these fetchers the same way as a rule against an index crawler. That was wrong on the operators' own word, and we have corrected it: each fetcher's page now carries the operator's caveat and the source it was read from. The full account is in [14 of 43 crawlers in our catalogue were classed wrong](/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong). CrawlCheck’s dataset when this page loaded: 570,959 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-does-not-stop-user-triggered-ai-fetchers%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-does-not-stop-user-triggered-ai-fetchers%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=robots.txt%20and%20AI%20fetchers%3A%20what%20seven%20operators%20say)[Email](mailto:?subject=robots.txt%20and%20AI%20fetchers%3A%20what%20seven%20operators%20say&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-does-not-stop-user-triggered-ai-fetchers)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Does robots.txt block ChatGPT-User? Not reliably. OpenAI's crawler page says robots.txt rules may not apply to ChatGPT-User because the visit is started by a person. To keep it out, refuse its user agent at the server or firewall as well. ### Which AI fetchers may ignore robots.txt? On their operators' own pages, read 13 November 2026: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Amzn-User (Amazon), Meta-ExternalFetcher (Meta), and Google-Agent and Google-GeminiNotebook (Google). ### Which user-triggered AI fetchers do obey robots.txt? Anthropic says site owners can keep Claude-User out with a robots.txt rule, and Mistral says its robots.txt tag governs which sites MistralAI-User requests can be made to. ### How do I actually block a user-triggered AI fetcher? Match its user agent in your web server or firewall and answer 403. Where the operator publishes address ranges, check the address too, because a user agent can be forged. ### Will blocking a fetcher remove my site from AI answers? It stops that assistant from reading your page when a person asks about it, so the person may see an error or an answer built without your page. It does not change what an index crawler or a training crawler has already collected. ## Related findings [2026-10-03 How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters m](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot)[2026-11-13 14 of 43 crawlers in our catalogue were classed wrong CrawlCheck classed 14 of 43 crawlers wrongly until 13 November 2026, including Amazonbot, msnbot and Yandex. What was wrong and wh](/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong)[2026-10-08 AI agents acting for people: 613 visits, one read robots.txt ChatGPT-User made 526 verified visits, Claude-User 79, Perplexity-User 8. Only Claude-User read robots.txt; none fetched a disallo](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [AI policy](/registry/aipolicy) · [jammiwoo.gr](/registry/jammiwoo.gr) · [patreon.com](/registry/patreon.com) · [whatsapp.com](/registry/whatsapp.com) · [agentalog.com](/registry/capabilities/agentalog.com) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) | **Datasets:** [Idaho](/datasets/state/idaho) · [Technologies: Divi](/datasets/technologies/divi) · [Datasets](/datasets) | **Blog:** [areaServed vs service area pages: 20 towns declared, 1 page](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/robots-txt-does-not-stop-user-triggered-ai-fetchers.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.