Crawler catalogue

# Ready-made robots.txt and server rules

Pick what you want to keep out and paste the file. Every file is generated from the [crawler catalogue](/ai-crawlers), so it changes when the catalogue does.

| Block training and corpus crawlers, keep search and answer engines | [robots.txt](/robots/block-ai-training.txt) · [nginx](/robots/block-ai-training.nginx.txt) · [Apache](/robots/block-ai-training.apache.txt) · [Caddy](/robots/block-ai-training.caddy.txt) · [Cloudflare](/robots/block-ai-training.cloudflare.txt) |
| --- | --- |
| Block training crawlers and AI answer engines, keep classic search | [robots.txt](/robots/block-all-ai.txt) · [nginx](/robots/block-all-ai.nginx.txt) · [Apache](/robots/block-all-ai.apache.txt) · [Caddy](/robots/block-all-ai.caddy.txt) · [Cloudflare](/robots/block-all-ai.cloudflare.txt) |
| Block SEO, advertising and monitoring crawlers | [robots.txt](/robots/block-seo-tools.txt) · [nginx](/robots/block-seo-tools.nginx.txt) · [Apache](/robots/block-seo-tools.apache.txt) · [Caddy](/robots/block-seo-tools.caddy.txt) · [Cloudflare](/robots/block-seo-tools.cloudflare.txt) |

## Two things to know before you paste

These files only block. A robots.txt group that names a crawler replaces your `User-agent: *` rules for that crawler, so a file that named crawlers just to allow them would quietly cancel your own Disallow lines for each one.

robots.txt is a request, not a lock. Some fetchers act for a person who asked for one page, and their operators say such a fetch may ignore robots.txt; each robots.txt file here lists them at the top. To keep those out, use the server rule as well.

## For programs

[This page as JSON](/robots/index.json) · [the catalogue](/ai-crawlers.json) · [what changed in it](/ai-crawlers/changes.json) · [operator-published IP ranges in one file](/ai-crawlers/ip-ranges.json)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [Place: Arvada](/registry/place/co/arvada) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Conduct: Google read aloud](/agents/conduct/google-read-aloud) | **Datasets:** [Maryland](/datasets/state/maryland) · [Technologies: Onetrust](/datasets/technologies/onetrust) · [Problems: Robots legacy agent only](/datasets/problems/robots-legacy-agent-only) | **Guides:** [Block AI crawlers nginx apache caddy](/guides/block-ai-crawlers-nginx-apache-caddy) | **Glossary:** [sitemap index](/glossary/sitemap-index) · [Exact-match retrieval](/glossary/exact-match-retrieval)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
