Ready-made robots.txt and server rules
Pick what you want to keep out and paste the file. Every file is generated from the crawler catalogue, so it changes when the catalogue does.
| Block training and corpus crawlers, keep search and answer engines | robots.txt · nginx · Apache · Caddy · Cloudflare |
|---|---|
| Block training crawlers and AI answer engines, keep classic search | robots.txt · nginx · Apache · Caddy · Cloudflare |
| Block SEO, advertising and monitoring crawlers | robots.txt · nginx · Apache · Caddy · Cloudflare |
Two things to know before you paste
These files only block. A robots.txt group that names a crawler replaces your User-agent: * rules for that crawler, so a file that named crawlers just to allow them would quietly cancel your own Disallow lines for each one.
robots.txt is a request, not a lock. Some fetchers act for a person who asked for one page, and their operators say such a fetch may ignore robots.txt; each robots.txt file here lists them at the top. To keep those out, use the server rule as well.
For programs
This page as JSON · the catalogue · what changed in it · operator-published IP ranges in one file