robots.txt
A file declaring which paths each user-agent may fetch.
What robots.txt does not show
It controls access, not indexing, and a 403 on robots.txt itself is read by most compliant crawlers as disallow-everything.
How CrawlCheck measures robots.txt
8 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
ROBOTS_NOT_200 | 6.1% | How to check whether AI crawlers and agents can read your site |
ROBOTS_BLOCKED | 0.4% | robots.txt 404, 403, 429 and 503: what each tells a crawler |
ROBOTS_IS_HTML | 0.2% | How to check whether AI crawlers and agents can read your site |
ROBOTS_IS_CATCHALL | 0.6% | robots.txt 404, 403, 429 and 503: what each tells a crawler |
ROBOTS_DISALLOW_ALL | 1.3% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_CTYPE | 0.1% | Does llms.txt actually do anything? |
ROBOTS_NO_SITEMAP | 8.4% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_RULES_SHADOWED | 12.6% | Do AI crawlers and agents respect robots.txt? |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where robots.txt comes up on this site
- Do AI crawlers and agents respect robots.txt?Asked plainly: the major named crawlers from OpenAI, Anthropic, Google, Microsoft and Apple publish their identities, publish IP ranges you can verify them against, and honour robots.txt directives.
- Meta robots, X-Robots-Tag and robots.txt: the three permission layers and the rule placed in the wrong oneThree different mechanisms are all called "robots", and they answer three different questions at three different moments. robots.txt is read before a request and decides whether a URL may be fetched at all.
- robots.txt 404, 403, 429 and 503: what each tells a crawlerEvery crawler that respects robots.txt fetches it before anything else, and the first thing it reads is not a rule.
- Cached robots.txt and llms.txt: stale, split, or blockedrobots.txt, llms.txt, sitemap.xml and an entity map are small files that change rarely, which is exactly the profile a CDN caches hardest.
- robots.txt user-agent groups: why named bots skip your rulesMost robots.txt files that go wrong do not look wrong.
Definitions that name robots.txt
homepage refused · Content-Signal · machine layer · blocked render resource · user-agent group · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · PerplexityBot vs Perplexity-User · CCBot · Bytespider · TDM exception · CAWG training and data mining assertion · Product token · Host politeness · Crawl-delay
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about robots.txt
What is robots.txt?
A file declaring which paths each user-agent may fetch.
What does robots.txt not show or guarantee?
It controls access, not indexing, and a 403 on robots.txt itself is read by most compliant crawlers as disallow-everything.
How does CrawlCheck measure robots.txt?
Through the finding codes ROBOTS_NOT_200, ROBOTS_BLOCKED, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_NO_SITEMAP, ROBOTS_RULES_SHADOWED, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does robots.txt belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← unverifiable · user-agent group →
See it in the full glossary · 731 terms across 19 areas.