robots.txt
A file declaring which paths each user-agent may fetch.
What robots.txt does not show
It controls access, not indexing, and a 403 on robots.txt itself is read by most compliant crawlers as disallow-everything.
How CrawlCheck measures robots.txt
12 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
ROBOTS_NOT_200 | 5.3% | How to check if AI crawlers can read your site |
ROBOTS_BLOCKED | 0.5% | robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers |
ROBOTS_IS_HTML | 0.1% | How to check if AI crawlers can read your site |
ROBOTS_IS_CATCHALL | 0.5% | robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers |
ROBOTS_DISALLOW_ALL | 1.1% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_CTYPE | 0.1% | Does llms.txt work? What is known and how to test it |
ROBOTS_NO_SITEMAP | 8.6% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_RULES_SHADOWED | 13% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_LEGACY_AGENT_ONLY | 0.2% | dataset |
ROBOTS_GROUP_NAME_TYPO | 0.4% | dataset |
ROBOTS_NOINDEX_LINE | 0.1% | dataset |
ROBOTS_SITEMAP_RELATIVE | 0.1% | dataset |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where robots.txt comes up on this site
- Do AI crawlers and agents respect robots.txt?Asked plainly: the major named crawlers publish their identities, publish IP ranges you can verify them against, and honour robots.txt.
- Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexingThree different mechanisms are all called "robots", and they answer three different questions at three different moments. robots.txt is read before a request and decides whether a URL may be fetched at all.
- robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlersEvery crawler that respects robots.txt fetches it before anything else, and the first thing it reads is not a rule.
- Stale cached robots.txt or llms.txt? Two fetches tell yourobots.txt, llms.txt, sitemap.xml and an entity map are small files that rarely change, which is exactly what a CDN caches hardest.
- Naming a crawler in robots.txt drops your Disallow rulesMost robots.txt files that go wrong do not look wrong.
Definitions that name robots.txt
homepage refused · Content-Signal · machine layer · blocked render resource · user-agent group · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · PerplexityBot vs Perplexity-User · CCBot · Bytespider · TDM exception · CAWG training and data mining assertion · Product token · Host politeness · Crawl-delay
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about robots.txt
What is robots.txt?
A file declaring which paths each user-agent may fetch.
What does robots.txt not show or guarantee?
It controls access, not indexing, and a 403 on robots.txt itself is read by most compliant crawlers as disallow-everything.
How does CrawlCheck measure robots.txt?
Through the finding codes ROBOTS_NOT_200, ROBOTS_BLOCKED, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_NO_SITEMAP, ROBOTS_RULES_SHADOWED, ROBOTS_LEGACY_AGENT_ONLY, ROBOTS_GROUP_NAME_TYPO, ROBOTS_NOINDEX_LINE, ROBOTS_SITEMAP_RELATIVE, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does robots.txt belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← unverifiable · user-agent group →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.