CrawlCheck

Agent view · Guides · 2026-11-13

Why a robots.txt allow list cancels your Disallow rules

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/robots-txt-allow-list-cancels-your-disallow-rules: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/robots-txt-allow-list-cancels-your-disallow-rules>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes11,838
words1,663
sha-256f721c25012af0f50341f10dce30e0e12946dffff76f8d6afe0ae73fede26d86b
repr-digestsha-256=:9yHCUBKvD1A0HxDc4w4OEpRt//92+Nav4K5z/t4m2Gs=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795181423;expires=1795185023;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="Mj_cdxELthAYTNXXeiIyVA";tag="crawlcheck-page"
signaturesig1=:wERJTiEzd56sfZKACKdcTaHAn6GjixL1YCk+tnPbmQvPRRN9xNs6aQNSMtABurvD4K6MbpgCnVnu46v/fgTRCg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3599 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author) · 1 view

# Why a robots.txt allow list cancels your Disallow rules

Naming a crawler in robots.txt to welcome it replaces every rule you wrote under User-agent: * for that crawler. The fix is one of two small changes.

In robots.txt a crawler obeys only the most specific group that names it. If you add a group such as User-agent: GPTBot with Allow: /, GPTBot stops reading your User-agent: * group, so every Disallow line there no longer applies to it. To welcome a crawler without opening closed paths, repeat your Disallow lines inside its group, or do not name it at all.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- How a crawler picks its rules
- A worked example
- We made this mistake ourselves
- Two fixes
- Why our ready-made files only block
- Check your own file

A popular way to show AI crawlers they are welcome is to paste a block that names each one and says `Allow: /`. It looks harmless. For every crawler it names, it cancels the rules you already had.

## How a crawler picks its rules #

The standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), says a crawler finds the group whose [user agent](/glossary/user-agent) matches its own name most specifically and obeys that group. Only if no group names it does it fall back to `User-agent: *`. Groups are not added together. A crawler that has its own group never reads the wildcard one.

## A worked example #

```
User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
Allow: /
```

| Crawler | Group it obeys | /checkout/ |
| --- | --- | --- |
| Googlebot | User-agent: * (not named anywhere) | closed |
| GPTBot | its own group | open |
| ClaudeBot | its own group | open |

The author meant "everyone stays out of checkout, and AI crawlers are welcome". The file says "everyone except the AI crawlers stays out of checkout".

## We made this mistake ourselves #

Earlier this year we added exactly this kind of block to our own sites as part of a [robots.txt](/glossary/robots-txt) clean-up. It cancelled every Disallow line for every crawler we had named, and nothing looked wrong: the file was valid and every crawler was still allowed at the homepage. We found it only when we checked a closed path crawler by crawler. Our scanner has reported the pattern on any site since, and the share of scanned sites that have it is on the [data page](/data).

## Two fixes #

**Repeat the rules.** If you want a named group, copy your Disallow lines into it. The group is then complete on its own.

```
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /checkout/
Disallow: /account/
Allow: /
```

**Do not name what you only want to allow.** A crawler that is not named follows `User-agent: *`, and if that group does not block it, it is allowed. An allow list adds nothing a crawler needs.

## Why our ready-made files only block #

The files on our [ready-made policies page](/robots-policies) contain only groups that end in `Disallow: /`. A crawler that is fully blocked has no closed paths left to open, so naming it is safe. We do not publish an allow list, for the reason on this page. For fetchers that may not read robots.txt at all, see [what seven operators say](/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers).

## Check your own file #

Pick a path you have closed and ask, for each crawler your file names, whether it is still closed. Our free [robots.txt tool](/tools) does this for every named crawler in one request and shows the line that decided each answer.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-allow-list-cancels-your-disallow-rules%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-allow-list-cancels-your-disallow-rules%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Why%20a%20robots.txt%20allow%20list%20cancels%20your%20Disallow%20rules)[Email](mailto:?subject=Why%20a%20robots.txt%20allow%20list%20cancels%20your%20Disallow%20rules&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-allow-list-cancels-your-disallow-rules)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Does naming a crawler in robots.txt override User-agent: *?

Yes. A crawler obeys only the most specific group that names it. Once a group names it, the rules under User-agent: * no longer apply to that crawler.

### Do I need Allow: / for AI crawlers to visit my site?

No. A crawler that is not named follows the User-agent: * group, and if that group does not block it, it is allowed. An explicit Allow: / group adds nothing and can cancel your other rules.

### How do I allow GPTBot but keep it out of private paths?

Give GPTBot its own group and repeat every Disallow line you want it to follow inside that group, then add Allow: /.

### Are robots.txt groups combined for a crawler?

No. Under RFC 9309 a crawler uses the single most specific matching group. Rules from other groups, including the wildcard group, are not merged in.

### Is it safe to name a crawler in order to block it?

Yes. A group that ends in Disallow: / blocks everything for that crawler, so there are no closed paths left for the group to open.

## Related findings

[2026-10-02 Naming a crawler in robots.txt drops your Disallow rules A crawler obeys only the robots.txt group that names it and ignores User-agent: *. A named Allow: / group voids your Disallow rule](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules)[2026-09-12 Naming a crawler cancels your * rules: Semrush's robots.txt A crawler obeys only the robots.txt group that names it, ignoring the * rules. Hand-verified on three of 21 audit tools, plus our](/blog/fifty-four-rules-that-do-not-apply-to-bingbot)[2026-08-28 AI crawler list: what each bot does and how to verify it GPTBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot: what each AI crawler does, which publish IP ranges, and how to allow one jo](/guides/ai-crawler-list-what-each-one-does)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) | **Blog:** [AI agents cannot refuse what they cannot see: 27 traps](/blog/ai-agents-cannot-refuse-what-they-cannot-see) | **AI crawlers:** [Blexbot](/ai-crawlers/blexbot) | **Glossary:** [AI classifiers: Temporal analysis](/glossary/ai-classifiers/temporal-analysis) · [Corpus](/glossary/corpus) · [DNS timeout](/glossary/dns-timeout) · [Broken link](/glossary/broken-link) · [NAP schema](/glossary/nap-schema) · [content negotiation](/glossary/content-negotiation)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/robots-txt-allow-list-cancels-your-disallow-rules.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.