CrawlCheck

Guides · 2026-11-13 · By · 1 view

Why a robots.txt allow list cancels your Disallow rules

Naming a crawler in robots.txt to welcome it replaces every rule you wrote under User-agent: * for that crawler. The fix is one of two small changes.

In robots.txt a crawler obeys only the most specific group that names it. If you add a group such as User-agent: GPTBot with Allow: /, GPTBot stops reading your User-agent: * group, so every Disallow line there no longer applies to it. To welcome a crawler without opening closed paths, repeat your Disallow lines inside its group, or do not name it at all.

Check your own site free

About a minute. First scan free with your email, nothing installed.

A popular way to show AI crawlers they are welcome is to paste a block that names each one and says Allow: /. It looks harmless. For every crawler it names, it cancels the rules you already had.

How a crawler picks its rules #

The standard, RFC 9309, says a crawler finds the group whose user agent matches its own name most specifically and obeys that group. Only if no group names it does it fall back to User-agent: *. Groups are not added together. A crawler that has its own group never reads the wildcard one.

BEFORE · GPTBOT IS NOT NAMEDUser-agent: *Disallow: /checkout/GPTBot obeys itstays out of /checkout/AFTER · A GROUP NAMES GPTBOTUser-agent: GPTBot · Allow: /GPTBot obeys only this/checkout/ is now open to it

A worked example #

User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
Allow: /
CrawlerGroup it obeys/checkout/
GooglebotUser-agent: * (not named anywhere)closed
GPTBotits own groupopen
ClaudeBotits own groupopen

The author meant "everyone stays out of checkout, and AI crawlers are welcome". The file says "everyone except the AI crawlers stays out of checkout".

We made this mistake ourselves #

Earlier this year we added exactly this kind of block to our own sites as part of a robots.txt clean-up. It cancelled every Disallow line for every crawler we had named, and nothing looked wrong: the file was valid and every crawler was still allowed at the homepage. We found it only when we checked a closed path crawler by crawler. Our scanner has reported the pattern on any site since, and the share of scanned sites that have it is on the data page.

Two fixes #

Repeat the rules. If you want a named group, copy your Disallow lines into it. The group is then complete on its own.

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /checkout/
Disallow: /account/
Allow: /

Do not name what you only want to allow. A crawler that is not named follows User-agent: *, and if that group does not block it, it is allowed. An allow list adds nothing a crawler needs.

Why our ready-made files only block #

The files on our ready-made policies page contain only groups that end in Disallow: /. A crawler that is fully blocked has no closed paths left to open, so naming it is safe. We do not publish an allow list, for the reason on this page. For fetchers that may not read robots.txt at all, see what seven operators say.

Check your own file #

Pick a path you have closed and ask, for each crawler your file names, whether it is still closed. Our free robots.txt tool does this for every named crawler in one request and shows the line that decided each answer.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Does naming a crawler in robots.txt override User-agent: *?

Yes. A crawler obeys only the most specific group that names it. Once a group names it, the rules under User-agent: * no longer apply to that crawler.

Do I need Allow: / for AI crawlers to visit my site?

No. A crawler that is not named follows the User-agent: * group, and if that group does not block it, it is allowed. An explicit Allow: / group adds nothing and can cancel your other rules.

How do I allow GPTBot but keep it out of private paths?

Give GPTBot its own group and repeat every Disallow line you want it to follow inside that group, then add Allow: /.

Are robots.txt groups combined for a crawler?

No. Under RFC 9309 a crawler uses the single most specific matching group. Rules from other groups, including the wildcard group, are not merged in.

Is it safe to name a crawler in order to block it?

Yes. A group that ends in Disallow: / blocks everything for that crawler, so there are no closed paths left for the group to open.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works