CrawlCheck

Guides · 2026-10-02 · By · 1 view

Naming a crawler in robots.txt drops your Disallow rules

A crawler obeys exactly one group in robots.txt, the most specific one that names it. Add a group for GPTBot with Allow: / and every Disallow under User-agent: * stops applying to GPTBot. How to see it, how to write the file so it cannot happen, and the full-block shape that only looks like the mistake.

Under RFC 9309 a crawler follows exactly one group in robots.txt: the group whose User-agent line matches it most specifically, or User-agent: * if none names it. Groups are not combined, so adding User-agent: GPTBot with only Allow: / removes every Disallow rule in the * group for GPTBot. The fix is to repeat the full * rule set inside every named group, or list several agents above one shared block. A named group with Disallow: / is safe: it blocks every path, so nothing is lost.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Most robots.txt files that go wrong do not look wrong. Every line parses, every path is spelled correctly, and the homepage is reachable for every crawler you care about. The failure is structural: someone added a block for a named crawler, usually an AI crawler, and that block quietly switched off the rules everyone assumed still applied. This guide covers the one rule behind it, how to check a file in under a minute, how to write named groups that keep your exclusions, and the one shape that looks like this mistake and is not.

How a crawler picks one robots.txt group and ignores the restTHE FILEUser-agent: *Disallow: /wp-admin/Disallow: /?s=Disallow: /feed/User-agent: GPTBotAllow: /WHAT GPTBOT OBEYSGPTBot group onlymost specific match* group ignored entirelyRESULT FOR GPTBOT/wp-admin/ open/?s= open/feed/ openRepeat the Disallows hereA named group does not add to *. It replaces it for that crawler. Repeat every * rule inside each named group.

The rule: one crawler, one group #

RFC 9309, the Robots Exclusion Protocol, says a crawler must obey the group whose User-agent line best matches its product token. If more than one group names the same crawler, those groups are merged into one. If no group names it, it obeys the User-agent: * group. What the standard does not do is combine a named group with the * group. The moment a group names a crawler, the * group no longer exists for that crawler. Google's robots.txt documentation describes the same behaviour for its crawlers.

So a file that reads User-agent: * followed by ten Disallow lines, and then User-agent: GPTBot followed by Allow: /, does not mean "everyone follows the ten rules and GPTBot is also welcome". It means GPTBot may fetch everything, including the ten paths you excluded. The same is true for any named crawler, including Googlebot and Bingbot.

Why is it so common now? #

The pattern spread with AI crawler allowlists. Plugin snippets and blog advice recommend an explicit Allow: / block for GPTBot, ClaudeBot, PerplexityBot and a dozen others so answer engines can reach the site. Each block is harmless alone. Together they create a dozen more specific groups, each of which deletes the * rules for one crawler, so search, feed, login, cart and parameter URLs kept out on purpose become open to exactly the crawlers the edit meant to welcome. We made this mistake ourselves: on sites we operate, the allowlist we deployed shadowed every Disallow rule for every crawler it named, and our own check graded the files clean because each named crawler could reach the homepage. Fifty-four rules that do not apply to Bingbot shows the same shape on a large public file.

The share of scanned sites where at least one named group drops rules that * carries: 14%.

How to check a file by hand #

Open the file at /robots.txt and walk it group by group. A group starts at one or more consecutive User-agent lines and runs until the next User-agent line that follows a rule. For each named group, list its Disallow lines and compare them with the * group. Every * Disallow that is missing from a named group is a rule that crawler does not follow.

File shapeWhat the named crawler obeysRules lost
* group onlythe * groupnone
* group + User-agent: GPTBot / Allow: /only Allow: /every * Disallow, for GPTBot
* group + named group that repeats every Disallowits own copy of the rulesnone
* declared twice, rules in the second blockboth * blocks, mergednone, but easy to misread
named group with its own different Disallowsonly its own Disallowsany * rule not repeated
/robots.txt returns the site's 404 page as 200nothing: no file existsall of them

Two traps make hand checks unreliable. First, a parser that closes a group at any unfamiliar line, such as Crawl-delay or Content-Signal, will attach the following rules to the wrong group; keep a group open across every directive until the next User-agent line that follows a rule. Second, groups that share a name merge, so a file that declares User-agent: * three times has one * group containing the union of all three. Count rules across every block with the same name before deciding anything is missing.

Does Disallow: / in a named group drop my rules? #

No. This is the shape that looks like the mistake and is not. The most common way to block a crawler outright is two lines:

User-agent: GPTBot
Disallow: /

That group does not repeat Disallow: /admin/ from the wildcard group, but it does not need to: / covers every path, /admin/ included. What matters is not whether the same rule text appears in the named group, but whether the named group’s rules cover every path the wildcard closes. A rule covers every path that starts with it, * matches any run of characters, and $ anchors the end.

Named groupWildcard group hasRules lost for that crawler
Disallow: /Disallow: /admin/none: everything is blocked
Disallow: /*Disallow: /cartnone
Disallow: /wp-Disallow: /wp-admin/none: the broader rule covers it
Allow: / onlyDisallow: /admin//admin/ is open to it
repeats some rules, not allseveral Disallow rulesevery rule it leaves out

Our own check got this wrong until 6 October 2026. It compared rule text rather than coverage, so it told sites that blocking a crawler completely had broken their rules for that crawler. We noticed because of how the false finding clustered while preparing the October dataset report: 83% of sites that block an AI crawler carried it, against 13% of sites that do not, and a defect independent of blocking would not cluster like that. The check now tests coverage. If a CrawlCheck report told you a full block was shadowing your rules, re-scan and the finding clears, marked as a change in our reading rather than a change you made. The real mistake, a named group that allows everything or repeats only some wildcard rules, is still reported.

How to write named groups safely #

If you need a named group at all, give it the full rule set. The simplest correct shape lists every named crawler on consecutive User-agent lines above a single block of rules, which is legal and makes them one group:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /feed/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /feed/
Allow: /

Ask first whether you need the named group. A crawler with no group of its own already follows *, so a site that wants AI crawlers to have the same access as everyone else needs no AI section at all. Named groups are for treating a crawler differently: a stricter rule set, a Crawl-delay, or a full block. If a plugin or theme generates the file, patch the generator rather than the output, because the next save will regenerate the shadowed version.

What a shadowed group costs #

The cost is crawling and possibly indexing of URLs you excluded on purpose: internal search results, feeds, faceted parameters, login and account pages, staging paths. For search crawlers that means crawl budget spent on duplicates and thin pages, and sometimes those pages appearing in results. For AI crawlers it means the excluded pages can be fetched and used, which is usually the opposite of the intent behind the exclusion. None of it shows up as an error anywhere. Search Console will not warn you, because the file is valid.

Allow, Disallow and the longest match #

Inside the group a crawler has chosen, a second rule decides between conflicting lines: the most specific path wins, measured by the length of the matching rule, and when an Allow and a Disallow match with equal length, RFC 9309 says the Allow wins. That is why Allow: /wp-admin/admin-ajax.php can sit beside Disallow: /wp-admin/ and still open the one file. It is also why the order of lines inside a group does not matter to Google. It does matter to some older parsers, including Python's standard urllib.robotparser, which applies the first matching rule and does not understand * or $ wildcards. A file tested only with that parser can look broken when it is correct, or correct when it is broken. Test with a parser that implements RFC 9309, and put each group's specific Allow exceptions first, then its Disallow lines, then any blanket Allow: / last, so every parser reaches the same answer.

Wildcards follow the same longest-match rule. Disallow: /*?replytocom= blocks every URL carrying that parameter, and Disallow: /*.pdf$ blocks URLs ending in .pdf. When a named group copies the * rules, copy the wildcard rules too; they are the ones most often dropped, because they look like noise next to the readable paths.

What to check #

Parse every group the way RFC 9309 describes, then compare each named group’s rules with the * group’s by coverage, not by text. Note how many agents are affected and the most rules any one of them drops, and treat a search crawler among them as urgent, because the paths it now ignores can enter the index. Check the file is a file, too: a robots.txt that answers with the site’s catch-all page shows up on 0.6% of scanned sites, and crawlers treat that as no file and allow everything. Which AI crawlers honour robots.txt at all is in the robots.txt compliance guide; testing a file from the crawler’s side is in how to test robots.txt for GPTBot and ClaudeBot; and a free scan from the CrawlCheck homepage runs the group comparison on any domain.

Share of scanned sites with each finding in this articlerobots.txt rules overridden by a later group14%robots.txt answered by a catch-all page0.6%Share of scanned sites with eachfinding in this articlerobots.txt rules overridden by a later group14%robots.txt answered by a catch-all page0.6%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Does a crawler combine its own robots.txt group with User-agent: *?

No. RFC 9309 says a crawler follows only the group that best matches its name. If a group names it, the * group does not apply to it at all.

What does Allow: / in a named group do?

It gives that crawler permission to fetch everything, and because the named group replaces the * group for that crawler, every Disallow rule in the * group stops applying to it.

Does Disallow: / in a named group shadow my other rules?

No. Disallow: / blocks every path for that crawler, including every path the wildcard group disallows. Compare groups by what their rules cover, not by whether the same rule text appears in both.

How do I keep my Disallow rules when I add an AI crawler group?

Repeat every Disallow and Allow line from the * group inside the named group, including wildcard rules, or list several crawlers on consecutive User-agent lines above one shared block of rules.

Do I need a named group to allow AI crawlers?

No. A crawler with no group of its own already follows User-agent: *. Named groups are only needed to treat a crawler differently from everyone else, such as a stricter rule set or a full block.

Will Search Console warn me about a shadowed group?

No. The file is syntactically valid, so robots.txt testers report no error. Only a comparison of each named group against the * group shows the lost rules.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works