Agent view · Guides · 2026-10-02
Naming a crawler in robots.txt drops your Disallow rules
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-dir |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 19,571 |
| words | 2,975 |
| sha-256 | ecd769bb46faf1f1ec738715efe2bedac8ef1b67700d2c78b753a0662cff837a |
| repr-digest | sha-256=:7Ndpu0b68fHsc4cV7+K+2sjvG2dwDSx4t1OgZiz/g3o=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311493;expires=1795315093;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="0u2vZ5yknA8hkH_oTo7NhQ";tag="crawlcheck-page" |
| signature | sig1=:iamsPRtJqcG/m+cIOdRr4mnuaYCHf7XZlpj7Jraj9BdnfHvHBSZvbWEpZb55ONQUut6zB8WRscUUkHhXNFT2DQ==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-02 · By [VSNARY | Emmanuel Orta](/author) · 1 view # Naming a crawler in robots.txt drops your Disallow rules A crawler obeys exactly one group in robots.txt, the most specific one that names it. Add a group for GPTBot with Allow: / and every Disallow under User-agent: * stops applying to GPTBot. How to see it, how to write the file so it cannot happen, and the full-block shape that only looks like the mistake. Under RFC 9309 a crawler follows exactly one group in robots.txt: the group whose User-agent line matches it most specifically, or User-agent: * if none names it. Groups are not combined, so adding User-agent: GPTBot with only Allow: / removes every Disallow rule in the * group for GPTBot. The fix is to repeat the full * rule set inside every named group, or list several agents above one shared block. A named group with Disallow: / is safe: it blocks every path, so nothing is lost. Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - The rule: one crawler, one group - Why is it so common now? - How to check a file by hand - Does Disallow: / in a named group drop my rules? - How to write named groups safely - What a shadowed group costs - Allow, Disallow and the longest match - What to check Most [robots.txt](/glossary/robots-txt) files that go wrong do not look wrong. Every line parses, every path is spelled correctly, and the homepage is reachable for every crawler you care about. The failure is structural: someone added a block for a named crawler, usually an AI crawler, and that block quietly switched off the rules everyone assumed still applied. This guide covers the one rule behind it, how to check a file in under a minute, how to write named groups that keep your exclusions, and the one shape that looks like this mistake and is not. ## The rule: one crawler, one group # RFC 9309, the Robots Exclusion Protocol, says a crawler must obey the group whose `User-agent` line best matches its product [token](/glossary/token). If more than one group names the same crawler, those groups are merged into one. If no group names it, it obeys the `User-agent: *` group. What the standard does not do is combine a named group with the `*` group. The moment a group names a crawler, the `*` group no longer exists for that crawler. Google's robots.txt documentation describes the same behaviour for its crawlers. So a file that reads `User-agent: *` followed by ten `Disallow` lines, and then `User-agent: GPTBot` followed by `Allow: /`, does not mean "everyone follows the ten rules and GPTBot is also welcome". It means GPTBot may fetch everything, including the ten paths you excluded. The same is true for any named crawler, including Googlebot and Bingbot. ## Why is it so common now? # The pattern spread with AI crawler allowlists. Plugin snippets and blog advice recommend an explicit `Allow: /` block for GPTBot, ClaudeBot, PerplexityBot and a dozen others so answer engines can reach the site. Each block is harmless alone. Together they create a dozen more specific groups, each of which deletes the `*` rules for one crawler, so search, feed, login, cart and parameter URLs kept out on purpose become open to exactly the crawlers the edit meant to welcome. We made this mistake ourselves: on sites we operate, the allowlist we deployed shadowed every Disallow rule for every crawler it named, and our own check graded the files clean because each named crawler could reach the homepage. [Fifty-four rules that do not apply to Bingbot](/blog/fifty-four-rules-that-do-not-apply-to-bingbot) shows the same shape on a large public file. The share of scanned sites where at least one named group drops rules that `*` carries: **14%**. ## How to check a file by hand # Open the file at `/robots.txt` and walk it group by group. A group starts at one or more consecutive `User-agent` lines and runs until the next `User-agent` line that follows a rule. For each named group, list its `Disallow` lines and compare them with the `*` group. Every `*` Disallow that is missing from a named group is a rule that crawler does not follow. | File shape | What the named crawler obeys | Rules lost | | --- | --- | --- | | `*` group only | the `*` group | none | | `*` group + `User-agent: GPTBot` / `Allow: /` | only `Allow: /` | every `*` Disallow, for GPTBot | | `*` group + named group that repeats every Disallow | its own copy of the rules | none | | `*` declared twice, rules in the second block | both `*` blocks, merged | none, but easy to misread | | named group with its own different Disallows | only its own Disallows | any `*` rule not repeated | | `/robots.txt` returns the site's 404 page as 200 | nothing: no file exists | all of them | Two traps make hand checks unreliable. First, a parser that closes a group at any unfamiliar line, such as `Crawl-delay` or `Content-Signal`, will attach the following rules to the wrong group; keep a group open across every directive until the next `User-agent` line that follows a rule. Second, groups that share a name merge, so a file that declares `User-agent: *` three times has one `*` group containing the union of all three. Count rules across every block with the same name before deciding anything is missing. ## Does `Disallow: /` in a named group drop my rules? # No. This is the shape that looks like the mistake and is not. The most common way to block a crawler outright is two lines: ``` User-agent: GPTBot Disallow: / ``` That group does not repeat `Disallow: /admin/` from the wildcard group, but it does not need to: `/` covers every path, `/admin/` included. What matters is not whether the same rule text appears in the named group, but whether the named group’s rules cover every path the wildcard closes. A rule covers every path that starts with it, `*` matches any run of characters, and `$` anchors the end. | Named group | Wildcard group has | Rules lost for that crawler | | --- | --- | --- | | `Disallow: /` | `Disallow: /admin/` | none: everything is blocked | | `Disallow: /*` | `Disallow: /cart` | none | | `Disallow: /wp-` | `Disallow: /wp-admin/` | none: the broader rule covers it | | `Allow: /` only | `Disallow: /admin/` | `/admin/` is open to it | | repeats some rules, not all | several Disallow rules | every rule it leaves out | Our own check got this wrong until 6 October 2026. It compared rule text rather than coverage, so it told sites that blocking a crawler completely had broken their rules for that crawler. We noticed because of how the false finding clustered while preparing [the October dataset report](/blog/state-of-ai-visibility-october-2026-139788-homepages): 83% of sites that block an AI crawler carried it, against 13% of sites that do not, and a defect independent of blocking would not cluster like that. The check now tests coverage. If a CrawlCheck report told you a full block was shadowing your rules, re-scan and the finding clears, marked as a change in our reading rather than a change you made. The real mistake, a named group that allows everything or repeats only some wildcard rules, is still reported. ## How to write named groups safely # If you need a named group at all, give it the full rule set. The simplest correct shape lists every named crawler on consecutive `User-agent` lines above a single block of rules, which is legal and makes them one group: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /feed/ User-agent: GPTBot User-agent: ClaudeBot User-agent: PerplexityBot Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /feed/ Allow: / ``` Ask first whether you need the named group. A crawler with no group of its own already follows `*`, so a site that wants AI crawlers to have the same access as everyone else needs no AI section at all. Named groups are for treating a crawler differently: a stricter rule set, a `Crawl-delay`, or a full block. If a plugin or theme generates the file, patch the generator rather than the output, because the next save will regenerate the shadowed version. ## What a shadowed group costs # The cost is crawling and possibly indexing of URLs you excluded on purpose: internal search results, feeds, faceted parameters, login and account pages, staging paths. For search crawlers that means [crawl budget](/glossary/crawl-budget) spent on duplicates and thin pages, and sometimes those pages appearing in results. For AI crawlers it means the excluded pages can be fetched and used, which is usually the opposite of the intent behind the exclusion. None of it shows up as an error anywhere. Search Console will not warn you, because the file is valid. ## Allow, Disallow and the longest match # Inside the group a crawler has chosen, a second rule decides between conflicting lines: the most specific path wins, measured by the length of the matching rule, and when an `Allow` and a `Disallow` match with equal length, RFC 9309 says the `Allow` wins. That is why `Allow: /wp-admin/admin-ajax.php` can sit beside `Disallow: /wp-admin/` and still open the one file. It is also why the order of lines inside a group does not matter to Google. It does matter to some older parsers, including Python's standard `urllib.robotparser`, which applies the first matching rule and does not understand `*` or `$` wildcards. A file tested only with that parser can look broken when it is correct, or correct when it is broken. Test with a parser that implements RFC 9309, and put each group's specific `Allow` exceptions first, then its `Disallow` lines, then any blanket `Allow: /` last, so every parser reaches the same answer. Wildcards follow the same longest-match rule. `Disallow: /*?replytocom=` blocks every [URL](/glossary/url) carrying that parameter, and `Disallow: /*.pdf$` blocks URLs ending in `.pdf`. When a named group copies the `*` rules, copy the wildcard rules too; they are the ones most often dropped, because they look like noise next to the readable paths. ## What to check # Parse every group the way RFC 9309 describes, then compare each named group’s rules with the `*` group’s by coverage, not by text. Note how many agents are affected and the most rules any one of them drops, and treat a search crawler among them as urgent, because the paths it now ignores can enter the index. Check the file is a file, too: a robots.txt that answers with the site’s catch-all page shows up on 0.6% of scanned sites, and crawlers treat that as no file and allow everything. Which AI crawlers honour robots.txt at all is in [the robots.txt compliance guide](/guides/do-ai-crawlers-respect-robots-txt); testing a file from the crawler’s side is in [how to test robots.txt for GPTBot and ClaudeBot](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot); and a free scan from the [CrawlCheck homepage](/) runs the group comparison on any domain. Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Naming%20a%20crawler%20in%20robots.txt%20drops%20your%20Disallow%20rules)[Email](mailto:?subject=Naming%20a%20crawler%20in%20robots.txt%20drops%20your%20Disallow%20rules&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Does a crawler combine its own robots.txt group with User-agent: *? No. RFC 9309 says a crawler follows only the group that best matches its name. If a group names it, the * group does not apply to it at all. ### What does Allow: / in a named group do? It gives that crawler permission to fetch everything, and because the named group replaces the * group for that crawler, every Disallow rule in the * group stops applying to it. ### Does Disallow: / in a named group shadow my other rules? No. Disallow: / blocks every path for that crawler, including every path the wildcard group disallows. Compare groups by what their rules cover, not by whether the same rule text appears in both. ### How do I keep my Disallow rules when I add an AI crawler group? Repeat every Disallow and Allow line from the * group inside the named group, including wildcard rules, or list several crawlers on consecutive User-agent lines above one shared block of rules. ### Do I need a named group to allow AI crawlers? No. A crawler with no group of its own already follows User-agent: *. Named groups are only needed to treat a crawler differently from everyone else, such as a stricter rule set or a full block. ### Will Search Console warn me about a shadowed group? No. The file is syntactically valid, so robots.txt testers report no error. Only a comparison of each named group against the * group shows the lost rules. ## Related findings [2026-11-13 Why a robots.txt allow list cancels your Disallow rules A robots.txt group that names a crawler replaces your User-agent: * rules for it, so an allow list can open paths you meant to clo](/guides/robots-txt-allow-list-cancels-your-disallow-rules)[2026-09-12 Naming a crawler cancels your * rules: Semrush's robots.txt A crawler obeys only the robots.txt group that names it, ignoring the * rules. Hand-verified on three of 21 audit tools, plus our](/blog/fifty-four-rules-that-do-not-apply-to-bingbot)[2026-10-03 How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters m](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) | **Glossary:** [agent task canary](/glossary/agent-task-canary) · [SearchAction](/glossary/searchaction) | **Registry:** [bbb.org](/registry/bbb.org) · [dimmittcadillac.com](/registry/dimmittcadillac.com) · [infinitechoice.com](/registry/infinitechoice.com) · [nolawriting.com](/registry/nolawriting.com) · [vercel.com](/registry/vercel.com) · [Trade: Landscaping](/registry/trade/landscaping) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.