Findings · 2026-09-12 · By VSNARY | Emmanuel Orta
Naming a crawler cancels your * rules: Semrush's robots.txt
We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and CrawlCheck marked it down for that, which was our defect, not theirs.
A crawler obeys only the robots.txt group that names it most specifically and ignores every other group, including User-agent: *. Adding a group for one crawler therefore replaces your general rules for it instead of adding to them. On semrush.com, the wildcard group has 55 Disallow rules but the Bingbot group has one of its own, so 54 of the 55 do not apply to Bingbot. We confirmed the pattern by hand on three of 21 audit tools. The fix is to repeat the wildcard rules in every named group.
On 12 September 2026 we ran CrawlCheck across twenty-one companies that sell website audits: SEO platforms, crawlers, accessibility checkers and the newer AI-visibility tools. It is the population with the least excuse for a robots.txt defect, which is the only reason it is interesting.
One finding came back on six of them. We were about to publish that number. Verifying it took it to five, cost one company an apology, and turned up a defect in our own instrument.
Why does naming a crawler delete your other rules? #
A crawler reads the User-agent group that matches it most specifically and ignores every other group, including *. That is the standard, not an implementation quirk. So naming an agent does not add a permission on top of your general rules. For that agent, it replaces them.
That makes a well-meant edit dangerous. You add a group for a crawler you want to welcome. Every Disallow you wrote under *, such as staging paths, internal search results and parameter traps, stops applying to it. The file parses, every validator is happy, and the crawler still reaches your homepage, so nothing looks wrong.
What does Semrush's file show? #
The wildcard group on semrush.com/robots.txt carries 55 Disallow rules. Below it sit named groups for Bingbot, BingPreview, Twitterbot, LinkedInBot, facebookexternalhit, Facebot and several others.
| Group | Its own Disallow rules | Wildcard rules that stop applying |
|---|---|---|
| Bingbot | 1 | 54 of 55 |
| BingPreview | 1 | 54 of 55 |
| Twitterbot | 0 | 55 of 55 |
| LinkedInBot | 0 | 55 of 55 |
| facebookexternalhit | 0 | 55 of 55 |
Bingbot obeys the one rule written for it and ignores the other fifty-four. That may be exactly what they intend; a motive is not something a fetch can see. What we can say is that the rules under * are not the policy Bing operates under, and a reader of that file would reasonably think they were.
Who else had it? #
robotsense.io is the sharpest case. Its wildcard group carries four Disallow rules; below it, seven named groups, including Googlebot, Bingbot, Twitterbot, facebookexternalhit and LinkedInBot, each declare only Allow: /. All four rules are void for Googlebot. The site publishes a crawler directory.
woorank.com has eight wildcard rules and an Ahrefsbot group declaring one of its own, so the eight do not apply to it. Smaller, same shape.
| Tool | Wildcard Disallow rules | Named groups that replace them |
|---|---|---|
| semrush.com | 55 | Bingbot, BingPreview, Twitterbot, LinkedInBot, facebookexternalhit and others |
| robotsense.io | 4 | seven, including Googlebot and Bingbot, each with only Allow: / |
| woorank.com | 8 | Ahrefsbot |
CrawlCheck reports the finding on two further tools in this set. We are not naming them, because both refused our requests for their robots.txt from two different addresses and two different user-agents, so we have CrawlCheck's reading and no independent one. A single unreproduced read is not enough to put a company's name in a sentence like this.
Which one did we get wrong? #
The sixth was screamingfrog.co.uk, and the mistake was ours, not theirs. Their robots.txt contains a joke. Under the real rules sit five groups protecting the frog team:
| User-agent | Rule |
|---|---|
pesticides | Disallow: /frogs/ |
toads | Disallow: /frogs/ |
common salt | Disallow: /frogs/ |
the french | Disallow: /frogs/legs/ |
recruitment agencies | Disallow: /frogs/phones/ |
There is an ASCII frog underneath and a link to their careers page. CrawlCheck read all five as named groups that fail to repeat the wildcard rules, reported a finding, and marked the grade down. Nothing was shadowed: no crawler on earth matches common salt. The report even showed that no search engine was affected, and raised the finding anyway, the same kind of mistake as three rows that passed because we measured nothing, in reverse.
What did we change? #
A group naming something that is not a real crawler no longer raises the finding. Screaming Frog no longer gets it; robotsense.io still does.
The finding's wording was wrong too. It said those groups declare no Disallow rules, when several declare their own, different ones. It now says they do not repeat the wildcard rules, which is what was measured.
How do I check my own file? #
List every User-agent group in your robots.txt. For each named group, compare its rules with the * group: any wildcard rule not repeated inside the named group does not apply to that crawler. If you want a crawler welcomed and still kept out of staging or search pages, copy those Disallow lines into its group. Groups named after things that are not real crawlers are harmless. The full walkthrough is in robots.txt user-agent groups, and a free scan shows each AI crawler's effective rules.
What is this evidence, and what is it not? #
Every reading here is a single fetch of a homepage and robots.txt from one address on one day, not a crawl. Each robots.txt claim above was verified by hand, twice, from a different network and user-agent than CrawlCheck used. The * rule counts come from those hand reads, not from our own output.
We do not treat a missing llms.txt as a defect for anyone, this cohort included; thirteen of the twenty-one have none. We stopped reporting it as a problem in September because nothing we have measured links publishing one to being cited.
If you are named here and you think the reading is wrong, tell us what we should have seen and we will re-measure and publish the correction in this post. This piece exists because we ran that check on ourselves first, and it cost us a finding.
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Questions this post answers
Does a named User-agent group in robots.txt add to the * rules?
No. A crawler follows only the group that names it most specifically and ignores every other group, including the wildcard. A named group replaces the general rules for that crawler.
How do I keep my Disallow rules when I add a group for one crawler?
Repeat every wildcard Disallow line you still want inside the named group. Rules are not inherited between groups.
Do validators catch robots.txt group shadowing?
Usually not. A shadowed file parses correctly and the crawler still reaches the homepage, so nothing looks broken. You find it by comparing each named group's rules with the wildcard group's.
Is a robots.txt group with a joke user-agent a problem?
No. A group naming something no real crawler sends, such as a joke name, matches nothing and changes nobody's rules.
Does a missing llms.txt count against a site in CrawlCheck?
No. CrawlCheck does not treat a missing llms.txt as a defect, because nothing measured so far links publishing one to being cited.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.