Findings · 2026-09-12 · By VSNARY | Emmanuel Orta · 0 views
Semrush's robots.txt carries 55 rules. Fifty-four of them do not apply to Bingbot.
We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and our scanner marked it down for that — which is our defect, not theirs.
On 12 September 2026 we ran our own scanner across twenty-one companies that sell website audits — SEO platforms, crawlers, accessibility checkers and the newer AI-visibility tools. It is the population with the least excuse for a machine-layer defect, which is the only reason it is interesting.
One finding came back on six of them. We were about to publish that number. Verifying it took it to five, cost one company an apology, and turned up a defect in our own instrument.
The rule that deletes your other rules
A crawler reads the User-agent group that matches it most specifically and ignores every other group, including *. That is the standard, not an implementation quirk. So naming an agent does not add a permission on top of your general rules. For that agent, it replaces them.
Which makes a well-meant edit dangerous. You add a group for a crawler you want to welcome. Every Disallow you wrote under * — the staging paths, the internal search results, the parameter traps — stops applying to it. The file parses. Every validator is happy. The crawler still reaches your homepage, so nothing looks wrong.
Semrush
The wildcard group on semrush.com/robots.txt carries 55 Disallow rules. Below it sit named groups for Bingbot, BingPreview, Twitterbot, LinkedInBot, facebookexternalhit, Facebot and several others. The Bingbot group reads, in full:
| Group | Its own Disallow rules | Wildcard rules that stop applying |
|---|---|---|
| Bingbot | 1 | 54 of 55 |
| BingPreview | 1 | 54 of 55 |
| Twitterbot | 0 | 55 of 55 |
| LinkedInBot | 0 | 55 of 55 |
| facebookexternalhit | 0 | 55 of 55 |
Bingbot obeys the one rule written for it and ignores the other fifty-four. We are not saying that is a mistake — it may be exactly what they intend, and a motive is not something a fetch can see. What we can say is that the rules written under * are not the policy Bing operates under, and a reader of that file would reasonably think they were.
Two more, and two we will not name
robotsense.io is the sharpest case. Its wildcard group carries four Disallow rules; below it, seven named groups — Googlebot, Bingbot, Twitterbot, facebookexternalhit, LinkedInBot and two more — each declare only Allow: / and nothing else. All four rules are void for Googlebot. The site publishes a crawler directory.
woorank.com has eight wildcard rules and an Ahrefsbot group declaring one of its own, so the eight do not apply to it. Smaller, same shape.
Our scanner reports the finding on two further tools in this set. We are not naming them, because both refused our requests for their robots.txt from two different addresses and two different user-agents, so we have our scanner's reading and no independent one. A single unreproduced read is not enough to put a company's name in a sentence like this.
The one we got wrong
The sixth was screamingfrog.co.uk, and the finding was ours, not theirs. Their robots.txt contains a joke. Under the real rules sit five groups protecting the frog team:
| User-agent | Rule |
|---|---|
pesticides | Disallow: /frogs/ |
toads | Disallow: /frogs/ |
common salt | Disallow: /frogs/ |
the french | Disallow: /frogs/legs/ |
recruitment agencies | Disallow: /frogs/phones/ |
There is an ASCII frog underneath and a link to their careers page. Our scanner read all five as named groups that fail to repeat the wildcard rules, reported them as a medium-severity finding, and took points off the grade. Nothing was shadowed: no crawler on earth matches common salt.
Worse, the finding's own evidence recorded search_engines_affected: 0 and it fired anyway. We had already computed the fact that would have stopped it and we were not reading it — the same shape of defect as three rows that passed because we measured nothing, with the sign reversed.
What we changed
A shadowed group now counts only if one of its user-agents is a crawler we actually publish and evaluate — the same list the crawler reference is built from. Groups naming something we do not recognise are still counted and still returned in the evidence; they simply cannot accuse anyone. On the shipping code, Screaming Frog went from five shadowed groups to zero and the finding disappeared; robotsense.io went from eleven to nine and still fires.
The detail sentence was wrong too. It said those groups declare no Disallow rules, when several of them declare their own, different ones — a claim the evidence printed directly beneath it contradicted. It now says they do not repeat the wildcard rules, which is what was measured.
Method, and the parts that are not evidence
Every score here is a single fetch of a homepage from one address on one day, not a crawl. Each robots.txt claim above was verified by hand, twice, from a different network and user-agent than the scanner used, and the two companies we could not read that way are unnamed for that reason. The * rule counts are from those hand reads, not from our own output.
We do not count a missing llms.txt against anyone, including this cohort — thirteen of the twenty-one have none. We stopped scoring it in September because nothing we have measured links publishing one to being cited, and it would be cheap to score a competitor down on a file we cannot show works.
If you are named here and think the reading is wrong, tell us what we should have seen and we will re-measure and publish the correction in this post. That offer is not decoration: this piece exists because we ran that check on ourselves first and it cost us a finding.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Related findings
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.