Findings · 2026-09-12 · By VSNARY | Emmanuel Orta · 0 views

Semrush's robots.txt carries 55 rules. Fifty-four of them do not apply to Bingbot.

We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and our scanner marked it down for that — which is our defect, not theirs.

On 12 September 2026 we ran our own scanner across twenty-one companies that sell website audits — SEO platforms, crawlers, accessibility checkers and the newer AI-visibility tools. It is the population with the least excuse for a machine-layer defect, which is the only reason it is interesting.

One finding came back on six of them. We were about to publish that number. Verifying it took it to five, cost one company an apology, and turned up a defect in our own instrument.

The rule that deletes your other rules #

A crawler reads the User-agent group that matches it most specifically and ignores every other group, including *. That is the standard, not an implementation quirk. So naming an agent does not add a permission on top of your general rules. For that agent, it replaces them.

Which makes a well-meant edit dangerous. You add a group for a crawler you want to welcome. Every Disallow you wrote under * — the staging paths, the internal search results, the parameter traps — stops applying to it. The file parses. Every validator is happy. The crawler still reaches your homepage, so nothing looks wrong.

Semrush #

The wildcard group on semrush.com/robots.txt carries 55 Disallow rules. Below it sit named groups for Bingbot, BingPreview, Twitterbot, LinkedInBot, facebookexternalhit, Facebot and several others. The Bingbot group reads, in full:

Bingbot obeys the one rule written for it and ignores the other fifty-four. We are not saying that is a mistake — it may be exactly what they intend, and a motive is not something a fetch can see. What we can say is that the rules written under * are not the policy Bing operates under, and a reader of that file would reasonably think they were.

Two more, and two we will not name #

robotsense.io is the sharpest case. Its wildcard group carries four Disallow rules; below it, seven named groups — Googlebot, Bingbot, Twitterbot, facebookexternalhit, LinkedInBot and two more — each declare only Allow: / and nothing else. All four rules are void for Googlebot. The site publishes a crawler directory.

woorank.com has eight wildcard rules and an Ahrefsbot group declaring one of its own, so the eight do not apply to it. Smaller, same shape.

Our scanner reports the finding on two further tools in this set. We are not naming them, because both refused our requests for their robots.txt from two different addresses and two different user-agents, so we have our scanner's reading and no independent one. A single unreproduced read is not enough to put a company's name in a sentence like this.

The one we got wrong #

The sixth was screamingfrog.co.uk, and the finding was ours, not theirs. Their robots.txt contains a joke. Under the real rules sit five groups protecting the frog team:

There is an ASCII frog underneath and a link to their careers page. Our scanner read all five as named groups that fail to repeat the wildcard rules, reported them as a finding, and marked the grade down. Nothing was shadowed: no crawler on earth matches common salt.

Worse, the finding’s own evidence already showed that no search engine was affected, and it fired anyway. We had computed the fact that would have stopped it and were not reading it — the same shape of defect as three rows that passed because we measured nothing, with the sign reversed.

What we changed #

We corrected the check: a group naming something that is not a real crawler can no longer accuse anyone. Screaming Frog no longer gets the finding; robotsense.io still does.

The detail sentence was wrong too. It said those groups declare no Disallow rules, when several declare their own, different ones. It now says they do not repeat the wildcard rules, which is what was measured.

Method, and the parts that are not evidence #

Every score here is a single fetch of a homepage from one address on one day, not a crawl. Each robots.txt claim above was verified by hand, twice, from a different network and user-agent than the scanner used, and the two companies we could not read that way are unnamed for that reason. The * rule counts are from those hand reads, not from our own output.

We do not count a missing llms.txt against anyone, including this cohort — thirteen of the twenty-one have none. We stopped scoring it in September because nothing we have measured links publishing one to being cited, and it would be cheap to score a competitor down on a file we cannot show works.

If you are named here and think the reading is wrong, tell us what we should have seen and we will re-measure and publish the correction in this post. That offer is not decoration: this piece exists because we ran that check on ourselves first and it cost us a finding.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works