Findings · 2026-09-12 · By VSNARY | Emmanuel Orta · 0 views
Three of our own rows passed because we measured nothing
A scan that read no file at all found zero disagreements between hosts, passed the only scored row in that section, and could report 100 out of 100 for a site we never managed to read.
A check can fail in two directions. It can mark a site down for something that is fine, which people notice and complain about. It can also mark a site up for something it never measured, which nobody complains about, because the number came out in their favour.
We went looking for the second kind in our own scorer today. The pattern to search for is a row whose pass condition gates on an object existing rather than on a measurement having happened. Three rows matched. One of them was the only scored row in its section.
Host agreement, and a section that could read 100
A crawler that resolves www must not be handed a different machine layer than one that resolves the apex, so we fetch the machine files on both hosts and count the files that differ. The count of divergent files is filtered out of the list of files we actually read. The row passed when that count was zero.
A scan that read no file on either host has a count of zero divergent files. So it passed. The gate asked whether a hosts object existed, and that object exists whether or not a single fetch came back. Host agreement is the only scored row in that part of the report, which means a site we could not read one machine file from could be handed a section score of 100.
Nothing anywhere reported a problem, because from the scorer's point of view nothing had gone wrong. Zero divergences is a true statement about an empty set. It is just not an answer to the question the row's label asks.
Two schema rows with the same shape
The same construction turned up twice more in the structured-data section. One row counts locations that declare no street address, and passed when that count was zero. A site with no schema at all declares no locations, none of them is missing a street, and the row passed. The other asks whether anything in sameAs points at a map or a place, which would claim the business is that location. A site declaring no sameAs points at nothing, and passed.
Both rows can only ever add credit. Neither can subtract. So a vacuous pass on them lands on exactly the sites with the least evidence - the ones with no structured data to judge - and quietly lifts the worst results in the corpus.
| Row | What the pass gated on | What a site with nothing measured got | What it returns now |
|---|---|---|---|
| Host agreement | A hosts object existing | Pass, and the section could read 100 | Null, and the row says no file was read |
| Locations missing a street | A graph object existing | Pass | Null, and the row says no location is declared |
| sameAs pointing at a place | A sameAs object existing | Pass | Null, and the row says no sameAs is declared |
Unmeasured is null, and null is not a third kind of pass
The rule this scanner is built on is that there are three outcomes and they are never collapsed into two: true, false, and not measured. A row that was not measured is excluded from its section rather than scored zero, because marking a site down for something we failed to read is our failure and not theirs.
What today's audit found is that the rule had only ever been enforced in one direction. Nobody had checked the other end - that a row cannot be scored true for something we failed to read either. A vacuous pass is the same defect as a false accusation with the sign flipped, and it is harder to find precisely because it never generates a complaint. Our earlier pass over every scored row unscored the rows that were cosmetic or duplicated. It did not ask this question.
The headline number would have stopped at three thousand
The same audit turned up a counter with the same character of defect. The number of distinct sites measured, on the front page, was assembled by listing stored keys three pages of a thousand at a time. That is a cap, not a budget: past three thousand keys the loop simply runs out of iterations, the completion flag is never set, and the count settles on whatever the first three thousand keys happened to hold.
At the current corpus size that is invisible, because the real number is under the cap. Past it, the front page would have frozen while the corpus kept growing, and the only visible symptom would have been a headline figure that looked like a stalled business. It now paginates to completion, and it records whether a ceiling was hit, because a truncated total published as a total is the exact defect we report on other people's sites.
A paid line item that did nothing
The least defensible find of the sweep was not in the scorer at all. Our pricing page sells branded, client-ready reports on a middle tier. Orders on that tier are recorded under a plan string that the branding gate on the report itself refused, while the gate in the dashboard tested a different field and accepted them.
So a customer on that tier could set their agency name and upload a logo, see it confirmed back to them, and then open every report unbranded. Two gates asking different questions about one entitlement, and the one that mattered was the stricter of the two. They now call a single predicate, and there is a check that fails if the two ever drift apart again. Nobody reported it, which by now is the theme.
Why we publish these
A scanner that grades other people's evidence has no standing to be shy about its own. Every defect above made something look better than the evidence supported: a site score, a corpus total, a plan's feature list. None of them would have produced a complaint, and that is precisely the argument for going looking.
The corpus figures these rows feed are on the dataset page, recomputed on every load rather than typed into a page once. If you run an audit tool of your own, the search is cheap: find every row whose pass condition tests that a container exists, and ask what it returns for a site where the fetch came back empty.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Related findings
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.