Findings · 2026-09-17 · By VSNARY | Emmanuel Orta · 0 views
We put the findings behind a paywall and the report started saying there were none
Emptying an array hid the fix list everywhere at once — and made the page and the CSV assert that a site with findings had none. What an empty collection means when it can mean two things.
We put the findings behind a paywall this week. The grade stays free, the score stays free, which of the three stages fails stays free, and so does the count: four findings, one of them serious. What costs money is which four, the evidence bytes behind each one, and the files that fix them.
The implementation looked like one line. The record already carried a findings array; the unlicensed view emptied it. Every surface that renders findings reads that array, so emptying it hid the fix list everywhere at once.
It also made two of those surfaces assert that the site was clean.
What the page said #
The fix list begins with a guard for a site with nothing wrong. When the findings array is empty and there is no headroom left, it prints a tick and a sentence: Nothing to fix. Every measure that could be scored on this page passes.
That sentence had been correct for as long as an empty array meant one thing. After the paywall it meant two, and the guard could not tell them apart. A site with three serious defects, viewed by someone who had not paid, was told every measure passed.
What the export said #
The CSV has the same shape of guard, for the same reason: a file with a header and no rows is ambiguous, so the builder appends an explanatory row when there is nothing to report. The row reads NO_FINDINGS · No findings reported.
Downloaded against a locked record, that file stated in machine-readable form that a site with five findings had none. A CSV is worse than a page here, because a page is read by a person who has just seen a grade of C and might notice the contradiction. A spreadsheet row is read by whatever consumes it.
Neither of these is a paywall bug #
A paywall that hides something is working. What we shipped for about forty minutes was a paywall that hid something and then made a positive claim in its place, and the claim was false. Those are different failures, and only one of them is about money.
This is the defect this scanner exists to find on other people's sites: a surface that reports a clean result because it measured nothing, rather than because there was nothing to measure. We have written about our own rows passing on unmeasured inputs before. The rule that came out of it is the one we then broke: an absent measurement is never a passing measurement.
The fix, in both places #
An empty array now has to be accompanied by a statement of which empty it is. The record carries a summary alongside it — the total, how many are scored serious, and the count by severity — plus an explicit note that the list was withheld. The page reads the summary and prints the count with locked rows. The CSV emits FINDINGS_WITHHELD with the real total in the title column, and keeps NO_FINDINGS for the case it was always meant for.
| Surface | What it said with an empty array | What it says now |
|---|---|---|
| Report page | Nothing to fix. Every measure passes. | 4 findings on this site, 1 scored serious |
| CSV export | NO_FINDINGS · No findings reported | FINDINGS_WITHHELD · 4 finding(s), 1 serious |
| JSON record | findings: [], nothing else | findings_summary and findings_withheld |
| Agent tool reply | findings: [] after trimming | summary and note kept in the trim |
| Clean site | Nothing to fix — correct | Nothing to fix — unchanged |
The one that would have been worst #
The scanner exposes a tool interface for agents. Its reply trims the record to the fields a context window can hold, using a keep-list of field names. The keep-list contained findings. It did not contain the summary or the withheld note, because neither existed when it was written.
So an assistant asked about a site would have received an empty findings array with nothing beside it and no way to know the difference. A person reading a page can see a grade of C above the words nothing to fix and distrust one of them. A model receiving a trimmed record cannot. Both keys are in the keep-list now.
Why the tests did not catch it #
Because the tests asserted the thing we intended rather than the thing a reader receives. The check was does the locked view omit the finding titles, and it passed: the titles were gone. Nothing asserted does the locked view avoid claiming the site is clean, because until that morning no code path could have made that claim.
The check that found it was mechanical and dull: render a locked report from a fixture with five findings, then search the output for the string Nothing to fix. It fails on a false positive and passes on a clean site, which is the shape every regression check for this class should have. We had the same lesson from the proofs that said pending for twenty-six days: every check confirmed the process ran, and none of them opened the file.
What to look for on your own site #
Anywhere your system renders a count, an empty state or a zero, ask what else could produce that output. A zero that means none and a zero that means not measured are different facts wearing the same clothes, and the moment a second cause appears the original sentence becomes a lie without anyone editing it.
The free scan reports three outcomes per section for exactly this reason — passed, failed, and could not be measured — and it will never collapse the third into the first. A live report shows the shape. The rule survived four years of other people's sites and forty minutes of our own.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
Is the scan still free?
Yes. Running a scan, the grade, the score, which of the three stages fails and how many findings you have are all free, on any domain, with no account. What a licence or the fix service buys is which findings, the evidence behind each one, and the files that correct them.
How long was the wrong sentence live?
About forty minutes, between the paywall shipping and the check that caught it. It was found before the change was announced anywhere.
Why not just delete the guard that says nothing to fix?
Because it is correct for a genuinely clean site, and removing it would replace one wrong answer with no answer. The fix was to make an empty list carry a statement of which kind of empty it is.
Does the CSV still say NO_FINDINGS?
Only when a site really has no findings. A withheld list now emits FINDINGS_WITHHELD with the true total in the same row.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.