Findings · 2026-08-18 · By VSNARY | Emmanuel Orta
When an LLM summarises your audit report and gets it wrong
An AI read one of our reports and confidently explained a grade penalty we had not applied. The report left the reason out, so the mistake was ours.
Someone asked a language model to review a CrawlCheck report. It concluded the grade was lowered for things the report admitted it could not measure. That was wrong: the grade was set by one real finding, a homepage served from a cache copy nearly four hours old. But the report never said what set the grade, so the model filled the gap with a plausible guess. Reports now name the finding behind the grade and say that unmeasured sections did not affect it. Test your own reports the same way.
Someone ran one of our reports through a language model for a second opinion. The site scored 90 out of 100, sat at the 86th percentile of our corpus, and still graded C. The model concluded the letter was being lowered for categories the report said it could not measure, such as speed data with no field measurements. That would be a bad design: it would punish a site for the tool's own gaps.
What actually set the grade? #
One finding of medium severity: the homepage was being served from a cache copy nearly four hours old. The unmeasured sections had no effect. You can check the cache problem on your own site with our guide to stale cached files behind the edge.
Why was the model's reading still our fault? #
The report showed the grade and the list of unmeasured sections side by side, and never said which caused what. Placing them together implied a link. A careful reader, human or machine, followed it. For a product built on legible evidence, that is a defect in the output, not in the reader.
| What a report shows | What it means for your grade |
|---|---|
| Measured, passed | Counts in full |
| Measured, failed | Lowers the score, and raises a finding if serious enough |
| Not measured | Named in the report, and does not lower the grade |
What changed? #
Reports now say which finding set the grade, and state that unmeasured sections did not affect it. The count of scored sections shown on the site is read from the scorer itself rather than typed by hand, so the two cannot drift apart. A check whose input never arrived is now shown differently from one that ran and found nothing. The report guide walks through each part.
How do you test your own reports? #
Reports are increasingly read by machines, and a machine does not get the context you would add in a meeting. Hand one of your reports to a model and ask it to explain the headline number. If it invents a plausible cause, your report implied one. Fix it by putting the relationship in the report itself: say which input changed which number. The same applies to facts on your own website: if it is not stated, an assistant will guess.
CrawlCheck’s dataset when this page loaded: 582,334 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What should a score always carry?
The input that produced it. If a grade is lowered, the finding that lowered it should be named next to it, so the number can be checked rather than trusted.
Does an unmeasured section lower a CrawlCheck grade?
No. Sections that could not be measured are named in the report and do not lower the grade.
Why did the language model get the grade wrong?
The report placed the grade next to the list of unmeasured sections without saying which caused what, so the model inferred a link that did not exist.
How can I test whether my reports are clear to machines?
Give a report to a language model and ask it to explain the headline number. If its explanation is wrong but plausible, the report is missing the link between the number and its cause.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.