Findings · 2026-08-26 · By VSNARY | Emmanuel Orta · 0 views

We measured a 225% spread between two Lighthouse runs of the same page

Nothing on the page changed. Blocking time came back 2,643 ms, then 812 ms. First paint moved 69%. If your report shows one lab number as a fact, it is overclaiming.

We ran Lighthouse twice against the same page. Nothing changed between the runs — same URL, same server, same code, minutes apart.

Blocking time varied by a factor of more than three. First paint by nearly 70%. On an identical page.

Why this happens #

A lab run is a simulation. Lighthouse throttles a virtual mobile device — slowed CPU, constrained network — on shared infrastructure it does not control. The throttling is deterministic; the machine underneath it is not. Total Blocking Time is the most sensitive of the four because it measures main-thread work, which is exactly what contention with other tenants affects.

None of this is a defect in Lighthouse. It is what a lab measurement is: one sample from a distribution, presented as a number.

Notice which way the four metrics moved. Blocking time fell by a factor of three while first paint nearly doubled — in the same pair of runs, on the same page. They did not move together, and neither run is the “good” one. That is the signature of noise rather than of a page behaving differently: a real change to a page moves the metrics it touches in a consistent direction, and this moved them in opposite ones.

Why it matters if you are acting on a score #

We hit this while checking whether a change had helped. We had removed two unused web fonts from a site and wanted to know the effect. The post-change numbers landed inside the spread of the two pre-change runs. We could not claim the change helped, and we did not.

That is the practical trap. You make a change, re-run the test, and the number moves. Almost every time, you attribute the movement to your change. Most of the time, for anything under roughly 30%, you are reading noise.

The font change we could not verify this way was, separately, a real improvement — first paint fell from 9,709 ms to 2,865 ms, which is well outside any plausible noise band. Large effects survive this problem. Small ones do not.

Your noise floor is yours to measure #

The general rule that follows is uncomfortable but simple: a measurement is only evidence for a change if the change is larger than the instrument’s own spread. Nobody publishes that spread for you, so you have to produce it — two runs of an unchanged page, before you touch anything, is enough to know what your own noise floor looks like.

What we changed in our own reports #

Three things, and the third is the one that matters.

First, every lab reading here carries the caveat in the report itself, not in a footnote: “One simulated-mobile Lighthouse run. Repeat runs of an unchanged page vary widely — we have measured a 225% spread on blocking time. Treat a single reading as an indication, not a verdict, and compare across dates rather than between two runs.”

Second, every cached reading is dated and its age printed in plain words — measured 40 minutes ago, or measured 6 hours ago; if the page changed since, the numbers below describe the older version. A measurement whose time you cannot see is a measurement you cannot reason about, and a cached lab run is a Tuesday audit of a Tuesday audit.

Third, and this is the structural fix: the lab numbers stopped feeding the grade. They are displayed, labelled synthetic, and scored nothing. That change cost the report a familiar-looking number and bought it a defensible one.

The reasoning: promoting a signal from a display row to a scored input raises the evidence bar it has to clear. A number good enough to show a reader is not automatically good enough to accuse a site with. A single lab run is exactly that case — worth seeing, not worth grading.

What to do instead #

Compare across dates, not between two runs. Treat any movement under about 30% on blocking time as unproven. Report a median of at least five runs, or say plainly that the number is a single sample — either is honest; a lone number presented as a fact is not.

Field data, and what its absence means #

Use field data — real visitors, p75 — when you have enough traffic for it to exist, and understand that most local businesses never will: Chrome only publishes field data for origins with enough visits, which is why so many small sites see “no field data available” and mistake it for a verdict on their speed. It is not a verdict. It is an absence, and the two are different, in the same way an unreadable file and a missing one are different.

The alternative for a site below that threshold is a first-party beacon: a few lines that record the real vitals of real visits to your own site and report them back to you. It is the only route to field data for a small site, and it is the route we took after a lab score could not tell us whether our own homepage was slow. It could not. The beacon could, immediately, and the first thing it found was a one-second delay we had built ourselves.

And if a tool hands you a single lab number with no indication of variance, that is a statement about the tool, not about your site.

Questions this post answers

How many Lighthouse runs do I need?

Enough to see the spread. Two runs minutes apart on an unchanged page moved Total Blocking Time by 225%. Report a median of at least five, or say the number is a single sample.

Why did CrawlCheck stop using lab data?

Because a finding built on one run accuses a site of something that may not be true on the next. Lab readings are still shown, labelled synthetic, but they no longer count toward the grade.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works