Findings · 2026-08-26 · By VSNARY | Emmanuel Orta
We measured a 225% spread between two Lighthouse runs of the same page
Nothing on the page changed. Blocking time came back 2,643 ms, then 812 ms. First paint moved 69%. If your report shows one lab number as a fact, it is overclaiming.
We ran Lighthouse twice against the same page. Nothing changed between the runs — same URL, same server, same code, minutes apart.
| Metric | Run 1 | Run 2 | Spread |
|---|---|---|---|
| Largest Contentful Paint | 5,069 ms | 6,361 ms | 25% |
| First Contentful Paint | 2,865 ms | 4,849 ms | 69% |
| Cumulative Layout Shift | 0.155 | 0.175 | 13% |
| Total Blocking Time | 2,643 ms | 812 ms | 225% |
Blocking time varied by a factor of more than three. First paint by nearly 70%. On an identical page.
Why this happens
A lab run is a simulation. Lighthouse throttles a virtual mobile device — slowed CPU, constrained network — on shared infrastructure it does not control. The throttling is deterministic; the machine underneath it is not. Total Blocking Time is the most sensitive of the four because it measures main-thread work, which is exactly what contention with other tenants affects.
None of this is a defect in Lighthouse. It is what a lab measurement is: one sample from a distribution, presented as a number.
Why it matters if you are acting on a score
We hit this while checking whether a change had helped. We had removed two unused web fonts from a site and wanted to know the effect. The post-change numbers landed inside the spread of the two pre-change runs. We could not claim the change helped, and we did not.
That is the practical trap. You make a change, re-run the test, and the number moves. Almost every time, you attribute the movement to your change. Most of the time, for anything under roughly 30%, you are reading noise.
The font change we could not verify this way was, separately, a real improvement — first paint fell from 9,709 ms to 2,865 ms, which is well outside any plausible noise band. Large effects survive this problem. Small ones do not.
What we changed in our own reports
Every lab reading CrawlCheck shows now carries this line: “One simulated-mobile Lighthouse run. Repeat runs of an unchanged page vary widely — we have measured a 225% spread on blocking time. Treat a single reading as an indication, not a verdict, and compare across dates rather than between two runs.”
We also date every cached reading and print its age, because a measurement whose time you cannot see is a measurement you cannot reason about.
What to do instead
Compare across dates, not between two runs. Treat any movement under about 30% on blocking time as unproven. Use field data — real visitors, p75 — when you have enough traffic for it to exist, and understand that most local businesses never will: Chrome only publishes field data for origins with enough visits, which is why so many small sites see “no field data available” and mistake it for a verdict on their speed.
And if a tool hands you a single lab number with no indication of variance, that is a statement about the tool, not about your site.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.