CrawlCheck

Library · Corrections

Corrections

In one sentence

Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.

Where CrawlCheck's own readings, reports or pipeline were wrong, and the fix.

107findings
66guides
12topics
9,163scans behind the live figures

2026-11-02 · Corrections13 views · 2 min

Our receipt checker would have rejected our own receipts

A flaw in our published offline verifiers would have called genuine CrawlCheck receipts forged after our first routine key change. Our tests caught it before any receipt was affected.

Read the measurements →

2026-11-02 · Corrections4 views · 2 min

For a week, CrawlCheck told AI agents not to pay us

Our own site made the kind of change a hijacker makes first. Until we confirmed it, agents asking to pay crawlcheck.io were stopped at the payment step.

Read the measurements →

2026-11-06 · Corrections3 views · 3 min

Most of our 'referring pages' were forged Referer headers

The owner panel that lists who links to a site was reporting its own IP addresses, parked lookalike domains and a radiology college. None of them had linked to anything. How a referrer is forged, and the rule that now counts one.

Read the measurements →

2026-11-02 · Corrections2 views · 2 min

Three of our own checks passed because we measured nothing

Three CrawlCheck checks passed on sites where there was nothing to measure, a homepage counter would have stopped counting, and a paid branding option did nothing. All fixed.

Read the measurements →

2026-11-13 · Corrections1 view · 2 min

Claude-User uses two published address lists. We checked one

Anthropic publishes a crawler address list and a separate list of outbound addresses. Claude-User can arrive from either. We compared against the first only, and filed real requests as failed.

Read the measurements →

2026-11-06 · Corrections1 view · 3 min

Our crawler counts missed about 45% of hits

Every published crawler hit count before 6 November 2026 is too low. Concurrent writes to one counter dropped nearly half of all hits. Here is the measurement, the cause and the fix.

Read the measurements →

2026-11-06 · Corrections1 view · 3 min

Our AI accuracy check ran for six days and measured nothing

The daily job that asks Gemini what it tells customers about each business got a 403 on every call from 31 October to 6 November 2026. The job reported success. What broke, why nothing alerted, and what a wrong answer looked like when it came back.

Read the measurements →

2026-11-02 · Corrections1 view · 4 min

Lazy-load placeholders: why image checks miss your photos

A lazy-load plugin puts a placeholder in src and the real URL one attribute over. Tools that read src see no photographs. Ours did: 14 placeholders, zero WebP files, on a page of 13 WebP photos.

Read the measurements →

2026-11-02 · Corrections1 view · 2 min

OpenTimestamps proof says pending? Upgrade the receipt

Every timestamp file this site served was the calendar's first receipt, not the finished Bitcoin proof. The anchoring was sound the whole time; the files did not show it.

Read the measurements →

2026-11-02 · Corrections1 view · 2 min

Our audit labelled one page and measured another for weeks

Every scan of a specific URL reported the homepage's figures. The report header showed the right page; the numbers belonged to another.

Read the measurements →

2026-11-02 · Corrections1 view · 2 min

Dead link or blind spot? When a checker's failure is its own

A site's schema pointed at a live profile page. Our check could not reach it and reported it dead. The failure was ours, and the report now says so.

Read the measurements →

2026-11-02 · Corrections1 view · 2 min

Unblocking AI crawlers: what an access repair proves

Twelve synthetic robots.txt cases show that the agent’s name decides which rules apply. They do not show access, citations or referrals, and this note says so.

Read the measurements →

2026-11-19 · Corrections6 min

Who reads llms.txt? Our check called 8 of 10 unverified

Our llms.txt reader check marked 8 of 10 readers 'unverified' and counted two of our own fetches as a confirmed AI assistant. Here is what changed and what the readers were.

Read the measurements →

2026-11-13 · Corrections3 min

14 of 43 crawlers in our catalogue were classed wrong

We counted a block on 43 crawlers against a site. On 13 November 2026 we read each one against its operator's own page. Fourteen were in the wrong class, and nine more were judged by the wrong rule.

Read the measurements →

2026-11-13 · Corrections2 min

Our mirror relayed real crawlers and inflated forgery counts

A copy of this site that we ran ourselves passed crawler requests back to us under the crawler's own name. Our verifier counted them as failed checks. The counts were ours to get right.

Read the measurements →

2026-11-13 · Corrections2 min

Both our SDKs cut a tool list at 300 with no warning

A fingerprint of an MCP server's tools is only useful if it covers all of them. Ours stopped at 300 and did not say so.

Read the measurements →

2026-11-13 · Corrections2 min

Booking links counted as missing sitemap pages

Our index alerts said five small sites had hundreds of new pages Google had never fetched. The sites had 15 to 55 pages each. The rest were booking links.

Read the measurements →

2026-11-02 · Corrections2 min

Unquoted HTML attributes: why we said a viewport was missing

A report scored one of our sites 0 out of 100 on mobile while its viewport tag sat in the page. The attributes were unquoted, which is valid HTML. What changed, and how to check yours.

Read the measurements →

2026-11-02 · Corrections2 min

Why a web server can gain and lose 43 million sites

Netcraft counts tens of millions of OpenResty sites; W3Techs does not list it. Both measure honestly, and the gap is a lesson in reading denominators.

Read the measurements →

2026-11-02 · Corrections2 min

A save that returned success rewrote every newline

The write reported success. Read back, the file was 16,330 bytes larger, exactly the number of newlines it contained. A store accepting your bytes is not the same as keeping them.

Read the measurements →

2026-11-02 · Corrections2 min

Analytics added 650 ms to every request: how we found it

Our own field data said 70% of the homepage's LCP was time to first byte. The server was recording each visit before it started answering. Moving that work after the response cut TTFB to 85 ms.

Read the measurements →

2026-11-02 · Corrections6 min

Is your host blocking ClaudeBot? Group the 403s by IP first

A 57% block rate in our own access logs looked like a hosting provider shutting out AI crawlers. Grouped by source address, almost all of it was one machine wearing seven different crawler names.

Read the measurements →

2026-11-02 · Corrections4 min

Is your auditor grading your firewall? A two-request test

We once published a failing grade for a competitor that measured only its firewall. The random-path test that would have caught it, and the curl commands to run it on your own site.

Read the measurements →

2026-11-02 · Corrections2 min

When an LLM summarises your audit report and gets it wrong

An AI read one of our reports and confidently explained a grade penalty we had not applied. The report left the reason out, so the mistake was ours.

Read the measurements →

2026-11-02 · Corrections2 min

Why caching only successes kept our scans slow

Scans of small business sites ran slow and varied from run to run, because we kept asking a question whose answer was always no. Our first fix made one case worse.

Read the measurements →

2026-11-02 · Corrections2 min

A placeholder check flagged NaN. It was inside a URL

Our check for leaked placeholder text flagged a national newspaper's record three times. All three hits were letters inside one URL slug.

Read the measurements →

2026-11-02 · Corrections3 min

Seven site types, and the report checks we corrected

Some report checks failed well-built sites for things that are not defects, and a few counted one fault twice. We read every graded check against seven very different sites and fixed what was wrong.

Read the measurements →

2026-11-02 · Corrections2 min

Correction: our paywall said sites with findings were clean

For about forty minutes, a locked report told unpaid readers there was nothing to fix, and its spreadsheet export said there were no findings. Hiding findings was intended; claiming there were none was the bug.

Read the measurements →

2026-11-02 · Corrections2 min

A 200 status hid an empty page for weeks

One of our own internal pages returned HTTP 200 while showing a single apology sentence instead of its content. Every check we ran passed, because none of them read the page.

Read the measurements →

2026-11-02 · Corrections2 min

Sites measured is not people: what our own logs showed

The sites-measured figure on our homepage is true, and it is not demand. One week of our own request logs shows who visits a site about crawlers: mostly crawlers.

Read the measurements →

2026-11-02 · Corrections2 min

Correction: the owner's own scans came back locked

For a time the site owner's scans came back gated like a stranger's while the report page showed everything. No score or customer was affected. The lesson is about testing gated content.

Read the measurements →

2026-11-02 · Corrections2 min

Five days of our alerts went to an address with no mailbox

After a mail migration, the address our alerts were sent to had no mailbox behind it. Every message was marked sent and none arrived, for five days.

Read the measurements →

2026-11-02 · Corrections2 min

Our security headers skipped every signed-in session

A new Content-Security-Policy and tighter CORS headers were checked live and called done. Signed-in sessions received neither. The check had only ever looked signed out.

Read the measurements →

2026-11-02 · Corrections2 min

Our extension read a page as 36.5% JavaScript. It was zero.

The CrawlCheck browser extension overstated how much of a page needed JavaScript, for two reasons that both pushed the number up. Version 1.0.2 fixed both.

Read the measurements →

2026-11-02 · Corrections2 min

Eight contradictions in our own copy, rechecked

An outside audit listed eight places where crawlcheck.io contradicted itself. Rechecked against the live site: which were real, which were two truths read as one, and the one still wrong.

Read the measurements →

2026-11-02 · Corrections3 min

Our directory froze for 20 hours and logged no error

For about 20 hours on 13 and 14 October 2026 the index behind our directory of SEO and AI tools did not rebuild, and eight cards opened reports that no longer existed. Nothing reported an error.

Read the measurements →

2026-11-02 · Corrections2 min

An access rule on one path locked our customers out

On 14 October 2026 an access rule meant for staff-only pages also covered customer sign-in, which shared the same address. For an hour, the Sign in link on 247 pages led to a login customers could not pass.

Read the measurements →

2026-11-02 · Corrections2 min

Three sections said “not scored”. Our grade counted them.

For part of 22 October 2026, three new report sections were labelled “measured, not scored” but still counted in the grade. Rescan if your report is from that window.

Read the measurements →

2026-11-02 · Corrections4 min

One 503 is not a blocked site: refusal or hiccup?

A test scan read one 503 from a site's machine files and called the origin blocked. The same files answered 200 to six requests straight after. What a single refusal means, what a repeated one means, and how to tell them apart from your own desk.

Read the measurements →

2026-11-02 · Corrections3 min

AI answer corrections: three clocks, not one update time

An earlier edition of this note read two archived observations as if they timed a correction in AI answers. They do not. This is what they show, and the log that would.

Read the measurements →

Questions about this page

QAre these real sites?
Yes, every finding is a measurement taken on a live site. Domains are not named unless the author owns them.
QHow is a finding different from a guide?
A finding reports one measured case and what it showed. A guide explains how to run and read the check yourself.

Keep reading