CrawlCheck

Library · Case studies

Case studies

In one sentence

Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.

One site, one defect, measured end to end.

107findings
66guides
12topics
9,163scans behind the live figures

2026-11-02 · Case studies5 views · 6 min

robots.txt returns 200 but no crawler can read it

Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it.

CHALLENGE_SERVED_2000.3%UNIFORM_REFUSAL16%

Read the measurements →

2026-11-02 · Case studies4 views · 4 min

Why word count cannot spot a parked domain: 783 words

A live domain-sale lander carried 783 words and passed every thin-content check we ran. What actually separates a parked page from a real site, and how to check one yourself from the page source.

Read the measurements →

2026-11-02 · Case studies3 views · 5 min

Google's /goto redirect: why referrer data can't show it

Google confirmed the google.com/goto redirect, but no landing page can detect it. The check that works is the ratio of Search Console clicks to GA4 organic sessions.

Read the measurements →

2026-11-06 · Case studies1 view · 3 min

18 DMARC reports, 7 domains: every failure was a spoofer

Before moving seven small-business domains from p=none to p=quarantine we read every aggregate report. Five addresses on hosting, business and residential networks were sending mail as two fencing brands. No legitimate message had failed. How the reading was done and what it showed.

Read the measurements →

2026-11-02 · Case studies1 view · 6 min

A newspaper’s schema coordinates put it in China

One missing minus sign. The address in the same schema block names a US city; the longitude below it points at northern China, and a range check calls it valid.

Read the measurements →

2026-11-02 · Case studies1 view · 3 min

www and non-www both return 200? A canonical won't fix it

Every page on crawlcheck.io existed at two addresses for months, and every canonical pointed at the right one. The canonical was not the part that mattered.

Read the measurements →

2026-11-02 · Case studies · Exclusive1 view · 4 min

Fake GPTBot and ClaudeBot requests scan for .env files

Of 874 recorded requests from clients claiming to be a named AI crawler, 69 asked for a file that holds secrets, such as cloud credentials, Terraform state and .env files, across at least 66 paths no real crawler requests.

Read the measurements →

2026-11-18 · Case studies3 min

The quote form was the slow part: 980 ms cut to 50 ms

A tree service home page froze for almost a second while it loaded. We blamed our own scripts first. Google's test showed the quote form was doing it.

Read the measurements →

2026-11-18 · Case studies3 min

"20 years of experience": the site's own records said 13

A concrete contractor's site said 20 years. Its Google Business Profile said it opened in 2013. The number was in more places than anyone remembered.

Read the measurements →

2026-11-13 · Case studies2 min

A newsletter form that lost every sign-up

The form looked fine and showed no error to us for as long as nobody tested it. Two small defects meant no address ever arrived.

Read the measurements →

2026-11-13 · Case studies2 min

A slider that took 8.8 seconds: three 328 KB PNGs

A home page slider on a site we run held the main image back for almost nine seconds in a lab test. Three ordinary causes, three fixes, and the numbers after each.

Read the measurements →

2026-11-02 · Case studies4 min

Naming a crawler cancels your * rules: Semrush's robots.txt

We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and CrawlCheck marked it down for that, which was our defect, not theirs.

Read the measurements →

2026-11-02 · Case studies6 min

agents.md: AI agent instructions you may not have written

Strip the brand and the domain out of two unrelated agents.md files and they become the same document. Neither owner wrote a word of it.

Read the measurements →

2026-11-02 · Case studies6 min

Your cache purge returned 200 and evicted nothing

Four purges returned HTTP 200 and cleared nothing; the edited page stayed invisible for two hours and twenty minutes. The two-request test that catches it, the header pair that names the stale layer, and the purge order that holds.

STALE_CACHE_SERVED13.4%

Read the measurements →

2026-11-02 · Case studies4 min

Email bounce case study: one typo, every address dead

A contractor site linked the same misspelled address on every page. Two DNS lookups proved which spelling could receive mail, and the fix took one line.

Read the measurements →

2026-11-02 · Case studies3 min

utm_source=chatgpt.com in a lead: what it proves

ChatGPT tags the links it hands out. A quote request on a Denver tree-service site we run arrived carrying that tag. How to capture the same evidence on your forms, and what one tagged lead does and does not prove.

Read the measurements →

2026-11-02 · Case studies4 min

areaServed vs service area pages: 20 towns declared, 1 page

Four home-service sites we operate declare 19–23 service areas each in structured data. Three of them have real landing pages for one or seven. A coverage claim lives in two files, and the pages are the ones a search engine can land on.

Read the measurements →

2026-11-02 · Case studies4 min

Brand schema for multi-location businesses: link both ways

One site we operate declared twelve businesses under one brand. Read on their own, the locations named no parent at all. The fix, with a copy-paste JSON-LD example.

Read the measurements →

2026-11-02 · Case studies5 min

Sitemap lastmod: 103 pages, one date, why engines skip it

Our sitemap told search engines that 103 pages last changed on 23 September, though most had been rebuilt since. Why Google and Bing ignore dates like that, how we dated each page after the fact, and the rule that keeps it right.

SITEMAP_NO_LASTMOD7.4%

Read the measurements →

Questions about this page

QAre these real sites?
Yes, every finding is a measurement taken on a live site. Domains are not named unless the author owns them.
QHow is a finding different from a guide?
A finding reports one measured case and what it showed. A guide explains how to run and read the check yourself.

Keep reading