CrawlCheck

Library · Case studies

Case studies

In one sentence

Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it, and none naming a scanned third-party domain.

One site, one defect, measured end to end.

71findings
34guides
47finding codes explained
2,474scans behind the live figures

2026-09-25 · Case studies8 min

Google's goto redirect is real. The referrer still can't show it.

Our one-session test saw no wrapped links. Checked against Google's confirmation, rank-tracker telemetry, other hands-on tests and the web specs, that sighting is an outlier. The argument about how the redirect works holds up.

Read the measurements →

2026-09-23 · Case studies4 min

Internal link audit: 2,260 links checked, 4 went nowhere

Every sitemap page answered 200. One link on the page that measures us pointed at a profile that has never existed, and three in-page links on non-local reports pointed at sections that were never drawn.

Read the measurements →

2026-09-17 · Case studies4 min

Thin-content checks miss parking pages: 783 words, still parked

Scanners grade parked domains as if they were sites. The obvious fix — flag pages with almost no text — fails, because commercial landers are not thin. What works is identifying the operator serving the page.

Read the measurements →

2026-09-17 · Case studies5 min

agents.md template inheritance: is the file on your domain yours?

Strip the brand and the domain out of two unrelated agents.md files and they become the same document. Neither owner wrote a word of it.

Read the measurements →

2026-09-17 · Case studies6 min

Your cache purge returned 200 and evicted nothing

Four successful-looking cache purges cleared zero objects. Here is the two-request test that catches it, and the script that runs it.

STALE_CACHE_SERVED16.7%

Read the measurements →

2026-09-17 · Case studies10 min

robots.txt returned 200 and no crawler could read it: challenge pages

A robots.txt that answers HTTP 200 with a bot-challenge page is read as allow-everything by most crawlers. Why every validator missed it, and the two-request test that catches it.

CHALLENGE_SERVED_2000.8%UNIFORM_REFUSAL3.3%

Read the measurements →

2026-09-17 · Case studies4 min

AI visibility tools that grade you 100% and still sell you the fix

When the call to action fires regardless of the result, the result was never the point.

Read the measurements →

2026-09-17 · Case studies5 min

Email bounce case study: every contact address failed for one character

A contractor site linked the same misspelled address on every page. Two DNS lookups proved it, and the fix took one line.

Read the measurements →

2026-09-16 · Case studies5 min

The parent declared the chain. None of the locations declared it back.

One site we operate points at twelve businesses under one brand. Six of those businesses, read on their own, named no parent at all. A page-subject check now reads the relation from both sides, in the vocabulary the recovered Maps schema uses.

Read the measurements →

2026-09-12 · Case studies4 min

Semrush's robots.txt carries 55 rules. Fifty-four of them do not apply to Bingbot.

We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and our scanner marked it down for that — which is our defect, not theirs.

Read the measurements →

2026-09-10 · Case studies6 min

The Denver Post publishes coordinates that put its newsroom 10,668 km away

One character. The address in the same block says 5990 Washington St., Denver. The coordinate two lines below it says northern China, and every check we run passed it as valid.

Read the measurements →

2026-09-05 · Case studies5 min

Your schema lists service areas your sitemap has pages for a fraction of them

Four home-service sites we operate declare 19–23 service areas each in structured data. Three of them have real landing pages for one or seven. A coverage claim lives in two files, and the pages are the ones a search engine can land on.

Read the measurements →

2026-09-02 · Case studies4 min

A tree-service lead arrived with utm_source=chatgpt.com. Here is what that does and does not prove.

A quote request on a Denver tree-service site carried the tag ChatGPT appends to links it hands out. One lead is not a rate, but it is the first outcome point on a record that had only measured inputs until today.

Read the measurements →

2026-09-01 · Case studies3 min

llms.txt returned 200 and was a login wall: why status codes lie about machine files

The same page that answered a nonexistent path answered /llms.txt. One layer of our scanner noticed; the other called it present.

Read the measurements →

2026-08-29 · Case studies3 min

Our canonical tag was correct and the site was still served twice

Every page on this site existed at two addresses for months. The canonical pointed at the right one. That turns out not to be the part that matters.

Read the measurements →

2026-08-21 · Case studies4 min

Your new cache rule did not fix the object it was written for

A robots.txt served with a one-year browser cache, 7.6 days old, on a zone whose cache rule capped the edge TTL at one hour. The rule was fine. It just does not apply to anything already in the cache.

Read the measurements →

2026-08-18 · Case studies4 min

Fake GPTBot and ClaudeBot requests are scanning for .env and .ssh files

Of 874 recorded requests from clients claiming to be a named AI crawler, 69 asked for a file that holds secrets — cloud credentials, Terraform state, .env files. Across 66 distinct paths, none of which any real crawler requests.

Read the measurements →

Questions about this page

QAre these real sites?
Yes, every finding is a measurement taken on a live site. Domains are not named unless the author owns them.
QHow is a finding different from a guide?
A finding reports one measured case and what it showed. A guide explains how to run and read the check yourself.

Keep reading