Library · What crawlers are actually served
What crawlers are actually served
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
Status codes, caches, JavaScript and duplicate hosts: the bytes a crawler receives next to what a browser gets.
2026-11-02 · Research · Exclusive6 min
AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites
We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open.
Read the measurements →
2026-11-025 min
Hreflang audit: fixing return-link and x-default errors
Hreflang tells a search engine which language or regional version of a page to show. It only counts when every version links back. The rules, the six errors that make Google ignore a set, and how to audit it from the outside.
HOST_DUPLICATE_2006.2%
Read the measurements →
2026-11-02 · Case studies4 min
Why word count cannot spot a parked domain: 783 words
A live domain-sale lander carried 783 words and passed every thin-content check we ran. What actually separates a parked page from a real site, and how to check one yourself from the page source.
Read the measurements →
2026-11-02 · Case studies3 min
www and non-www both return 200? A canonical won't fix it
Every page on crawlcheck.io existed at two addresses for months, and every canonical pointed at the right one. The canonical was not the part that mattered.
Read the measurements →
2026-11-02 · Research6 min
What AI crawlers receive from local business sites
Three things we found by fetching real local business sites the way answer engines do, what moved the numbers on sites we run, and the commands to repeat it on yours.
PAGE_IS_MOSTLY_CODE27.3%
Read the measurements →
2026-11-027 min
Stale cached robots.txt or llms.txt? Two fetches tell you
Why one fetch cannot tell whether a machine file is healthy, the two-fetch check that can, the four outcomes, why a new cache rule did not fix a one-year robots.txt, and the purge order that works.
MACHINE_FILE_CACHE_STALE0.5%MACHINE_FILE_CACHE_SPLIT0%ORIGIN_BLOCKED_BEHIND_CACHE0.2%+1
Read the measurements →
2026-11-024 min
AI crawlers refused, redirected or served less: how to check
Fetch the same page as each named crawler and as an ordinary browser, at the same moment, and compare. Four outcomes are possible, and only a difference from the browser counts.
ANSWER_ENGINE_REFUSED6.1%CRAWLER_REDIRECTED_AWAY0.2%CRAWLER_SERVED_LESS0.6%+2
Read the measurements →
2026-11-026 min
Do AI crawlers render JavaScript? Check what they see
Assume not, and the question becomes simpler: is your content in the response the server sends before anything runs? One command checks it, and four test pages show why a fact in the source is not always page text.
PAGE_IS_MOSTLY_CODE27.3%CONTENT_NEEDS_JAVASCRIPT1.7%RENDER_RESOURCE_BLOCKED2.7%
Read the measurements →
2026-11-02 · Research3 min
Lighthouse scores vary between runs: two runs, 225% apart
Nothing on the page changed. Blocking time came back 2,643 ms, then 812 ms, and first paint moved 69%. What two runs prove, what they do not, and how to measure your own noise floor before trusting a score.
Read the measurements →
2026-11-02 · Research6 min
Text extractors dropped our schema and still got us right
Two unrelated services that turn pages into text for language models both dropped the whole head of our page, and every JSON-LD node with it. Both still named the company correctly. Across ten of our own sites, 87% of our structured data sits where they do not look.
Read the measurements →
2026-11-02 · Corrections4 min
Is your auditor grading your firewall? A two-request test
We once published a failing grade for a competitor that measured only its firewall. The random-path test that would have caught it, and the curl commands to run it on your own site.
Read the measurements →
2026-11-02 · Case studies6 min
Your cache purge returned 200 and evicted nothing
Four purges returned HTTP 200 and cleared nothing; the edited page stayed invisible for two hours and twenty minutes. The two-request test that catches it, the header pair that names the stale layer, and the purge order that holds.
STALE_CACHE_SERVED15.4%
Read the measurements →
2026-11-026 min
HTTP status codes for crawlers: what a 200 can hide
Challenge pages ship as 200, walls answer every path the same, and a crawler can get half the words a browser got. How to read the rest of the response, with the commands to test it.
CHALLENGE_SERVED_2000.4%CHALLENGE_PINNED_AT_EDGE0.3%UNIFORM_REFUSAL11.7%+3
Read the measurements →
2026-11-026 min
Canonical URL audit: do tags, redirects and sitemap agree?
The canonical tag is one line, and most audits stop at it. The audit that matters counts how many addresses answer 200 for the same page, whether the sitemap and links agree, and whether the hostname is a site at all. With the curl commands to check each.
HOST_DUPLICATE_2006.2%HOMEPAGE_REDIRECTS_OFF_HOST2.9%DOMAIN_IS_ALIAS0.8%+1
Read the measurements →
2026-11-025 min
Text-to-HTML ratio: does it matter, and how to check it
Not a ranking factor, but on a large page under five percent text, crawlers and AI agents download CSS, scripts and state before a sentence, and pay for it in tokens. Two commands to measure it and the block to remove first.
PAGE_IS_MOSTLY_CODE27.3%
Read the measurements →
2026-11-024 min
Markdown for AI agents: serve text/markdown by negotiation
An agent that sends Accept: text/markdown can get a page's words without the scripts and navigation. Our measurement on four pages, the headers that must come back, and the caching trap.
Read the measurements →
2026-11-025 min
Core Web Vitals without CrUX: field data for small sites
PageSpeed Insights says "no data" for most small sites because the Chrome UX Report needs more real users than they have. Lab scores are not a substitute. How to collect your own field data, the thresholds to judge it against, and the sample size below which a percentile means nothing.
Read the measurements →
2026-11-026 min
Crawl budget: measure what a crawler reads in 50 fetches
A crawler spends a limited number of fetches and leaves. How to measure what yours buys for 25, 50 or 100 fetches, which pages come last, and where fetches are wasted, with the waste signals measured on 111 sites.
Read the measurements →
2026-11-023 min
How to check if your website supports HTTP/3
A site tells browsers it speaks HTTP/3 in two places: a response header and a DNS record. Here is how to read both, what three real sites showed, and why search and AI crawlers do not use HTTP/3 yet.
Read the measurements →