Library

# What agents are served, *measured on real sites.*

Every finding is one measurement of what AI agents and crawlers were actually served; every guide is how to run the same check on your own site. Organised by topic, most read first. **A business found with a defect is named only after its owner has been contacted**; public vendors and AI operators measured on their own sites are named.

New findings are posted on [LinkedIn](https://www.linkedin.com/in/emmanuelorta) the day they publish, and the [feed](/feed.xml) carries every one.

In one sentence

Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.

Read it as an agent

[JSON FeedGET /feed.json](/feed.json)[AtomGET /feed.xml](/feed.xml)[Any page as MarkdownAccept: text/markdown](/blog/our-llms-txt-reader-check-called-8-of-10-readers-unverified)[The site map for modelsGET /llms.txt](/llms.txt)

[Guides 66](/guides)[All findings 107](/blog)[Exclusive 20](/blog?kind=exclusive)[Case studies 19](/blog?kind=cases)[Research 40](/blog?kind=research)[Releases 8](/blog?kind=releases)[Corrections 40](/blog?kind=corrections)

Crawler access and robots.txt 17Crawler identity and verification 18What crawlers are actually served 20AI agents and MCP 12llms.txt, sitemaps and machine files 10Schema and entities 8Local business data 11Measuring AI visibility 18Site health and security 13Web-wide census 7More findings 1Corrections and releases 38

**107**findings

**66**guides

**12**topics

**9,163**scans behind the live figures

20 exclusive findings

## Only measured here

Measured only on crawlcheck.io: findings never published, or never observed at this size, anywhere else.

[2026-11-02 · Research · Exclusive 5 views · 6 min ### Google's Googlebot check passes 198 of 200 rented servers Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot. Read the measurements →](/blog/googles-googlebot-verification-rule-passes-any-google-cloud-server)[2026-11-02 · Research · Exclusive 2 views · 7 min ### State of AI visibility, October 2026: 139,788 homepages The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see. ROBOTS_RULES_SHADOWED14% STALE_CACHE_SERVED13.4% NO_LLMS_TXT34.2% +2 Read the measurements →](/blog/state-of-ai-visibility-october-2026-139788-homepages)[2026-11-02 · Research · Exclusive 1 view · 5 min ### Google-Extended has no user agent. 1,582 requests used it Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway. Read the measurements →](/blog/a-name-google-says-it-never-sends)

[Only 6% of websites block an AI crawler: the full breakdown Exclusive 2026-11-02 · Research · 1 view · 3 min](/blog/only-six-percent-of-websites-block-an-ai-crawler)[Which AI crawlers the top 10,000 sites block most, October 2026 Exclusive 2026-11-02 · Research · 1 view · 2 min](/blog/which-ai-crawlers-top-sites-block-most-october-2026)[Robots.txt allows it, the server refuses: 57 top sites Exclusive 2026-11-02 · Research · 3 min](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)[402 Payment Required: what top sites send AI crawlers Exclusive 2026-11-02 · Research · 5 min](/blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price)[MCP Registry census: what 25,983 endpoints answered Exclusive 2026-11-02 · Research · 2 views · 5 min](/blog/mcp-registry-census-1851-servers-what-they-answer)[MCP reliability, October 2026: 39,484 handshakes measured Exclusive 2026-11-02 · Research · 1 view · 2 min](/blog/mcp-reliability-october-2026)[MCP servers that ask for a login: what 1,366 request first Exclusive 2026-11-02 · Research · 2 views · 4 min](/blog/mcp-servers-that-ask-for-a-login-what-they-request-by-default)[Agent conduct, October 2026: 34,676 verified requests Exclusive 2026-11-02 · Research · 2 min](/blog/agent-conduct-october-2026)[AI agents cannot refuse what they cannot see: 27 traps Exclusive 2026-11-02 · Research · 3 views · 3 min](/blog/ai-agents-cannot-refuse-what-they-cannot-see)[Do AI agents stop before acting on a lookalike? We tested 4 Exclusive 2026-11-02 · Research · 4 min](/blog/do-ai-agents-stop-before-acting-on-a-lookalike-site)[AI agents acting for people: 613 visits, one read robots.txt Exclusive 2026-11-02 · Research · 3 min](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt)[Eight weeks of AI crawlers on one site: who was really who Exclusive 2026-11-02 · Research · 10 views · 5 min](/blog/eight-weeks-of-agent-conduct-who-was-really-who)[Fake GPTBot and ClaudeBot requests scan for .env files Exclusive 2026-11-02 · Case studies · 1 view · 4 min](/blog/the-ai-crawler-that-wanted-your-env-file)[How much AI crawler traffic is forged? We check the IPs Exclusive 2026-11-02 · Research · 6 min](/blog/how-much-ai-crawler-traffic-is-forged)[AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites Exclusive 2026-11-02 · Research · 6 views · 6 min](/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites)[Google's Maps provider list: 793 names, no Yelp Exclusive 2026-11-02 · Research · 3 views · 6 min](/blog/googles-maps-provider-list-has-no-yelp-in-it)[AI visibility vendors audited: 11 of 23 have llms.txt Exclusive 2026-11-02 · Research · 5 views · 3 min](/blog/we-scanned-the-tools-that-audit-your-site)

[Measured something nobody has published? Submit your finding →](/submit-finding)

Find a finding or guide

14 guides · 3 findings

## Crawler access and robots.txt

Whether a crawler is let in, and whether the server agrees with what robots.txt says.

[2026-11-13 8 views · 3 min ### robots.txt and AI fetchers: what seven operators say A robots.txt rule stops a crawler that chooses to obey it. Five AI operators say their on-demand fetchers may not. Here is what each one publishes, and what to do about it. Read the measurements →](/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers)[2026-11-02 5 views · 7 min ### How to block AI crawlers like GPTBot and ClaudeBot Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that usually get made as one, and the one people regret is rarely the one they intended. AI_OPTOUT_SET1.8% ROBOTS_RULES_SHADOWED14% ANSWER_ENGINE_REFUSED5.8% +2 Read the measurements →](/guides/how-to-block-ai-crawlers)[2026-11-02 · Case studies 5 views · 6 min ### robots.txt returns 200 but no crawler can read it Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it. CHALLENGE_SERVED_2000.3% UNIFORM_REFUSAL16% Read the measurements →](/blog/a-file-that-answered-200-and-could-not-be-read)

[Is Cloudflare blocking AI crawlers on your site? 2026-11-02 · 4 views · 5 min](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site)[How to check if AI crawlers can read your site 2026-11-02 · 4 views · 5 min](/guides/how-to-check-if-ai-crawlers-can-read-your-site)[Cloudflare AI Crawl Control vs CrawlCheck: block vs measure 2026-11-02 · 4 views · 7 min](/guides/cloudflare-ai-crawl-control-vs-crawlcheck)[How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared 2026-11-02 · 4 views · 9 min](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot)[Content-Signal in robots.txt: search, ai-input, ai-train 2026-11-02 · 3 views · 4 min](/guides/content-signals-in-robots-txt-search-ai-input-ai-train)[Do AI crawlers and agents respect robots.txt? 2026-11-02 · 2 views · 5 min](/guides/do-ai-crawlers-respect-robots-txt)[Why a robots.txt allow list cancels your Disallow rules 2026-11-13 · 1 view · 2 min](/guides/robots-txt-allow-list-cancels-your-disallow-rules)[Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexing 2026-11-02 · 1 view · 7 min](/guides/meta-robots-and-x-robots-tag-the-three-layers-of-crawler-permission)[robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers 2026-11-02 · 1 view · 6 min](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler)[Naming a crawler in robots.txt drops your Disallow rules 2026-11-02 · 1 view · 7 min](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules)[Google-Extended and Applebot-Extended: block AI training 2026-11-02 · 1 view · 2 min](/guides/google-extended-and-applebot-extended-opt-out-of-ai-training-keep-search)[Block AI crawlers in nginx, Apache and Caddy (tested) 2026-11-13 · 2 min](/guides/block-ai-crawlers-nginx-apache-caddy)[Naming a crawler cancels your * rules: Semrush's robots.txt 2026-11-02 · Case studies · 4 min](/blog/fifty-four-rules-that-do-not-apply-to-bingbot)[402 Payment Required: what top sites send AI crawlers Exclusive 2026-11-02 · Research · 5 min](/blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price)

5 guides · 13 findings

## Crawler identity and verification

Which requests really came from the crawler they name, and how to check your own logs.

[2026-11-02 · Research · Exclusive 10 views · 5 min ### Eight weeks of AI crawlers on one site: who was really who Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt. Read the measurements →](/blog/eight-weeks-of-agent-conduct-who-was-really-who)[2026-11-02 6 views · 7 min ### How to verify Googlebot, GPTBot and other crawlers by IP A user-agent string is a claim anyone can type. Verification is reverse DNS with a forward confirm, or a match against the operator's published IP list, and the answer has three outcomes, not two. Read the measurements →](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers)[2026-11-02 · Research · Exclusive 5 views · 6 min ### Google's Googlebot check passes 198 of 200 rented servers Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot. Read the measurements →](/blog/googles-googlebot-verification-rule-passes-any-google-cloud-server)

[Web Bot Auth: how signed bot requests work, and who signs 2026-11-02 · 3 views · 4 min](/guides/web-bot-auth-how-signed-crawler-requests-work)[AI crawler list: what each bot does and how to verify it 2026-11-02 · 2 views · 10 min](/guides/ai-crawler-list-what-each-one-does)[Verify GPTBot, ClaudeBot and PerplexityBot: tools compared 2026-11-02 · 2 views · 4 min](/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared)[How to verify Amazonbot and Amzn-SearchBot by IP address 2026-11-02 · 2 views · 3 min](/guides/how-to-verify-amazonbot-and-amzn-searchbot)[Claude-User uses two published address lists. We checked one 2026-11-13 · Corrections · 1 view · 2 min](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[Google-Extended has no user agent. 1,582 requests used it Exclusive 2026-11-02 · Research · 1 view · 5 min](/blog/a-name-google-says-it-never-sends)[Fake GPTBot and ClaudeBot requests scan for .env files Exclusive 2026-11-02 · Case studies · 1 view · 4 min](/blog/the-ai-crawler-that-wanted-your-env-file)[Who reads llms.txt? Our check called 8 of 10 unverified 2026-11-19 · Corrections · 6 min](/blog/our-llms-txt-reader-check-called-8-of-10-readers-unverified)[Our mirror relayed real crawlers and inflated forgery counts 2026-11-13 · Corrections · 2 min](/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts)[AI crawler IP ranges from 18 operator lists, in one file 2026-11-13 · Releases · 2 min](/blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file)[Is your host blocking ClaudeBot? Group the 403s by IP first 2026-11-02 · Corrections · 6 min](/blog/we-almost-published-that-our-host-was-blocking-claude)[Which AI crawlers can't be verified by IP range? 2026-11-02 · Research · 5 min](/blog/the-crawlers-nobody-can-check)[How much AI crawler traffic is forged? We check the IPs Exclusive 2026-11-02 · Research · 6 min](/blog/how-much-ai-crawler-traffic-is-forged)[AI agents acting for people: 613 visits, one read robots.txt Exclusive 2026-11-02 · Research · 3 min](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt)[Agent conduct, October 2026: 34,676 verified requests Exclusive 2026-11-02 · Research · 2 min](/blog/agent-conduct-october-2026)

11 guides · 9 findings

## What crawlers are actually served

Status codes, caches, JavaScript and duplicate hosts: the bytes a crawler receives next to what a browser gets.

[2026-11-02 6 views · 5 min ### Hreflang audit: fixing return-link and x-default errors Hreflang tells a search engine which language or regional version of a page to show. It only counts when every version links back. The rules, the six errors that make Google ignore a set, and how to audit it from the outside. HOST_DUPLICATE_2005.5% Read the measurements →](/guides/hreflang-audit-return-links-x-default-and-the-common-errors)[2026-11-02 · Research · Exclusive 6 views · 6 min ### AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open. Read the measurements →](/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites)[2026-11-02 · Case studies 4 views · 4 min ### Why word count cannot spot a parked domain: 783 words A live domain-sale lander carried 783 words and passed every thin-content check we ran. What actually separates a parked page from a real site, and how to check one yourself from the page source. Read the measurements →](/blog/a-783-word-parking-page-defeats-every-thin-content-check)

[Stale cached robots.txt or llms.txt? Two fetches tell you 2026-11-02 · 2 views · 7 min](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge)[AI crawlers refused, redirected or served less: how to check 2026-11-02 · 2 views · 4 min](/guides/crawler-delivery-comparison-refused-redirected-or-served-less)[www and non-www both return 200? A canonical won't fix it 2026-11-02 · Case studies · 1 view · 3 min](/blog/our-canonical-was-right-and-the-site-was-still-served-twice)[What AI crawlers receive from local business sites 2026-11-02 · Research · 1 view · 6 min](/blog/what-a-crawler-actually-receives)[HTTP status codes for crawlers: what a 200 can hide 2026-11-02 · 1 view · 6 min](/guides/http-status-codes-what-a-crawler-actually-gets)[Canonical URL audit: do tags, redirects and sitemap agree? 2026-11-02 · 1 view · 6 min](/guides/canonical-url-audit-one-page-one-address)[Text-to-HTML ratio: does it matter, and how to check it 2026-11-02 · 1 view · 5 min](/guides/text-to-html-ratio-when-a-page-is-mostly-code)[Markdown for AI agents: serve text/markdown by negotiation 2026-11-02 · 1 view · 4 min](/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation)[Core Web Vitals without CrUX: field data for small sites 2026-11-02 · 1 view · 5 min](/guides/core-web-vitals-field-data-for-low-traffic-sites-without-crux)[Crawl budget: measure what a crawler reads in 50 fetches 2026-11-02 · 1 view · 6 min](/guides/crawl-budget-how-much-of-your-site-a-crawler-reads-in-50-fetches)[How to check if your website supports HTTP/3 2026-11-02 · 1 view · 3 min](/guides/http-3-how-to-check-whether-your-site-offers-it)[A slider that took 8.8 seconds: three 328 KB PNGs 2026-11-13 · Case studies · 2 min](/blog/a-slider-that-took-8-seconds-three-328-kb-pngs)[Do AI crawlers render JavaScript? Check what they see 2026-11-02 · 6 min](/guides/do-ai-crawlers-render-javascript)[Lighthouse scores vary between runs: two runs, 225% apart 2026-11-02 · Research · 3 min](/blog/one-lighthouse-run-cannot-support-a-finding-225-percent-variance)[Text extractors dropped our schema and still got us right 2026-11-02 · Research · 6 min](/blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway)[Is your auditor grading your firewall? A two-request test 2026-11-02 · Corrections · 4 min](/blog/the-site-that-scored-15-because-it-refused-us)[Your cache purge returned 200 and evicted nothing 2026-11-02 · Case studies · 6 min](/blog/the-purge-returned-200-and-evicted-nothing)

5 guides · 7 findings

## AI agents and MCP

What an AI agent can see, use and verify on a site, and the MCP servers agents connect to.

[2026-11-02 10 views · 6 min ### How an AI agent should verify a website before it acts In one benchmark run, four AI models caught every lookalike domain on their own and missed the dangers a page cannot show. What an agent should check before it reads, cites, connects or pays, and what counts as evidence. Read the measurements →](/guides/how-an-ai-agent-verifies-a-website-before-it-acts)[2026-11-02 · Research · Exclusive 3 views · 3 min ### AI agents cannot refuse what they cannot see: 27 traps A revoked credential and an out-of-date fact look exactly like a valid one on the page. So the models walked straight through them. Read the measurements →](/blog/ai-agents-cannot-refuse-what-they-cannot-see)[2026-11-02 · Research · Exclusive 2 views · 5 min ### MCP Registry census: what 25,983 endpoints answered We sent the MCP handshake to every remote endpoint the official MCP Registry lists, read the tool lists that came back, and read every flagged description by hand. Most of it is clean. The exceptions matter because an agent connects to them without a person reading a word. Read the measurements →](/blog/mcp-registry-census-1851-servers-what-they-answer)

[MCP servers that ask for a login: what 1,366 request first Exclusive 2026-11-02 · Research · 2 views · 4 min](/blog/mcp-servers-that-ask-for-a-login-what-they-request-by-default)[Agent readiness checkers: Is It Agent Ready? vs CrawlCheck 2026-11-02 · 1 view · 6 min](/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck)[What is the agentic web? What it means for your website 2026-11-02 · 1 view · 6 min](/guides/what-the-agentic-web-means-for-a-site-owner)[How to check an MCP server is safe before an agent connects 2026-11-02 · 1 view · 3 min](/guides/how-to-check-an-mcp-server-before-an-agent-connects)[Can an AI agent use your contact form? A two-minute test 2026-11-02 · 1 view · 4 min](/guides/can-an-ai-agent-use-your-contact-form)[MCP reliability, October 2026: 39,484 handshakes measured Exclusive 2026-11-02 · Research · 1 view · 2 min](/blog/mcp-reliability-october-2026)[One in four MCP hosts changed its tools in four days. Nothing told the agent. 2026-11-14 · Research · 3 min](/blog/one-in-four-mcp-hosts-changed-its-tools-in-four-days)[Both our SDKs cut a tool list at 300 with no warning 2026-11-13 · Corrections · 2 min](/blog/our-sdks-cut-a-tool-list-at-300-with-no-warning)[Do AI agents stop before acting on a lookalike? We tested 4 Exclusive 2026-11-02 · Research · 4 min](/blog/do-ai-agents-stop-before-acting-on-a-lookalike-site)

6 guides · 4 findings

## llms.txt, sitemaps and machine files

The files written for machines, and whether anything reads them.

[2026-11-02 9 views · 12 min ### XML sitemap audit: check the file and every URL in it A sitemap can return 200, open in a browser and still be unusable. The audit has two layers, verify the document and then verify every URL it declares, and most audits stop after the first. NO_SITEMAP_FOUND9.8% ROBOTS_NO_SITEMAP9.2% DECLARED_SITEMAP_BROKEN0.8% +6 Read the measurements →](/guides/xml-sitemap-audit)[2026-11-02 7 views · 4 min ### Does llms.txt work? What is known and how to test it No model provider documents it as an input, so the honest answer is a test, not an opinion. We ran that test on our own domain, and you can run it on yours. NO_LLMS_TXT34.2% ROBOTS_NOT_2004.7% ROBOTS_IS_HTML0.1% +5 Read the measurements →](/guides/does-llms-txt-actually-work)[2026-11-02 · Research 4 views · 5 min ### Linked from every page, our llms.txt was fetched 8 times In two weeks, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io and asked for llms.txt eight times. Per URL, that is the rate of an ordinary page, not a front door. Read the measurements →](/blog/our-llms-txt-is-linked-from-every-page-crawlers-fetch-it-like-a-blog-post)

[IndexNow: which engines use it and how to check it worked 2026-11-06 · 3 views · 6 min](/guides/indexnow-what-it-does-who-reads-it-and-how-to-check-it-worked)[Best llms.txt generators and validators compared 2026-11-02 · 3 views · 4 min](/guides/best-llms-txt-generators-and-validators-compared)[security.txt (RFC 9116): fields, location and expiry rule 2026-11-02 · 2 views · 4 min](/guides/security-txt-rfc-9116-fields-expiry-and-where-to-serve-it)[How to write an llms.txt and check it is actually served 2026-11-02 · 1 view · 6 min](/guides/how-to-write-an-llms-txt-and-verify-it)[Booking links counted as missing sitemap pages 2026-11-13 · Corrections · 2 min](/blog/booking-links-counted-as-missing-sitemap-pages)[agents.md: AI agent instructions you may not have written 2026-11-02 · Case studies · 6 min](/blog/the-instructions-on-your-domain-you-did-not-write)[Sitemap lastmod: 103 pages, one date, why engines skip it 2026-11-02 · Case studies · 5 min](/blog/sitemap-lastmod-103-pages-one-date-why-engines-ignore-it)

4 guides · 4 findings

## Schema and entities

Structured data that parses, and structured data that is true.

[2026-11-02 4 views · 5 min ### Rich Results Test vs Schema Markup Validator vs CrawlCheck Three free structured-data tools answer three questions: Google rich-result eligibility, schema.org validity, and whether the markup describes one business that matches the page. ENTITY_COLLISION2.6% ENTITY_NO_COORDINATES6.1% NAP_UNDECLARED_LISTING3.6% +1 Read the measurements →](/guides/schema-validators-compared-rich-results-test-vs-schema-markup-validator)[2026-11-02 3 views · 6 min ### What is entity SEO? What machines must resolve to cite you Being read is not being identified. If a system cannot tell you apart from a similar business two towns over, it names the wrong one or nobody, and both look the same from your side. ENTITY_COLLISION2.6% ENTITY_NO_COORDINATES6.1% Read the measurements →](/guides/what-is-entity-seo)[2026-11-02 2 views · 6 min ### JSON-LD @id errors: what breaks and how to fix it Six identifier defects no vocabulary validator reports: blank nodes, path @ids that 404, Unicode normalisation, raw IDN hosts, encoding drift and rival roots. Plus the duplicate that is not a defect. ENTITY_COLLISION2.6% Read the measurements →](/guides/json-ld-id-and-iri-errors-what-breaks-and-how-to-fix-them)

[A newspaper’s schema coordinates put it in China 2026-11-02 · Case studies · 1 view · 6 min](/blog/the-newsroom-that-published-coordinates-in-china)[Lazy-load placeholders: why image checks miss your photos 2026-11-02 · Corrections · 1 view · 4 min](/blog/our-scanner-counted-placeholders-not-photographs)[LocalBusiness schema for service-area businesses: example 2026-11-02 · 1 view · 6 min](/guides/localbusiness-schema-for-service-area-businesses)[areaServed vs service area pages: 20 towns declared, 1 page 2026-11-02 · Case studies · 4 min](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them)[Brand schema for multi-location businesses: link both ways 2026-11-02 · Case studies · 4 min](/blog/the-parent-declared-the-chain-none-of-the-locations-declared-it-back)

2 guides · 9 findings

## Local business data

Where maps and AI assistants get business facts, and how to keep yours consistent.

[2026-11-02 · Research 5 views · 6 min ### Does Data Axle feed Google? Where business data comes from Local SEO advice says data aggregators feed Google. Google's own notice names no US supplier, no AI assistant documents Data Axle as a source, and eight of nine businesses we checked have no Data Axle record. NAP_UNDECLARED_LISTING3.6% Read the measurements →](/blog/where-google-and-ai-assistants-get-business-data)[2026-11-02 · Research · Exclusive 3 views · 6 min ### Google's Maps provider list: 793 names, no Yelp A recovered Geostore enumeration names every source a fact about a place can come from. The consumer directories that citation services sell by the dozen are absent. The business's own website is present, with the same standing as a purchased feed. Read the measurements →](/blog/googles-maps-provider-list-has-no-yelp-in-it)[2026-11-02 · Research 2 views · 4 min ### Google Places API vs Google Maps rankings: why they don't match (tested in 4 cities) Same query, four Denver-area cities, one day. The API and Maps agreed on who competes, disagreed on the order, and the API could not find two businesses Maps ranks. Read the measurements →](/blog/google-places-api-ranking-is-not-google-maps-ranking)

[How to get your local business recommended by AI 2026-11-02 · 2 views · 4 min](/guides/how-to-get-your-local-business-recommended-by-ai-assistants)[AI picked a local contractor 422 times: rating decided it 2026-11-02 · Research · 1 view · 3 min](/blog/ai-picked-a-local-contractor-422-times-the-rating-decided-it)[NAP consistency: how to check your name, address and phone 2026-11-02 · 1 view · 5 min](/guides/nap-consistency-audit-phone-address-and-coordinates)["20 years of experience": the site's own records said 13 2026-11-18 · Case studies · 3 min](/blog/a-site-said-20-years-of-experience-its-own-records-said-13)[Opening hours and prices: the facts sites don’t publish 2026-11-02 · Research · 3 min](/blog/the-two-facts-almost-nobody-publishes)[utm_source=chatgpt.com in a lead: what it proves 2026-11-02 · Case studies · 3 min](/blog/a-tree-service-lead-arrived-with-utm-source-chatgpt-com)[Four AI models, three local businesses: who got it right 2026-11-02 · Research · 6 min](/blog/what-four-ai-models-told-customers-about-our-three-businesses)[Google AI Overview sources for Denver tree removal cost 2026-11-02 · Research · 3 min](/blog/google-ai-overview-tree-removal-cost-denver-six-sources)

11 guides · 7 findings

## Measuring AI visibility

What visibility scores and audit tools measure, compared, with CrawlCheck included.

[2026-11-18 15 views · 9 min ### Web scraping APIs compared: the real cost per page ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl, Apify and Browse AI converted from credits into pages, plus Crawl4AI, Scrapy and Jina Reader for free, and the one question none of them answers about your own site. ANSWER_ENGINE_REFUSED5.8% CRAWLER_SERVED_LESS1.1% CHALLENGE_SERVED_2000.3% +3 Read the measurements →](/guides/best-web-scraping-apis-compared)[2026-11-02 11 views · 5 min ### YouTube SEO case study: 81 videos rewritten, measured A dormant channel had 81 titles and descriptions rewritten in two dated batches, with the baseline published first. Search views rose 19% against a flat year-earlier control; the flat and falling rows are printed beside it. Read the measurements →](/guides/youtube-seo-case-study-81-videos-rewritten-measured-before-and-after)[2026-11-02 10 views · 3 min ### How to read a CrawlCheck report: grade, fixes, evidence What each part of the report answers, which part to open first, and how to check a finding against the bytes behind it. Read the measurements →](/guides/how-to-read-a-crawlcheck-report)

[AI visibility tools compared: what each one measures 2026-11-02 · 8 views · 7 min](/guides/ai-visibility-tools-what-each-one-measures)[How to measure AI visibility yourself: three checks 2026-11-02 · 8 views · 5 min](/guides/how-to-measure-ai-visibility)[AI visibility vendors audited: 11 of 23 have llms.txt Exclusive 2026-11-02 · Research · 5 views · 3 min](/blog/we-scanned-the-tools-that-audit-your-site)[Google's /goto redirect: why referrer data can't show it 2026-11-02 · Case studies · 3 views · 5 min](/blog/we-clicked-a-real-google-result-to-test-the-goto-redirect)[Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits 2026-11-02 · 3 views · 9 min](/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit)[Free AI visibility checkers: how to tell if one is any good 2026-11-02 · 2 views · 5 min](/guides/free-ai-visibility-checkers-what-they-measure)[Missing referrer is not direct traffic: 93% of our arrivals sent none 2026-11-02 · Research · 2 views · 3 min](/blog/ninety-three-percent-of-our-arrivals-sent-no-referrer)[GEO vs AEO vs LLM SEO: the terms, and what is measurable 2026-11-02 · 1 view · 4 min](/guides/geo-aeo-llm-seo-what-the-terms-mean)[AI crawler monitoring vs uptime monitors: what each one catches 2026-10-03 · 1 view · 4 min](/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors)[How to rank in ChatGPT: the part that is in your control 2026-11-02 · 4 min](/guides/how-to-rank-in-chatgpt)[AI text watermarks only work if the provider opted in 2026-11-02 · Research · 3 min](/blog/ai-text-watermarks-only-work-if-the-provider-opted-in)[AI visibility scores measure an API, not ChatGPT 2026-11-02 · Research · 4 min](/blog/ai-visibility-scores-measure-an-api-not-chatgpt)[Site audits go stale: why one clean scan proves little 2026-11-02 · Research · 3 min](/blog/your-site-was-clean-the-day-you-audited-it)[Did your SEO fix work? Compare fixed and untouched pages 2026-11-02 · 5 min](/guides/how-to-tell-if-a-website-fix-moved-traffic-fixed-vs-untouched-pages)[AI citations on two dates: seven matched questions 2026-11-02 · Research · 4 min](/blog/seven-matched-questions-ai-citations-october-2026)

7 guides · 6 findings

## Site health and security

Links, contrast, security headers and email: checks that break quietly.

[2026-11-02 7 views · 11 min ### How to test text color contrast on hover and keyboard focus Static contrast checkers measure a page at rest. On our own site, hover and focus found 65 more failure patterns than the at-rest check did. Six steps to test them, and the usual causes. Read the measurements →](/guides/how-to-test-color-contrast-in-hover-and-focus-states)[2026-11-02 4 views · 9 min ### How to move DMARC from p=none to p=quarantine without losing email The sequence for tightening DMARC safely: inventory every sender, prove alignment from received headers, create the report mailbox first, then raise the policy in stages. Read the measurements →](/guides/how-to-move-dmarc-from-none-to-quarantine-without-losing-mail)[2026-11-02 3 views · 10 min ### How to find broken internal links and dead anchor links on your site A link audit is an inventory, a fetch, a classification and a second pass over in-page anchors. On our own site, 2,260 link targets came down to four real defects, all from shared templates. Read the measurements →](/guides/how-to-audit-internal-links-and-dead-anchors)

[How to find websites using your brand name, and prove you were first 2026-11-02 · 3 views · 8 min](/guides/how-to-check-who-else-uses-your-brand-name-online)[CSP report-only: how to roll out a Content-Security-Policy without breaking your site 2026-11-02 · 2 views · 6 min](/guides/content-security-policy-rollout-report-only-to-enforced)[Staging site leak: how to find a staging URL in your live pages and close it 2026-11-02 · 2 views · 5 min](/guides/staging-site-leak-how-a-staging-host-ends-up-in-your-live-pages)[18 DMARC reports, 7 domains: every failure was a spoofer 2026-11-06 · Case studies · 1 view · 3 min](/blog/eighteen-dmarc-reports-every-failure-was-spoofing)[The quote form was the slow part: 980 ms cut to 50 ms 2026-11-18 · Case studies · 3 min](/blog/the-quote-form-was-the-slow-part-980-ms-to-50)[A newsletter form that lost every sign-up 2026-11-13 · Case studies · 2 min](/blog/a-newsletter-form-that-lost-every-sign-up)[Six in ten websites already use post-quantum encryption. Whether yours does mostly depends on one thing 2026-11-03 · Research · 3 min](/blog/post-quantum-key-exchange-six-in-ten-websites-november-2026)[Email bounce case study: one typo, every address dead 2026-11-02 · Case studies · 4 min](/blog/one-character-of-undeliverable-email)[Page builders drop the main landmark: nine HTML checks 2026-11-02 · Releases · 3 min](/blog/accessibility-structure-nine-things-a-screen-reader-needs-before-contrast-matters)[HSTS check: missing header, bad preload, removed entries 2026-11-02 · 6 min](/guides/hsts-audit-missing-header-preload-ineligible-and-removed-entries)

7 findings

## Web-wide census

Measured across the web's homepages, not one site.

[2026-11-02 · Research · Exclusive 2 views · 7 min ### State of AI visibility, October 2026: 139,788 homepages The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see. ROBOTS_RULES_SHADOWED14% STALE_CACHE_SERVED13.4% NO_LLMS_TXT34.2% +2 Read the measurements →](/blog/state-of-ai-visibility-october-2026-139788-homepages)[2026-11-02 · Research · Exclusive 1 view · 3 min ### Only 6% of websites block an AI crawler: the full breakdown Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site. Read the measurements →](/blog/only-six-percent-of-websites-block-an-ai-crawler)[2026-11-02 · Research · Exclusive 1 view · 2 min ### Which AI crawlers the top 10,000 sites block most, October 2026 CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says. Read the measurements →](/blog/which-ai-crawlers-top-sites-block-most-october-2026)

[Top 1,000 websites: every outside tracker we logged 2026-11-15 · Research · 4 min](/blog/top-1000-websites-third-party-trackers-november-2026)[One Cloudflare setting is behind most of the new AI training blocks on top sites 2026-11-03 · Research · 4 min](/blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026)[MobileOptimized tag: on 1.65M sites, none in the top 10,000 2026-11-02 · Research · 3 min](/blog/the-meta-tag-a-million-sites-have-and-the-top-ten-thousand-do-not)[Robots.txt allows it, the server refuses: 57 top sites Exclusive 2026-11-02 · Research · 3 min](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)

1 guide

## More findings

Findings not yet filed under a topic.

[2026-11-15 3 min ### People-search opt-outs: 7 sites tested, every step Every step seven people-search sites asked for when we tried to remove one person's listing, and how to do it yourself. Read the measurements →](/guides/people-search-opt-out-seven-sites-tested)

#### 38 corrections and release notes

## How we fix CrawlCheck itself

Every defect found in CrawlCheck, told against ourselves, with what changed.

[Our receipt checker would have rejected our own receipts 2026-11-02 · Corrections · 13 views · 2 min](/blog/our-receipt-checker-would-have-rejected-our-own-receipts)[For a week, CrawlCheck told AI agents not to pay us 2026-11-02 · Corrections · 4 views · 2 min](/blog/for-a-week-crawlcheck-told-agents-not-to-pay-us)[Most of our 'referring pages' were forged Referer headers 2026-11-06 · Corrections · 3 views · 3 min](/blog/most-of-our-referring-pages-were-forged-referer-headers)[Three of our own checks passed because we measured nothing 2026-11-02 · Corrections · 2 views · 2 min](/blog/three-of-our-own-rows-passed-because-we-measured-nothing)[Our crawler counts missed about 45% of hits 2026-11-06 · Corrections · 1 view · 3 min](/blog/our-crawler-counts-missed-45-percent-of-hits)[Our AI accuracy check ran for six days and measured nothing 2026-11-06 · Corrections · 1 view · 3 min](/blog/our-ai-accuracy-check-ran-for-six-days-and-measured-nothing)[OpenTimestamps proof says pending? Upgrade the receipt 2026-11-02 · Corrections · 1 view · 2 min](/blog/our-proofs-said-pending-for-26-days)[Our audit labelled one page and measured another for weeks 2026-11-02 · Corrections · 1 view · 2 min](/blog/our-audit-labelled-one-page-and-measured-another)[Dead link or blind spot? When a checker's failure is its own 2026-11-02 · Corrections · 1 view · 2 min](/blog/the-scanner-that-cannot-fetch-its-own-host)[CrawlCheck in German, French, Spanish and Portuguese 2026-11-02 · Releases · 1 view · 2 min](/blog/crawlcheck-in-german-french-spanish-and-portuguese)[Unblocking AI crawlers: what an access repair proves 2026-11-02 · Corrections · 1 view · 2 min](/blog/unblocking-ai-crawlers-access-is-not-a-citation-result)[14 of 43 crawlers in our catalogue were classed wrong 2026-11-13 · Corrections · 3 min](/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong)[Release notes: 7 to 13 November 2026 2026-11-13 · Releases · 2 min](/blog/release-notes-7-to-13-november-2026)[Unquoted HTML attributes: why we said a viewport was missing 2026-11-02 · Corrections · 2 min](/blog/the-viewport-was-there-all-along)[Why a web server can gain and lose 43 million sites 2026-11-02 · Corrections · 2 min](/blog/the-web-server-that-gained-and-lost-43-million-sites)[A save that returned success rewrote every newline 2026-11-02 · Corrections · 2 min](/blog/a-write-that-returned-success-and-corrupted-every-newline)[Analytics added 650 ms to every request: how we found it 2026-11-02 · Corrections · 2 min](/blog/our-analytics-was-adding-650ms-to-every-request)[When an LLM summarises your audit report and gets it wrong 2026-11-02 · Corrections · 2 min](/blog/the-grade-that-could-not-explain-itself)[Why caching only successes kept our scans slow 2026-11-02 · Corrections · 2 min](/blog/cache-your-misses-a-lookup-that-only-caches-on-success)[A placeholder check flagged NaN. It was inside a URL 2026-11-02 · Corrections · 2 min](/blog/nan-inside-a-url-slug-a-placeholder-check-that-accused-a-newspaper)[Seven site types, and the report checks we corrected 2026-11-02 · Corrections · 3 min](/blog/we-read-every-scored-row-against-seven-sites-here-is-what-we-unscored)[Correction: our paywall said sites with findings were clean 2026-11-02 · Corrections · 2 min](/blog/the-paywall-that-said-your-site-was-clean)[A 200 status hid an empty page for weeks 2026-11-02 · Corrections · 2 min](/blog/the-control-panel-that-answered-200-with-nothing-in-it)[Sites measured is not people: what our own logs showed 2026-11-02 · Corrections · 2 min](/blog/our-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week)[Correction: the owner's own scans came back locked 2026-11-02 · Corrections · 2 min](/blog/the-owner-could-not-read-his-own-scans)[Release notes 29 Sep 2026: finding states, sitemap, trust 2026-11-02 · Releases · 2 min](/blog/release-notes-29-september-2026-finding-states-sitemap-index-trust-center)[Five days of our alerts went to an address with no mailbox 2026-11-02 · Corrections · 2 min](/blog/five-days-of-our-alerts-went-to-an-address-with-no-mailbox)[Our security headers skipped every signed-in session 2026-11-02 · Corrections · 2 min](/blog/our-security-headers-skipped-every-signed-in-session)[Our extension read a page as 36.5% JavaScript. It was zero. 2026-11-02 · Corrections · 2 min](/blog/the-page-our-extension-read-as-36-percent-javascript-was-zero)[Eight contradictions in our own copy, rechecked 2026-11-02 · Corrections · 2 min](/blog/eight-contradictions-in-our-own-copy-rechecked)[Release notes 3 Oct 2026: Resolve, census, rulebook 2026-11-02 · Releases · 3 min](/blog/release-notes-3-october-2026-resolve-protocol-census-rulebook)[Share of preflight, October 2026: 0 of 5,643 requests 2026-11-02 · Releases · 2 min](/blog/share-of-preflight-october-2026)[Release notes 14 Oct 2026: doors, weekly figures, trial 2026-11-02 · Releases · 3 min](/blog/release-notes-14-october-2026-doors-signed-weekly-figures-watch-trial)[Our directory froze for 20 hours and logged no error 2026-11-02 · Corrections · 3 min](/blog/a-cron-job-ran-one-minute-late-and-our-directory-froze-for-20-hours)[An access rule on one path locked our customers out 2026-11-02 · Corrections · 2 min](/blog/an-access-rule-on-one-path-locked-our-customers-out-of-sign-in)[Three sections said “not scored”. Our grade counted them. 2026-11-02 · Corrections · 2 min](/blog/three-report-sections-said-not-scored-the-grade-counted-them)[One 503 is not a blocked site: refusal or hiccup? 2026-11-02 · Corrections · 4 min](/blog/one-503-is-not-a-blocked-site-how-to-tell-a-refusal-from-a-hiccup)[AI answer corrections: three clocks, not one update time 2026-11-02 · Corrections · 3 min](/blog/ai-business-corrections-three-clocks-not-one)

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

## Questions about this page

#### Q Are these real sites?

Yes, every finding is a measurement taken on a live site. Domains are not named unless the author owns them.

#### Q How is a finding different from a guide?

A finding reports one measured case and what it showed. A guide explains how to run and read the check yourself.

## Keep reading

[The dataset behind these write-ups Every figure in these posts is recomputed live from it.](/data)[The crawler reference Who sends each named agent, and what it is for.](/ai-crawlers)[Run the checks yourself The two scripts these findings came from.](/tools)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [contentsquare.com](/registry/contentsquare.com) · [foryouproperty.com](/registry/foryouproperty.com) · [localgtm.com](/registry/localgtm.com) · [slack.com](/registry/slack.com) · [Trade: Roofing](/registry/trade/roofing) · [stripe.dev](/registry/capabilities/stripe.dev) | **Datasets:** [Technologies: Calendly](/datasets/technologies/calendly) · [Problems: Image format mismatch](/datasets/problems/image-format-mismatch)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
