# CrawlCheck - full technical guide for agents and developers

> CrawlCheck is the verification layer for the agentic web. Before an agent reads, cites, connects to or transacts with a site, one call returns a signed, offline-verifiable answer about it: what each named crawler was really served, what the site declares (robots.txt, sitemap, llms.txt, agents.md, entitymap.json, MCP and API surfaces), and what an agent may do there. Every figure comes from a fetch made at measurement time.

## Before an agent acts

- GET https://crawlcheck.io/api/v1/resolve?domain=example.com - one Ed25519-signed answer about the domain: per-action decisions for read, cite, connect and transact, freshness, and evidence links. Spec: https://crawlcheck.io/spec/resolve. Keys: https://crawlcheck.io/keys. Verify offline with https://crawlcheck.io/crawlcheck-verify.mjs.
- POST https://crawlcheck.io/api/v1/preflight - body {"domain":"example.com","action":"transact"} - allow, warn, require_confirmation or block for that one action, with a receipt the visited site can check.
- GET https://crawlcheck.io/api/v1/visitor?ip=&ua= - whether a request claiming a named crawler came from its operator's published ranges: verified, spoofed or unverifiable.
- POST https://crawlcheck.io/api/v1/outcome - report what happened when you acted, so the record can be corrected.

This file is the technical companion to https://crawlcheck.io/llms.txt. It covers the API, the MCP server, the record format, and how to read a grade honestly.

## Endpoints (the 238 documented; everything else is internal)

- POST https://crawlcheck.io/api/scan - body {"domain":"example.com"} - returns the scan record (JSON). Rate limit 5 per 10 seconds per address; 429 means wait, not failed.
- POST https://crawlcheck.io/api/tool/verify - body {"lines":"<raw access-log lines>"} - each crawler claim adjudicated against the operator's published IP ranges: verified, spoofed, or unverifiable.
- GET https://crawlcheck.io/api/telemetry - verified-crawler traffic observed at this origin.
- GET https://crawlcheck.io/api/corpus/state - dataset coverage by platform, rendering, size, language, kind.
- GET https://crawlcheck.io/api/public/counts - the figures this site quotes about itself; CORS open, cached 60 s.
- GET https://crawlcheck.io/api/registry?host= or ?shelf= - the registry: which measured domains serve an llms.txt, agents.md, media kit, reciprocity-tested entity graph, named AI access policy or agent-callable surfaces.
- GET https://crawlcheck.io/api/globe.json - every city a visitor scan has been run from, all recorded time, aggregated; no domain is named.
- GET https://crawlcheck.io/api/tool/robots?domain=&path= - every named crawler resolved against the domain's robots.txt, group shadowing shown.
- GET https://crawlcheck.io/api/tool/llms?domain= - an llms.txt draft from the domain's own pages (&format=txt for the file).
- GET https://crawlcheck.io/api/fix/robots?domain= - the served robots.txt, corrected, every change listed at the top.
- GET https://crawlcheck.io/api/fix/entitymap?domain= - a starter entitymap.json from the latest scan record; needs a prior scan.
- GET https://crawlcheck.io/api/flow?domain=&agent=&fresh= - the observed data flow: where each crawler identity's path dies (edge, robots.txt file, robots.txt rules) and which stores it reached; scans first when no record is on file.
- GET https://crawlcheck.io/api/video?v= | /api/video/channel?handle= | /api/video/site?domain= - YouTube video, channel and site checks.
- Licence-gated for a domain the key covers (x-crawlcheck-key, 404 otherwise): GET /api/flow/gate?domain=&fresh=&since= (CI: 200 pass, 412 fail when an open crawler path closes or a declared path is violated); GET/PUT/DELETE /api/flow/declared?domain= (the owner's declared flow; PUT {"seed":"observed"} to start from the latest scan).
- Licence-gated (x-crawlcheck-key, 404 without): POST /api/scan/licensed, POST /api/depth (result at GET /api/depth/{id}), POST /api/entity/observe.

OpenAPI 3.1: https://crawlcheck.io/openapi.json. Human reference: https://crawlcheck.io/docs/api. Everything else the service answers is internal and may change.

## MCP server

Streamable HTTP, stateless, JSON-RPC 2.0 at POST https://crawlcheck.io/mcp. Card: https://crawlcheck.io/.well-known/mcp/server-card.json. Tools: scan_domain {domain}, verify_crawler_log {lines}, corpus_state {}, resolve_robots {domain, path?}, draft_llms {domain}, fix_robots {domain}, fix_entitymap {domain}, video_check {v, fresh?}, video_channel {handle | id}, video_site {domain, handle?}, crawler_path {domain, agent?, fresh?}, public_counts {}, telemetry {}, fix_mostly_code {domain | id}, fix_stale_cache {domain | id}, fix_machine_chain {domain | id}, explain_finding {id | domain, code?, path?}, list_verified_capabilities {domain, verified_only?}, machine_record {id | domain} - one per key-free endpoint (explain_finding and machine_record open every finding when the caller's x-crawlcheck-key licenses the domain). Each tool is a wrapper over the endpoint above with the same rate limit; scan_domain returns a trimmed record plus report_url and record_url.

## Reading a record

- grade: A-F, or null. null means the scan could not be graded - most often refused=true, meaning the homepage and a randomised path that cannot exist answered identically (a bot wall, not a site). Report that as unmeasured. It is never a low score.
- overall: AI visibility 0-100, composed as reach (can a crawler get in), then read (can it parse what came back), then quote (is there a specific fact to cite). Reach is a ceiling: nothing downstream of a refusal reaches an engine.
- section_scores[]: id, title, score. A null score means the section's inputs were not measurable and it is excluded, never counted as zero.
- findings[]: code, severity 1-5, path, title, detail, meaning, evidence. The letter grade is capped by the worst finding (medium -> C, high -> D, critical -> F) and the cap reason is stated in grade_detail.
- self_audit: invariants checked on the record itself. A finding here is a defect in CrawlCheck, not in the site.
- headroom: arithmetic on the failing rows - the points each would add if fixed. Not a forecast.

## Report and record URLs

https://crawlcheck.io/r/{id} renders the report; https://crawlcheck.io/r/{id}.json exports it as PublicReportV1 (the /api/v1 shape), .csv the findings, and .raw.json the internal record with a licence key. Reports regenerate from the stored record, so a link stays true to the measurement it was taken from.

## What this tool does not do

It does not impersonate crawlers to get past a block - being blocked is the measurement. It cannot say whether ChatGPT or Perplexity cites a site; nobody can from the server side. It cannot scan crawlcheck.io itself.

## Every standing page

- https://crawlcheck.io/api-terms - API terms: what a key may be used for, quotas, attribution, retention
- https://crawlcheck.io/listing-check - free check: the business record your site publishes against what directories print, and joining the registry
- https://crawlcheck.io/how-it-works - How one answer is made: measured from outside, decided by published rules, signed, and verifiable offline
- https://crawlcheck.io/compare - compare two sites side by side: what each one delivers to the same crawler identities, graded the same way
- https://crawlcheck.io/llms-txt-audit - llms.txt audit: does the file resolve, parse and match the pages the site declares
- https://crawlcheck.io/robots-txt-for-ai-crawlers - robots.txt resolved per AI crawler, compared with what the edge actually served each one
- https://crawlcheck.io/entity-map-validator - entitymap.json parsed and checked against the page JSON-LD and every sameAs target
- https://crawlcheck.io/seo-audit-tool - free SEO audit tool: technical SEO checks plus what each AI and search crawler is served, graded A to F with a fix list
- https://crawlcheck.io/technical-seo-audit - technical SEO audit: robots.txt, sitemap, hosts, HTTPS/HSTS, internal links, rendering and Core Web Vitals, every crawler compared
- https://crawlcheck.io/local-seo-audit-tool - local SEO audit tool: name, phone, address, service areas and LocalBusiness schema checked against the directories
- https://crawlcheck.io/white-label-seo-audit - SEO audit reports for agencies: branded, client-safe reports on 10 or 40 client domains, re-measured twice a day
- https://crawlcheck.io/what-is-a-website-audit - what a website audit is, what it checks and costs, and a free audit to run now
- https://crawlcheck.io/what-is-an-seo-audit - what an SEO audit is, what it should include, the order to fix things, and a free audit to run
- https://crawlcheck.io/spec/guard - the guard acceptance suite: 45 signed cases every agent guard must pass before acting
- https://crawlcheck.io/assurance - assurance: availability against target, the checks before acting on a receipt, who runs each observer, coverage denominators and outside adoption
- https://crawlcheck.io/aeo-audit - AEO audit: reach, read and quote, graded A to F with every defect counted
- https://crawlcheck.io/terms - terms of service: plans, billing, cancelling, acceptable use, liability
- https://crawlcheck.io/refunds - refund policy: 7 days on a first subscription charge, the fix until work starts
- https://crawlcheck.io/verify - verify a record yourself: download its evidence bundle and check the signature, Merkle path and Bitcoin timestamp in the browser or with one Node file
- https://crawlcheck.io/trust - trust center: data location and retention, security headers, email security, timestamping, outside services
- https://crawlcheck.io/status - status: whether scans, signing, the daily seal, backups, the security self-test and observers are working now, and the signed incident log
- https://crawlcheck.io/contact - contact: email, security reports, opt-out, disputes and sales, each with the route that handles it
- https://crawlcheck.io/subprocessors - subprocessors and data location: every outside service, what it receives, where each kind of data is stored, what is logged
- https://crawlcheck.io/security - security and vulnerability disclosure: how to report a problem, scope, and response times
- https://crawlcheck.io/globe - where scans come from: every city a visitor scan has been run from, all recorded time, on a globe; no domain is named
- https://crawlcheck.io/registry - the registry: every measured domain filed on the shelves it earned — llms.txt, agents.md, media kit, entity graph, AI access policy, agent-callable
- https://crawlcheck.io/for-agents - For AI agents: ask, verify and cite CrawlCheck as a verifiable source — one call with no key, a signed answer checkable offline, a permanent cite-as URL for every answer
- https://crawlcheck.io/for-sites - for site owners: scan free, have the fix list implemented, keep it fixed with a certificate — what an AI agent gets from your site
- https://crawlcheck.io/developers - for developers and vendors: the API by lane, the specifications, SDKs, signing keys and published quotas
- https://crawlcheck.io/learn - learn: guides, measured findings and the glossary, each written from a measurement with its live rate
- https://crawlcheck.io/press - press kit: the facts, live numbers with dates, the vocabulary, the marks and colours, for writing about CrawlCheck
- https://crawlcheck.io/app - the CrawlCheck app, live on the web: the site's record on the phone, receipts, and a field kit of phone-only instruments; follows /tokens.css and /app.json live
- https://crawlcheck.io/forms - a quote form in one script tag that records first- and last-touch source and files each lead beside the site record
- https://crawlcheck.io/demo - A full report on a real site beside the signed answer an agent gets about it, rendered from its latest scan
- https://crawlcheck.io/ - The verification layer for the agentic web: one signed answer about any site before an agent reads, cites, connects or transacts, and a free A-F scan for site owners
- https://crawlcheck.io/ai-crawlers - Every named AI crawler and agent, what each is for, how to verify a request, and how each has behaved
- https://crawlcheck.io/tools - Eight free checks: what an agent is told about a site, machine files, crawler claims, robots.txt per agent, llms.txt draft, compare, video, Web Bot Auth
- https://crawlcheck.io/tools/agent-view - Agent view: the signed answer an agent gets about any domain, laid out for a person — the decision per action and the reason
- https://crawlcheck.io/extension - The browser extension: what an agent is served from your own address, joined to the signed reading
- https://crawlcheck.io/rules - The rulebook: every finding CrawlCheck can raise, what it means, its fix and share of scans
- https://crawlcheck.io/fix/wordpress - How to fix crawler findings on WordPress
- https://crawlcheck.io/fix/shopify - How to fix crawler findings on Shopify
- https://crawlcheck.io/fix/webflow - How to fix crawler findings on Webflow
- https://crawlcheck.io/fix/squarespace - How to fix crawler findings on Squarespace
- https://crawlcheck.io/fix/wix - How to fix crawler findings on Wix
- https://crawlcheck.io/fix/nextjs - How to fix crawler findings on Next.js
- https://crawlcheck.io/fix/drupal - How to fix crawler findings on Drupal
- https://crawlcheck.io/fix/cloudflare - How to fix crawler findings on Cloudflare
- https://crawlcheck.io/metrics - CrawlCheck in numbers: decisions issued to AI agents, API and MCP calls, SDK downloads, false-positive rate on appeal, service objectives
- https://crawlcheck.io/executive-report - The executive report: a two-page PDF per site, monthly or weekly
- https://crawlcheck.io/disputes - Dispute and correction ledger: findings re-measured on appeal from more than one network, with signed resolutions
- https://crawlcheck.io/keys - Signing keys: the current crawler and answer signing key, retired keys, and every key event in a hash-chained log whose head is sealed daily
- https://crawlcheck.io/enterprise - Enterprise pilot: 30 days, integrate then shadow then a signed readout; SSO, SLA, private observers, MCP monitoring, metered Resolve at volume
- https://crawlcheck.io/spec/sla - Service level agreement: 99.9% monthly availability for Resolve and preflight, how it is measured from outside Cloudflare, and the service credits
- https://crawlcheck.io/spec/considerations - Errors, security and privacy considerations: the error envelope with every machine code, and what a consumer of CrawlCheck documents must check
- https://crawlcheck.io/spec/resolve - Resolve protocol: the open, versioned specification of the signed answer an AI agent reads before acting on a domain, with endpoints, signature, verification and change log
- https://crawlcheck.io/docs/observer-protocol - Open observer protocol: how anyone runs an independent observer, enrols a key and is corroborated field by field
- https://crawlcheck.io/docs/lineage-coverage - How much of the evidence path every finding family carries, computed from stored records
- https://crawlcheck.io/workflow - Problem detected to business outcome for the three most common findings, each step a link to its record
- https://crawlcheck.io/workflow/no-llms-txt - No llms.txt: the fix end to end on a real site
- https://crawlcheck.io/workflow/page-is-mostly-code - Page is mostly code: the fix end to end on a real site
- https://crawlcheck.io/workflow/stale-cache-served - Stale cache served: the fix end to end on a real site
- https://crawlcheck.io/docs/quickstart - Add CrawlCheck preflight to an agent: MCP, curl, Python, OpenAI Agents, Claude tool use, LangChain, Cloudflare Workers
- https://crawlcheck.io/docs/ci - CrawlCheck in CI: fail a pull request that breaks crawler access — the gate, the exit codes and the workflow file
- https://crawlcheck.io/docs/api/compatibility - Every API field tagged stable, provisional or internal, and the deprecation window for each
- https://crawlcheck.io/sdk - The typed SDK: install from this site with a pinned integrity hash; bundles the offline verifier
- https://crawlcheck.io/data/share-of-preflight - Monthly share of verified agent requests that carried a CrawlCheck receipt, by agent
- https://crawlcheck.io/editions - The four recurring editions: State of AI Visibility, agent conduct, MCP reliability, share of preflight
- https://crawlcheck.io/data/state-of-ai-visibility - State of AI Visibility: the monthly edition, every issue and its data
- https://crawlcheck.io/data/north-star - Decisions to outcomes: signed decisions, receipts sites checked, outcomes agents reported, and where CrawlCheck was wrong
- https://crawlcheck.io/data/agentic-web-map - Agentic web map: DNS hosts, networks, bot protection and site platforms between AI agents and the most-visited sites, with what each layer's sites do to AI crawlers; signed monthly
- https://crawlcheck.io/data/census/weekly - Weekly agentic web census edition: the most-visited sites, this week against last, figure by figure, signed
- https://crawlcheck.io/compliance - Decision receipts mapped to EU AI Act Article 12 and NIST AI RMF, with a signed export per framework
- https://crawlcheck.io/traffic-index - Agent Traffic Index: which AI and search agents reached reporting sites each month, verified against operator ranges, signed
- https://crawlcheck.io/benchmark - Agent safety benchmark: whether AI agents stop on lookalikes, poisoned tools, revoked receipts and stale answers, alone and with CrawlCheck
- https://crawlcheck.io/agents/conduct - How each AI and search agent actually behaves here: robots.txt, rate, forged claims, from verified requests only
- https://crawlcheck.io/conformance - Conformance fixtures built from this site's own records, with the expected result of every verifier check
- https://crawlcheck.io/extension/privacy - What the browser extension sends, by version — never a URL or page content
- https://crawlcheck.io/author - Who writes the findings posts here, and every post published.
- https://crawlcheck.io/docs/api - The API agents call before they act: Resolve, preflight, receipts, Visitor Resolve, MCP tools, every endpoint with an example
- https://crawlcheck.io/glossary - Definitions for every machine-layer term CrawlCheck measures.
- https://crawlcheck.io/guides - Every guide published here, and what each one measures.
- https://crawlcheck.io/datasets - Datasets: daily files of measured websites by lane, with a free sample
- https://crawlcheck.io/datasets/technologies - Websites by technology: one lead list per fingerprinted technology, priced by size
- https://crawlcheck.io/datasets/problems - Websites by problem: one lead list per finding, priced by size
- https://crawlcheck.io/datasets/ai-consent - AI consent dataset: per-site robots.txt answers for named AI agents, Content-Signal and llms.txt, signed
- https://crawlcheck.io/credits - Data credits: prepaid packs spent on any list, technology list, state or lane
- https://crawlcheck.io/data - Open data: the daily signed Resolve snapshot, the census of the most-visited sites, agent conduct and finding rates
- https://crawlcheck.io/blog - Measured case studies from real sites
- https://crawlcheck.io/pricing - Free for agents and free to scan; what Watch, Fix, Agency, the API plans and the Data add-on add
- https://crawlcheck.io/video - YouTube video check: chapters, captions, description, language and topics against what a search result can anchor to
- https://crawlcheck.io/video/channel - YouTube channel check: every upload read and tallied against the video anchor model
- https://crawlcheck.io/video/site - Does your site carry your videos: embeds, VideoObject schema, video spread across your own pages
- https://crawlcheck.io/directory - The companies that build for AI agents, measured on their own sites — re-measured daily; dated change ledger; CSV and JSON at /directory.csv and /directory.json
- https://crawlcheck.io/proof - How every report is fingerprinted and anchored to Bitcoin
- https://crawlcheck.io/policy - What CrawlCheck requests, what it stores, and how to opt out
- https://crawlcheck.io/blog/fifty-four-rules-that-do-not-apply-to-bingbot - A crawler obeys only the User-agent group that matches it best and ignores every other group, including the wildcard. Naming an agent therefore replaces your general rules for it rather than adding to them. We found it hand-verified on three of twenty-one audit tools we measured, including the largest. We also found CrawlCheck itself firing the same finding on user-agents that do not exist.
- https://crawlcheck.io/blog/a-name-google-says-it-never-sends - We read Cloudflare's own verification verdict on every crawler that reached sixteen zones over seven days. Five names were never verified once: PerplexityBot, CCBot, Google-Extended, Claude-SearchBot and Perplexity-User. One of those five is not a crawler at all. And our own figures had been publishing 0% verified for Googlebot, which is our defect, not Google's.
- https://crawlcheck.io/blog/three-of-our-own-rows-passed-because-we-measured-nothing - We audited our own scored rows for vacuous truth - a pass earned by the absence of a measurement rather than by a good result. Three rows failed the audit, a headline counter was capped where nobody would notice, and a paid line item did nothing at all. Every one of them made a site or a plan look better than the evidence supported.
- https://crawlcheck.io/blog/the-newsroom-that-published-coordinates-in-china - A LocalBusiness node on denverpost.com declares latitude 39.7392 and longitude 104.9903 - Denver's own centroid with the minus sign missing, which lands 10,668 km away in northern China. The address in the same node is correct. Our coordinate check passed it, and we explain why, what a range test can and cannot catch, and how to check your own.
- https://crawlcheck.io/blog/our-scanner-counted-placeholders-not-photographs - The homepage serves 13 webp photographs. CrawlCheck recorded 14 data URIs and zero modern-format images, because a lazyload placeholder is a perfectly good string and the real URL was never read. What the fix changed, why the controls stayed still, why no grade moved, and the check you can run on your own markup.
- https://crawlcheck.io/blog/our-proofs-said-pending-for-26-days - A reader who ran the check our proof page invites got “pending” on all 26 sealed days, including days whose blocks had confirmed weeks earlier. What the two stages of a timestamp are, why only the second one proves anything, the byte-for-byte check against the reference client, and what stops it drifting back.
- https://crawlcheck.io/blog/the-viewport-was-there-all-along - Minified HTML with unquoted attributes read as having no viewport, no canonical and no meta description. All three were present. What broke, why the obvious fix was also wrong, and the check now watching for it.
- https://crawlcheck.io/blog/our-canonical-was-right-and-the-site-was-still-served-twice - A correct canonical tag did not stop www and the apex both answering 200 with the same pages, so a crawler fetched the whole site twice. What the audit showed, why it survives so long, and the check now added to CrawlCheck.
- https://crawlcheck.io/guides/is-cloudflare-blocking-ai-crawlers-on-your-site - Cloudflare's AI crawler defaults change on 15 September 2026, including a rule that blocks multi-purpose crawlers such as Googlebot and Applebot when Training crawlers are blocked. How to check what your zone is actually doing, and the failure modes that look like a block but are not.
- https://crawlcheck.io/guides/how-to-check-if-ai-crawlers-can-read-your-site - A step-by-step check of whether GPTBot, ClaudeBot and PerplexityBot can actually read your site: content type, cache state, crawler-versus-browser diff, a nonexistent-path control, and text ratio. With the failure rates read live from the public dataset.
- https://crawlcheck.io/guides/ai-visibility-tools-what-each-one-measures - A comparison of AI visibility and GEO tools by what they measure rather than what they cost: prompt monitors, SEO-suite modules, free graders, machine-layer scanners and log verification. Includes what happened when 33 of these vendors were scanned with a supply-side scanner.
- https://crawlcheck.io/guides/how-to-rank-in-chatgpt - What ranking in ChatGPT actually means, why there is no position to hold, and the four verifiable conditions — reachable, readable, quotable, resolvable — that decide whether an answer engine can use your site. With failure rates read live from the public dataset.
- https://crawlcheck.io/guides/do-ai-crawlers-respect-robots-txt - Whether GPTBot, ClaudeBot and PerplexityBot obey robots.txt, why obedience is the wrong thing to worry about, and the policy conflict that fires when robots.txt invites a crawler the edge then refuses.
- https://crawlcheck.io/guides/does-llms-txt-actually-work - Whether publishing an llms.txt has any effect: adoption figures, what the file can and cannot do, why the vendors selling the advice mostly have not taken it, and the log test that answers the question for your site specifically.
- https://crawlcheck.io/guides/do-ai-crawlers-render-javascript - Whether answer-engine crawlers execute JavaScript, why the safe assumption is that they do not, and a one-command test for whether your content survives in the raw HTML response.
- https://crawlcheck.io/guides/ai-crawler-list-what-each-one-does - A reference list of AI crawlers including GPTBot, ClaudeBot, PerplexityBot, Applebot and Google-Extended: what each fetches for, whether the operator publishes verifiable IP ranges, and measured forgery rates per identity.
- https://crawlcheck.io/guides/geo-aeo-llm-seo-what-the-terms-mean - Definitions of generative engine optimization, answer engine optimization and LLM SEO, how they differ from each other and from SEO, and which activities under those labels produce verifiable measurements.
- https://crawlcheck.io/guides/seo-audit-tools-what-each-one-actually-fetches - A comparison of SEO audit tool categories by what each one requests: desktop crawlers, platform site audits, Lighthouse and PageSpeed, and machine-layer scanners. Includes a measured 225% variance between two runs of the same page.
- https://crawlcheck.io/guides/how-to-write-an-llms-txt-and-verify-it - What goes in an llms.txt, how to structure it, whether generators are worth using, and the checks that confirm the file a crawler receives is the file you published.
- https://crawlcheck.io/guides/free-ai-visibility-checkers-what-they-measure - What free AI visibility checkers and AEO graders actually test, why high scores are becoming meaningless, and the checks a free tool has to run to tell you anything useful.
- https://crawlcheck.io/guides/what-is-entity-seo - What entity SEO means in practice, the structured facts that make a business resolvable, the collision and identity errors that no validator reports, and how to check your own.
- https://crawlcheck.io/guides/how-to-measure-ai-visibility - A method for measuring AI visibility: verifying that crawlers can fetch and parse your site, verifying which crawlers actually arrive using published IP ranges, and recording answer-engine presence in a way that can be checked.
- https://crawlcheck.io/guides/how-to-block-ai-crawlers - How to block AI crawlers at both the robots.txt and network layers, why the three crawler categories should be decided separately, and what each block actually costs in citation eligibility.
- https://crawlcheck.io/blog/ninety-three-percent-of-our-arrivals-sent-no-referrer - We measured arrivals on five sites we operate with our own first-party beacon. 92.9% arrived with no referrer. That number decides what any AI-attribution dashboard can honestly say.
- https://crawlcheck.io/blog/the-web-server-that-gained-and-lost-43-million-sites - One survey ranks OpenResty the fourth most popular web server on earth. Another leaves it off the board entirely. Segmented by traffic, 99.81% of the detections sit outside the top million sites.
- https://crawlcheck.io/blog/we-clicked-a-real-google-result-to-test-the-goto-redirect - Google confirmed google.com/goto on 26 Aug, but who sees it depends on vantage. The referrer can never show the hop, so grepping logs for /goto proves nothing. Watch the Search Console to GA4 ratio instead.
- https://crawlcheck.io/blog/one-lighthouse-run-cannot-support-a-finding-225-percent-variance - Two runs, same unchanged URL, minutes apart: TBT 2,643 ms then 812 ms, FCP 2,865 ms then 4,849 ms. What that means for anyone acting on a single PageSpeed score.
- https://crawlcheck.io/blog/our-audit-labelled-one-page-and-measured-another - A single hard-coded address meant every page-level audit returned homepage data under the correct page's name. How we found it, and why the label agreeing with itself is not verification.
- https://crawlcheck.io/blog/ai-text-watermarks-only-work-if-the-provider-opted-in - What SynthID-Text actually does, what the research says about removing it, and why 'Google will detect your AI content' does not follow.
- https://crawlcheck.io/blog/a-write-that-returned-success-and-corrupted-every-newline - A backup write reported success and silently converted every line ending. The archive ran, the hashes did not match, nothing flagged it. Verify a snapshot by reading it back.
- https://crawlcheck.io/blog/a-783-word-parking-page-defeats-every-thin-content-check - A domain parking page with 783 words passed every thin-content check we ran. Word count is not content. What separates a page that says something from one that fills space.
- https://crawlcheck.io/blog/our-analytics-was-adding-650ms-to-every-request - Our own analytics call added 650 milliseconds to every request, and a code comment kept it there for weeks. Two routes on opposite sides of the code found it in one request. Measure, then blame.
- https://crawlcheck.io/blog/a-cache-rule-does-not-evict-what-is-already-cached - Cache rules are not retroactive. The object was stored before the rule existed, so the rule would have taken effect in 2027.
- https://crawlcheck.io/blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway - Two LLM pipelines discarded every JSON-LD node we publish and still described the business correctly. What they read instead, and where entity identity really lives on a page.
- https://crawlcheck.io/blog/ai-visibility-scores-measure-an-api-not-chatgpt - Three substitutions sit between an API answer and a person's screen: the surface, the sample and the moment. A number that cannot be wrong is not a measurement.
- https://crawlcheck.io/blog/we-almost-published-that-our-host-was-blocking-claude - Hundreds of ClaudeBot 403s in the logs, and we nearly published that our host blocked Anthropic. Grouped by IP, most came from one address wearing seven crawler names.
- https://crawlcheck.io/blog/we-scanned-the-tools-that-audit-your-site - We ran 33 AI-visibility and SEO audit tools through CrawlCheck itself. Eleven publish an llms.txt, none publish an entity map, and several block the crawlers they grade you on.
- https://crawlcheck.io/blog/the-site-that-scored-15-because-it-refused-us - A site scored 15 out of 100 because its firewall refused CrawlCheck, not because the site was bad. A refusal must be reported as a refusal, never scored as a defect.
- https://crawlcheck.io/blog/a-header-we-built-to-be-ignored - The design rule behind our self-identification header, and why letting it change a verdict would have handed every forger a bypass.
- https://crawlcheck.io/blog/the-grade-that-could-not-explain-itself - An AI read one of our reports and confidently explained a grade cap we had not applied. The report had left the reason out. Now the data carries the explanation, not the reader.
- https://crawlcheck.io/blog/we-counted-our-own-users-as-forgers - The tool that measures forged crawler identities was counting its own users' probes as forgeries. Here is how we found it, what the number was, and what it is now.
- https://crawlcheck.io/blog/the-crawlers-nobody-can-check - Some crawler operators publish the IP ranges their bots use; some publish nothing. 251 of 950 named-crawler requests could be neither confirmed nor accused. Unverifiable is not forgery.
- https://crawlcheck.io/blog/the-ai-crawler-that-wanted-your-env-file - Requests wearing AI-crawler names were probing for credential files, not pages. Telling a real crawler from a scanner using the operator's IP ranges, and why a 403 here is correct.
- https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged - Every request claiming to be a known AI crawler, checked against the operator's published IP ranges. A third were not who they said. The per-crawler forged rate, on live traffic.
- https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it - Most measured sites carry a crawler-visible defect today; nearly all carried one at some point on record. A one-time audit sees a day. The record sees the other days, and the live figures are on the page.
- https://crawlcheck.io/blog/what-a-score-is-measuring-when-it-scores-every-site-the-same - A service business loses points for no checkout; a one-page site for no pagination. What a score measures when it scores every site the same, and the rule a fair one follows.
- https://crawlcheck.io/blog/the-meta-tag-a-million-sites-have-and-the-top-ten-thousand-do-not - A tag for a browser that no longer exists is spreading across the web while the largest sites strip it out. The gap between those two lines is the whole story.
- https://crawlcheck.io/blog/the-instructions-on-your-domain-you-did-not-write - Two unrelated storefronts served agents.md files 79% identical raw and 100% identical with brand names stripped. A file instructing AI agents on your behalf, written by a template.
- https://crawlcheck.io/blog/eight-crawlers-eight-companies-one-decision - Eight named AI crawlers from eight companies, grouped by purpose: training, answering, search. Blocking one is a policy decision, and the robots.txt line differs for each.
- https://crawlcheck.io/blog/the-two-facts-almost-nobody-publishes - Across every site scanned here, only a minority publish an llms.txt and far fewer an entity map. The measured adoption rates against BuiltWith's web-wide counts, updated live.
- https://crawlcheck.io/blog/what-a-crawler-actually-receives - We measured the bytes an answer engine gets from real local business sites. On most, under 10% is text; the rest is markup and scripts the crawler pays for on every page.
- https://crawlcheck.io/blog/the-purge-returned-200-and-evicted-nothing - Four successful-looking cache purges cleared zero objects. Here is the two-request test that catches it, and the script that runs it.
- https://crawlcheck.io/blog/a-file-that-answered-200-and-could-not-be-read - A robots.txt that answers HTTP 200 with a bot-challenge page is read as allow-everything by most crawlers. Why every validator missed it, and the two-request test that catches it.
- https://crawlcheck.io/blog/which-ai-crawlers-actually-visit-a-small-business-site - Not a probe: real, unasked-for visits to sites we operate, each checked against the operator's published IP ranges. Which AI crawlers turned up, how often, and which were forged.
- https://crawlcheck.io/blog/cache-your-misses-a-lookup-that-only-caches-on-success - A lookup that only caches successes re-runs the expensive call on every miss, forever. Measured on a site with no field data: the same failing request on every scan. Cache the miss.
- https://crawlcheck.io/blog/two-nodes-with-the-same-id-are-one-node-parsed-twice - Two schema nodes sharing an @id are one record read twice, not two businesses sharing a phone. How CrawlCheck published that false accusation, and the rule that stops it recurring.
- https://crawlcheck.io/blog/a-competitor-graded-us-100-percent-and-still-sold-us-the-fix - A competitor's scanner gave crawlcheck.io a perfect score, then recommended a paid fix. What a grade means when the seller also sells the cure, and how to read one.
- https://crawlcheck.io/blog/one-character-of-undeliverable-email - Every published contact email on a live business site bounced because of one wrong character in the domain. How long it went unnoticed, and the check that catches it in a second.
- https://crawlcheck.io/blog/an-llms-txt-that-answered-200-with-a-login-wall - A large community site answered /llms.txt with HTTP 200 and 8,401 bytes of HTML. One part of CrawlCheck read it as absent, another as present. The self-audit caught the disagreement, and the fix changes what a 200 is allowed to mean.
- https://crawlcheck.io/blog/the-scanner-that-cannot-fetch-its-own-host - A site's founder Person node listed crawlcheck.io/author as a sameAs profile. The page is live from any client on earth, and our corroboration check reported it dead, because CrawlCheck cannot fetch pages on its own host. Unverifiable from this vantage is not dead.
- https://crawlcheck.io/blog/nan-inside-a-url-slug-a-placeholder-check-that-accused-a-newspaper - A self-audit invariant hunts stored records for placeholders that have leaked into output: NaN, undefined, [object Object]. It fired on a national newspaper's record. The three NaNs were the letters n-a-N inside a live-blog URL slug. A substring is not a token.
- https://crawlcheck.io/blog/e-e-a-t-is-not-on-the-wire-what-a-scanner-can-measure-instead - We added fourteen rows that measure the machine-readable statements a site makes about who is behind it: a named person, what qualifies them, whether anyone corroborates them, dated and authored articles, an organisation that names its founder. None of them is expertise. All of them are checkable.
- https://crawlcheck.io/blog/accessibility-structure-nine-things-a-screen-reader-needs-before-contrast-matters - We added an accessibility section that reads only what the server delivers: a language, a title, names on images, inputs, buttons and frames, one main landmark, a way past the navigation, unique ids. Two of the sites we operate had no main landmark at all. Two false positives were caught before it shipped.
- https://crawlcheck.io/blog/we-read-every-scored-row-against-seven-sites-here-is-what-we-unscored - We pulled the pass/fail for every scored row on seven real reports into one matrix and read every row that looked wrong. Seven rows accused sites of things that are not defects; four were counting a defect twice or measuring cosmetics. What changed, what stayed, and how a score version is bumped honestly.
- https://crawlcheck.io/guides/how-to-read-a-crawlcheck-report - Every CrawlCheck report opens on an Overview and keeps five more views behind tabs: the fix list, every section scored, what changed since the last scan, the evidence transcript, and how to read the scoring. This guide walks through each, explains how the grade relates to the three stages, and shows where to dispute a finding on its bytes.
- https://crawlcheck.io/guides/how-to-check-if-a-gptbot-request-in-your-logs-is-real - Requests wearing AI crawler names are routinely forged; on one site every GPTBot 403 in a week came from a single address wearing seven crawler identities. This guide shows how to verify crawler claims against the IP ranges each operator publishes, using the free verifier or the API, and why the answer has three outcomes rather than two.
- https://crawlcheck.io/blog/state-of-ai-visibility-september-2026 - Seven findings from the CrawlCheck dataset as of 2 September 2026, every one sourced to the open data page or a published corpus threshold: which defects are most common, how often they cap a grade, why AI opt-outs move in lockstep, how few homepages are written to be quoted, and how much crawler traffic is not who it claims to be.
- https://crawlcheck.io/blog/a-tree-service-lead-arrived-with-utm-source-chatgpt-com - On 2 September 2026 a quote form on treeservicedenverllc.com recorded a submission whose landing URL carried utm_source=chatgpt.com. This post explains where that tag comes from, why it is the only attribution that survives the trip, what the site's record looked like in the week before, and why a single lead is recorded as one data point rather than a claim.
- https://crawlcheck.io/guides/youtube-seo-case-study-81-videos-rewritten-measured-before-and-after - A worked example of the YouTube check on a real channel: what the tally found, what was changed on 1 September and 5 September 2026, the windows the result will be read across, and the measurement design that will be applied on 3 October. It names what the case can and cannot show.
- https://crawlcheck.io/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them - Local service sites declare where they work in schema (areaServed) and in their pages. CrawlCheck reads both and reports the gap. On four sites we operate: 23 declared / 21 with a page, then 19 / 7, 19 / 1 and 20 / 1. Why the over-promising direction is a real defect a schema validator will not catch, and how the check itself was tightened to measure it honestly.
- https://crawlcheck.io/guides/xml-sitemap-audit - Audit an XML sitemap from the response bytes to every listed URL: the impossible-path control that tells a file from a firewall, HTML fallbacks, redirected and broken children, canonical drift, ghost and orphan pages, and honest lastmod values. With the failure rate for each layer read live from the public dataset.
- https://crawlcheck.io/blog/googles-maps-provider-list-has-no-yelp-in-it - We extracted all 793 entries of Google's recovered DataSourceProvider enumeration and searched it for the directories the local-SEO industry syndicates to. Yelp, Foursquare, Yext, Bing, Apple, Facebook, BBB and TripAdvisor: zero occurrences. What is on the list instead, how sources are reconciled when they disagree, and what that does to the citation-count pitch.
- https://crawlcheck.io/blog/our-crawl-waste-check-named-a-parameter-that-does-not-exist - How a one-line decoding gap put fictional parameter names into a week of crawl-waste records, why the correct counts hid it, what the fix changed on a rescanned site, and what a read of 111 records found once the names were right: 46% of sites carry at least one crawl-waste signal, 18% carry two.
- https://crawlcheck.io/blog/the-parent-declared-the-chain-none-of-the-locations-declared-it-back - A brand declared from the parent page only, six locations that never said so, and the check added to every scan that reads which business entity a page is about and how every other one on it is related: member of chain, subsidiary, department, or undeclared. Fixed on five of six, with the sixth named as still open.
- https://crawlcheck.io/guides/http-status-codes-what-a-crawler-actually-gets - How to read HTTP status codes the way a crawler experiences them: what each class actually tells a crawler, five ways a 200 is not the page, the impossible-path control that separates a firewall from a website, per-client answers, the special rules for robots.txt, redirects read by final hop and host, and cached failures. Each failure class carries its live share of scans from the public dataset.
- https://crawlcheck.io/guides/canonical-url-audit-one-page-one-address - How to audit canonical URLs from the responses rather than the tag: host duplicates where apex and www both serve the page, aliases where every path redirects elsewhere, off-host homepage redirects, per-page canonicals on a depth crawl, sitemap agreement, and the parameters that manufacture duplicates. Each failure class carries its live share of scans from the public dataset.
- https://crawlcheck.io/guides/meta-robots-and-x-robots-tag-the-three-layers-of-crawler-permission - How to audit crawler permissions across the three layers a crawler actually reads: robots.txt for fetching, meta robots and X-Robots-Tag for indexing and snippets, and the Content-Signal vocabulary for AI use. The group rule that silently drops default rules for named agents, the noindex no crawler can reach, header-versus-tag precedence, and the signal conflicts, each with its live share of scans.
- https://crawlcheck.io/blog/the-paywall-that-said-your-site-was-clean - Emptying an array hid the fix list everywhere at once — and made the page and the CSV assert that a site with findings had none. What an empty collection means when it can mean two things.
- https://crawlcheck.io/blog/the-control-panel-that-answered-200-with-nothing-in-it - A function called a block with an argument it never received. The route caught the error, replaced the panel with one sentence, and kept returning 200 — so every check we had passed.
- https://crawlcheck.io/blog/our-llms-txt-is-linked-from-every-page-crawlers-fetch-it-like-a-blog-post - Sixteen AI and search crawlers made 12,831 requests to crawlcheck.io in two weeks. Eight were for llms.txt, linked from every page. Per URL, that is an ordinary page's rate, not a front door's.
- https://crawlcheck.io/blog/our-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week - A read of one week of crawlcheck.io's own edge logs, path by path and user-agent by user-agent: 9,485 requests for the pricing page, almost all from Amazonbot, Ahrefs, ClaudeBot, SemrushBot, GPTBot and PerplexityBot; every blog post fetched about fifty times by sitemap walkers; and a scan endpoint used by a handful of people. Why the headline number is correct, what it is not, and the two figures the homepage should show instead.
- https://crawlcheck.io/blog/the-owner-could-not-read-his-own-scans - The scan API recognised the owner but gated its reply like a stranger's. How the authorization bug hid itself, and the one-rule fix.
- https://crawlcheck.io/blog/we-checked-2260-links-on-our-own-site-four-went-nowhere - We audited every internal link on crawlcheck.io: 924 sitemap pages returned 200, six 400s were correct, and four links went nowhere. Causes and fixes.
- https://crawlcheck.io/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler - What a 404, 403, 429 or 503 on robots.txt means to Google and other crawlers under RFC 9309: when crawling is allowed, paused or blocked, and what to fix.
- https://crawlcheck.io/guides/how-to-audit-internal-links-and-dead-anchors - A 7-step internal link audit: inventory from your sitemap, check every link, tell broken links from correct errors, and find dead in-page anchor links.
- https://crawlcheck.io/blog/googles-maps-stack-names-two-us-business-data-suppliers - Google's Maps provider enum names US business-data suppliers including Data Axle (as INFOUSA and INFOGROUP) and Acxiom. Data Axle is the one an owner can check free from a browser. Nine Denver businesses: one listed, eight absent.
- https://crawlcheck.io/blog/google-places-api-ranking-is-not-google-maps-ranking - Same local query, four cities, one day: Maps’ #1 was the Places API’s #5 and #7, and two businesses Maps ranks were missing from the API.
- https://crawlcheck.io/blog/we-failed-our-own-contrast-check-hover-states-were-worse - A 1.1:1 line in our own call-to-action box led to a full audit, and to 65 more failures in hover, focus and script states that a static check misses.
- https://crawlcheck.io/guides/how-to-test-color-contrast-in-hover-and-focus-states - A six-step method to test WCAG contrast on hover, keyboard focus and script-rendered states, with the four usual causes and how to fix each one.
- https://crawlcheck.io/guides/how-to-check-who-else-uses-your-brand-name-online - Find every site using your brand name, prove who registered first with RDAP, publish a factual notice, and help AI answer engines tell you apart.
- https://crawlcheck.io/blog/release-notes-29-september-2026-finding-states-sitemap-index-trust-center - 29 September 2026: every finding now carries a lifecycle state and a fingerprint, the sitemap is an index of five, and a trust center went up.
- https://crawlcheck.io/guides/hsts-audit-missing-header-preload-ineligible-and-removed-entries - Three HSTS states a crawler can see from the homepage response: no header, a preload token the header cannot support, and a list entry that no longer qualifies.
- https://crawlcheck.io/guides/content-security-policy-rollout-report-only-to-enforced - A CSP is the one header that can break a page at HTTP 200. The safe path is report-only first, and two reporting mistakes make that step silently useless.
- https://crawlcheck.io/guides/how-to-move-dmarc-from-none-to-quarantine-without-losing-mail - Moving DMARC from none to quarantine is safe when every sender aligns DKIM and the report address exists. The order of those steps is the whole method.
- https://crawlcheck.io/guides/json-ld-id-and-iri-errors-what-breaks-and-how-to-fix-them - An @id is an IRI, not a label. Six ways it goes wrong on real pages, what each does to the graph a consumer builds, and the one-pass fix.
- https://crawlcheck.io/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge - A CDN caches small policy files hardest. Two fetches per file — bare and cache-busted — reveal when the edge and origin disagree, and in which of four ways.
- https://crawlcheck.io/guides/nap-consistency-audit-phone-address-and-coordinates - Five local-entity findings, from a missing coordinate pair to a directory printing the wrong name, and what each does to how an engine resolves the business.
- https://crawlcheck.io/guides/crawler-delivery-comparison-refused-redirected-or-served-less - Four outcomes when the same URL is fetched as named crawlers and as a browser control, and why the control is a browser.
- https://crawlcheck.io/guides/staging-site-leak-how-a-staging-host-ends-up-in-your-live-pages - A search-and-replace that missed leaves the staging host in the live page. A crawler follows it to an indexable second copy. How to find and close it.
- https://crawlcheck.io/blog/five-days-of-our-alerts-went-to-an-address-with-no-mailbox - A hard bounce put our operator address on a suppression list. For five days every alert said sent and nobody received it. Every check measured the wrong side.
- https://crawlcheck.io/blog/our-security-headers-skipped-every-signed-in-session - One missing await meant our new CSP and CORS headers were applied to anonymous visitors and skipped for everyone signed in. Verified signed out only.
- https://crawlcheck.io/blog/crawlcheck-in-german-french-spanish-and-portuguese - Four languages, zero untranslated text on a public check. What is translated, what stays English by design, and how coverage is measured.
- https://crawlcheck.io/blog/the-page-our-extension-read-as-36-percent-javascript-was-zero - Two defects, neither visible in the number: entities spelled two ways, and an admin bar only the owner sees. The fix, and the controls that keep it honest.
- https://crawlcheck.io/blog/eight-contradictions-in-our-own-copy-rechecked - Eight claimed self-contradictions, rechecked against the live site: three were two truths read as one, four fixed, one still wrong today.
- https://crawlcheck.io/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules - A crawler obeys only the robots.txt group that names it and ignores User-agent: *. A named group with Allow: / silently voids every Disallow rule you set.
- https://crawlcheck.io/guides/localbusiness-schema-for-service-area-businesses - Google requires name and address. The failures are in geo, telephone, @id and areaServed. How to write a LocalBusiness node and verify each property.
- https://crawlcheck.io/guides/text-to-html-ratio-when-a-page-is-mostly-code - Text-to-HTML ratio is no ranking factor, but a large page with under 5% text sends crawlers mostly CSS, scripts and state. How to measure and fix it.
- https://crawlcheck.io/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers - A user-agent string is a claim. Verify crawlers by reverse DNS with a forward confirm, or by the operator's IP list. The method and file for each crawler.
- https://crawlcheck.io/guides/content-signals-in-robots-txt-search-ai-input-ai-train - Content-Signal states preferences for search, ai-input and ai-train. It blocks nothing. Refusing ai-train costs no reach; refusing ai-input or search does.
- https://crawlcheck.io/guides/web-bot-auth-how-signed-crawler-requests-work - Web Bot Auth signs crawler requests with Ed25519 keys published at a well-known directory. The headers, how a site verifies one, and what a signature proves.
- https://crawlcheck.io/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation - Serve Markdown when an agent sends Accept: text/markdown: same URL, Vary: Accept, text/markdown content type. Headers to return and the caching trap.
- https://crawlcheck.io/guides/indexnow-what-it-does-who-reads-it-and-how-to-check-it-worked - IndexNow tells engines a URL changed. One submission reaches Bing, Yandex, Seznam, Naver, Yep and others; Google does not take part. Setup and status codes.
- https://crawlcheck.io/guides/security-txt-rfc-9116-fields-expiry-and-where-to-serve-it - RFC 9116 security.txt needs Contact and Expires, served at /.well-known/security.txt over HTTPS. The fields, the expiry rule, and how to check your file.
- https://crawlcheck.io/guides/core-web-vitals-field-data-for-low-traffic-sites-without-crux - Most small sites get no CrUX field data, and lab scores are not field data. Collect your own LCP, INP and CLS, and do not trust a p75 under 100 samples.
- https://crawlcheck.io/guides/hreflang-audit-return-links-x-default-and-the-common-errors - Hreflang only counts when every version links back. Return links, self-reference, x-default, language codes and canonicals: the hreflang rules and errors.
- https://crawlcheck.io/blog/release-notes-3-october-2026-resolve-protocol-census-rulebook - A signed Resolve answer for AI agents, a census of the top 1,000 sites read twice, 75 published rules, signed receipts, a dispute ledger, four observers and an SDK.
- https://crawlcheck.io/guides/best-ai-web-crawler-on-a-budget - Firecrawl, ScrapingBee, Apify, Browse AI and Crawl4AI: what $19 a month buys in JavaScript pages, the hidden proxy costs, and a free AI crawler check.
- https://crawlcheck.io/guides/best-ai-visibility-tools-compared - Ten AI visibility tools compared on published prices and engines covered, including Profound, Otterly, Ahrefs, Semrush and CrawlCheck.
- https://crawlcheck.io/guides/cloudflare-ai-crawl-control-vs-crawlcheck - Cloudflare AI Crawl Control blocks or allows AI crawlers; CrawlCheck measures what each is actually served. Features, limits and when to use both.
- https://crawlcheck.io/guides/how-to-test-robots-txt-for-gptbot-and-claudebot - How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters more than the file.
- https://crawlcheck.io/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit - Screaming Frog, Sitebulb and CrawlCheck compared for AI crawler audits: pages crawled, crawler identities tested, robots.txt handling and price.
- https://crawlcheck.io/guides/best-llms-txt-generators-and-validators-compared - llms.txt generators and validators compared: Firecrawl, Yoast, Mintlify and CrawlCheck, and how to check the file reaches AI crawlers.
- https://crawlcheck.io/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared - How to verify GPTBot, ClaudeBot and PerplexityBot requests: published IP lists, CrawlCheck's free log check and Cloudflare verified bots compared.
- https://crawlcheck.io/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck - Cloudflare's Is It Agent Ready? and CrawlCheck compared: agent standards checked, declared vs working capabilities, AI crawler access and price.
- https://crawlcheck.io/guides/schema-validators-compared-rich-results-test-vs-schema-markup-validator - Google's Rich Results Test, the Schema Markup Validator and CrawlCheck compared: what each checks, valid markup that still fails, and which to use.
- https://crawlcheck.io/guides/pagespeed-insights-and-lighthouse-vs-an-ai-crawler-audit - PageSpeed Insights and Lighthouse measure speed for people, not AI crawler access. What a perfect score misses, with live figures from CrawlCheck scans.
- https://crawlcheck.io/guides/best-web-scraping-apis-compared - Web scraping APIs compared on real cost per page: ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl and Apify, plus where CrawlCheck fits.
- https://crawlcheck.io/guides/open-source-crawlers-for-llms-compared - Open-source crawlers for LLMs compared: Crawl4AI, Firecrawl, Scrapy and Jina Reader on licence, JavaScript rendering, output and limits.
- https://crawlcheck.io/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors - UptimeRobot, Visualping and CrawlCheck Watch compared: what each monitors, how often, price, and the AI crawler failures uptime checks miss.
- https://crawlcheck.io/blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price - We read the top 1,000 sites as four AI crawlers. Three answered with 402 Payment Required and none quoted a price: a refusal, a toll, a contradiction.
- https://crawlcheck.io/blog/state-of-ai-visibility-october-2026-139788-homepages - 139,788 homepages measured 3-6 October 2026: 6.3% block an AI crawler, 15.4% publish llms.txt, and the median homepage is 3.6% text.
- https://crawlcheck.io/blog/our-robots-txt-check-misread-a-full-block-as-a-broken-rule - Our robots.txt check reported Disallow: / as shadowing a site's own rules. 83% of AI-blocking sites got the false finding. What changed.
- https://crawlcheck.io/blog/only-six-percent-of-websites-block-an-ai-crawler - 6.3% of 139,788 sites block an AI crawler. GPTBot leads at 3.7%; search agents are blocked far less. The full per-crawler table.
- https://crawlcheck.io/blog/eight-weeks-of-agent-conduct-who-was-really-who - 24,627 verified crawler requests and 787 forged ones on crawlcheck.io, 13 August to 7 October 2026. Googlebot is forged 1.3% of the time, GPTBot 2.4%, Claude-User 40%, Perplexity-User 86%. Google-Extended, Claude-SearchBot and anthropic-ai were claimed 90 times and verified never. 47,700 visits came from names with no published ranges.
- https://crawlcheck.io/blog/mcp-registry-census-1851-servers-what-they-answer - All 25,983 remote endpoints in the official MCP Registry, read on 7 October 2026: 15,575 returned tool lists (288,806 tools), 6,422 asked for credentials, 712 of those without saying where. Eleven servers carry tool text aimed at the model: four tell it to keep something from the user.
- https://crawlcheck.io/guides/what-the-agentic-web-means-for-a-site-owner - Answer engines read and cite. Agents open pages, fill in forms and call tools for a person. Four layers decide whether they can: permission, delivery, identity and tools, each measurable from outside.
- https://crawlcheck.io/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt - ChatGPT-User 526 verified and 72 forged, Claude-User 79 and 53, Perplexity-User 8 and 50. Only Claude-User fetched robots.txt. Google-Agent never arrived. 2.4% of verified agent traffic.
- https://crawlcheck.io/guides/how-to-check-an-mcp-server-before-an-agent-connects - Handshake, list the tools, read every description as an instruction to a model, check how it asks for credentials, and pin what you read. Eleven registry servers failed step three.
- https://crawlcheck.io/guides/can-an-ai-agent-use-your-contact-form - Open it signed out, clear nothing, find the form in the page source, check every field has a label, and give the agent a path that is not a puzzle. Pass/fail table included.
- https://crawlcheck.io/blog/who-signs-agent-requests-web-bot-auth-october-2026 - OpenAI signs ChatGPT’s cloud browser as chatgpt.com. Google is experimenting with agent.bot.goog for Google-Agent. Cloudflare lists Browserbase, Anchor Browser and Block’s Goose. Anthropic and Perplexity document ranges, not signatures.
- https://crawlcheck.io/guides/ai-website-audit-tools-compared-semrush-ahrefs-screaming-frog-sitebulb-crawlcheck - Semrush, Ahrefs, Screaming Frog and Sitebulb add AI to a classic SEO audit. CrawlCheck audits the site as AI crawlers and agents see it. Prices and the question each answers, from each vendor’s own pages.
- https://crawlcheck.io/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites - 980 sites in the Tranco top 200,000, measured 9 October 2026: llms.txt on 98% of Shopify and 96% of Wix sites but 1% of Squarespace; Wix serves AI crawlers a separate, lighter page with the same text; refusals come from site settings, not platforms.
- https://crawlcheck.io/blog/agent-conduct-october-2026 - 14 agents assessed from 34,676 verified requests to crawlcheck.io since 12 August 2026: 9 of the 12 crawlers that robots.txt governs read it, 3 never fetched it under their own name, 4 fetches of disallowed paths were counted before per-fetch records existed and are not judged, and 2.4% of requests using a tracked agent's name were forged.
- https://crawlcheck.io/blog/mcp-reliability-october-2026 - 6,000 remote MCP endpoints from 4,985 publishers, 8–15 October 2026: 98.7% of 39,484 handshakes answered, median time to answer 1.2 s, 53 endpoints went down at least once, and 19.7% of endpoints that publish a tool list changed it during the week.
- https://crawlcheck.io/blog/share-of-preflight-october-2026 - 7 connected sites reported 9,883 agent requests in October 2026 to date; 5,643 were verified against the operators' published ranges and none carried a CrawlCheck receipt. The 68 receipts sites accepted all came from test agents.
- https://crawlcheck.io/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026 - Census of the top 1,000 sites, 15 October 2026: of 418 sites that answered a browser and served robots.txt, 79 refused at least one of GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot at the server, and at 57 of them robots.txt allows that crawler.
- https://crawlcheck.io/guides/how-an-ai-agent-verifies-a-website-before-it-acts - Six questions an agent should answer before it reads, cites, connects or pays, the evidence each needs, and how to cite the answer so it can be checked.
- https://crawlcheck.io/blog/what-four-ai-models-told-customers-about-our-three-businesses - Gemini with Search got every phone number right but gave two service-area firms streets they lack. gpt-oss invented numbers. Llama and Mistral declined.
- https://crawlcheck.io/blog/mcp-servers-that-ask-for-a-login-what-they-request-by-default - Of 1,366 login-gated MCP servers, 597 had a finding: 244 default to every scope including write, 152 publish no metadata, 39 request admin rights.
- https://crawlcheck.io/blog/do-ai-agents-stop-before-acting-on-a-lookalike-site - Alone, four models stopped at 5 to 12 of 14 traps; with a signed answer, all 14. Each caught every lookalike. Revoked receipts caught them all out.
- https://crawlcheck.io/blog/where-google-and-ai-assistants-get-business-data - Google reads your website, your Business Profile, your customers and its own checks. No engine documents Data Axle as a source today, and its listings site blocks ChatGPT.
- https://crawlcheck.io/blog/release-notes-14-october-2026-doors-signed-weekly-figures-watch-trial - crawlcheck.io on 14 Oct 2026: four audience doors, signed weekly editions and feeds for every live figure, a 30-day Watch trial, and the day's fixes.
- https://crawlcheck.io/blog/sitemap-lastmod-103-pages-one-date-why-engines-ignore-it - 103 pages in our sitemap shared one lastmod though most had been rebuilt. Why Google and Bing ignore dates like that, and how to date each page properly.
- https://crawlcheck.io/blog/a-cron-job-ran-one-minute-late-and-our-directory-froze-for-20-hours - A scheduled job checked the clock instead of the schedule, so our directory index stopped rebuilding for 20 hours without one error. Cause, fix, timeline.
- https://crawlcheck.io/blog/our-llms-txt-failed-our-own-prompt-injection-check - One descriptive sentence in our own llms.txt tripped our prompt-injection check and cost us two points. The sentence and the rule were both fixed.
- https://crawlcheck.io/blog/cloudflare-access-on-one-path-locked-our-customers-out-of-sign-in - Gating our owner login with Cloudflare Access also gated customer sign-in at the same path. 247 pages linked to it for an hour. What we changed.
- https://crawlcheck.io/guides/how-to-get-your-local-business-recommended-by-ai-assistants - A plain checklist for local businesses: what to fix on the website, what to make consistent across listings, and how to check each month whether AI assistants name you.

## Translated editions

- https://crawlcheck.io/de/ · https://crawlcheck.io/fr/ · https://crawlcheck.io/es/ · https://crawlcheck.io/pt/
- https://crawlcheck.io/de/pricing · https://crawlcheck.io/fr/pricing · https://crawlcheck.io/es/pricing · https://crawlcheck.io/pt/pricing
- https://crawlcheck.io/de/how-it-works · https://crawlcheck.io/fr/how-it-works · https://crawlcheck.io/es/how-it-works · https://crawlcheck.io/pt/how-it-works
- https://crawlcheck.io/de/tools · https://crawlcheck.io/fr/tools · https://crawlcheck.io/es/tools · https://crawlcheck.io/pt/tools
- https://crawlcheck.io/de/for-agents · https://crawlcheck.io/fr/for-agents · https://crawlcheck.io/es/for-agents · https://crawlcheck.io/pt/for-agents
- https://crawlcheck.io/de/for-sites · https://crawlcheck.io/fr/for-sites · https://crawlcheck.io/es/for-sites · https://crawlcheck.io/pt/for-sites
- https://crawlcheck.io/de/developers · https://crawlcheck.io/fr/developers · https://crawlcheck.io/es/developers · https://crawlcheck.io/pt/developers
- https://crawlcheck.io/de/learn · https://crawlcheck.io/fr/learn · https://crawlcheck.io/es/learn · https://crawlcheck.io/pt/learn
- https://crawlcheck.io/de/tools/agent-view · https://crawlcheck.io/fr/tools/agent-view · https://crawlcheck.io/es/tools/agent-view · https://crawlcheck.io/pt/tools/agent-view

## Other machine files on this host

- https://crawlcheck.io/robots.txt
- https://crawlcheck.io/sitemap.xml
- https://crawlcheck.io/llms.txt
- https://crawlcheck.io/agents.md
- https://crawlcheck.io/SKILL.md
- https://crawlcheck.io/entitymap.json
- https://crawlcheck.io/.well-known/api-catalog
- https://crawlcheck.io/.well-known/agent-skills/index.json