CrawlCheck
The verification layer for the agentic web

Can AI read your site?

CrawlCheck is the verification layer for the agentic web. It measures what a site actually serves — fetched as GPTBot, ClaudeBot, PerplexityBot, Googlebot and eight other crawler identities, plus its machine files, crawl policy, declared rights and capabilities — decides findings by published rules, and signs the answer so an agent can verify it offline before it reads, cites, connects or acts. Site owners get the same measurement as a free A–F audit with every defect named.

Free. No signup, nothing installed, nothing changed on your site.

Where scans come from →
Signed answer · a public site, measured daily · observed yesterdaydenverpost.comreadwarncitewarnconnectnone declaredtransactneeds a human8 of 9 AI-crawler identities were served the page; robots.txt allows 73 of 114 tested agentsCrawl policydeclared · 73 of 114 agents allowedDeliveryobserved · 14 identities, 1 refusedMachine filesobserved · 5 read, bodies hashedFindings10 open · 4 seriousEd25519 · verifies offlineOpen the signed answer →

Last scan finished 48 minutes ago · 88 domains measured today

How it works

We visit your site as twelve crawlers and tell you what each one got.

GPTBot, ClaudeBot, PerplexityBot, Googlebot and eight more, from our own network. Then your robots.txt, your machine files and what you have told agents they may do. The answer is graded by published rules and signed, so anyone can check it offline.

Paying never changes a reading. The scan, step by step · Live figures

Independent research

Being readable beats being talked about.

A September 2026 study ran 37,927 AI agent sessions about 1,056 real businesses, each one a buyer’s question. When the agent could fetch and read the business’s own site, it did better on every measure. That reading is what CrawlCheck checks.

1.9×as often clearly recommended when agents could read the site
78% vs 56%of answers built from the business’s own pages, readable site against the rest
41%more accurate answers when built from the site, same business and question
7–10%of the final answer came from what the model already knew; the rest was read live

Source: Finder, Elovic, Shalev and Yosef, “AX is the New AEO”, arXiv:2609.34951, 29 September 2026. CrawlCheck was not involved in the study.

Questions

What people ask before they check.

QCan ChatGPT read my website?
Its crawler, GPTBot, has to be able to fetch your pages and find the words on them, and many sites block it in robots.txt or behind a bot wall without knowing. That is the access we can test, not the only way an AI service may reach your content. The free check fetches your homepage as GPTBot and eleven other crawlers from our own network and shows exactly what each one got.
QWhy doesn’t my business show up in AI answers?
Three usual reasons: the crawler is blocked, the page it gets has no readable text, or there is no clear fact to quote: a name, address, hours and prices that match everywhere. The report names which one applies to you.
QIs the check free, and does it change anything on my site?
Free, no account. Every request is a read, the same GET a crawler makes. Nothing is installed, submitted or changed.
QWhat does registering my site do?
It tells every agent that asks who runs the site and what it declares, verified by a fetch. Free, from the registry page.