We ask as each crawler
One address, the same page, requested as each of the 16 named crawler profiles plus a desktop browser, a smartphone Googlebot and plain controls.
See what your website serves to AI crawler request profiles. CrawlCheck compares real responses from its own network, names what went wrong, and shows you what to fix first.
First scan free with your email, then 3 credits ($3) a scan. Nothing installed, nothing changed on your site. 52 sections, read as 16 named crawler request profiles and 3 comparison controls.
Example report · a public site, checked daily www.denverpost.com Score81 / 1008 things to fixShare of the page text each crawler profile receivedGPTBot100%ClaudeBot100%Googlebot100%Bytespider403 AI assistants like ChatGPT can read this site, but find little they can quote from its pages. See the full reportscansto Denver
Where scans come from. Every dot is a city or country a visitor has run a scan from, shown at city level. Open the globe
01 · How it works
We compare what your site serves to search crawlers, training crawlers, user-triggered fetchers and controls from CrawlCheck's network. That includes separate desktop and smartphone Googlebot requests. We also read your robots.txt, machine files and declared agent permissions. The answer is graded by published rules and signed, so anyone can check it offline.
CrawlCheck is the verification layer for the agentic web. It measures what a site actually serves — tested with named crawler request profiles, including desktop and smartphone Googlebot, alongside its machine files, crawl policy, declared rights and capabilities — decides findings by published rules, and signs the answer so an agent can verify it offline before it reads, cites, connects or acts. Site owners get the same measurement as a free A–F audit with every defect named.
One address, the same page, requested as each of the 16 named crawler profiles plus a desktop browser, a smartphone Googlebot and plain controls.
Status, size, content and cache state for each of the 19 replies, side by side, with your robots.txt, machine files and declared agent permissions read alongside.
A grade from A to F across 52 sections, every finding shown in full with its fix steps, and the one thing to fix first at the top.
Paying never changes a reading. The scan, step by step
02 · Independent research
In 37,927 AI agent sessions about 1,056 real businesses, agents did better on every measure when they could read the business’s own site.
Source: Finder, Elovic, Shalev and Yosef, “AX is the New AEO”, arXiv:2609.34951, 29 September 2026. CrawlCheck was not involved in the study.
03 · Four doors
04 · Questions
ChatGPT reaches the web through more than one crawler: OAI-SearchBot finds pages for ChatGPT search, and GPTBot collects content for model training. Allowing GPTBot is not needed to appear in ChatGPT search, and blocking it does not remove you. Many sites block one or both in robots.txt or behind a bot wall without knowing. That is the access we can test, not the only way an AI service may reach your content. The free check fetches your homepage as OAI-SearchBot, GPTBot and fourteen other crawlers from our own network and shows exactly what each one got.
Three usual reasons: the crawler is blocked, the page it gets has no readable text, or there is no clear fact to quote: a name, address, hours and prices that match everywhere. The report names which one applies to you.
Your first scan is free with your email, then 3 credits ($3) a scan; reports already run stay free to open. Every request is a read, the same GET a crawler makes. Nothing is installed, submitted or changed.
It tells every agent that asks who runs the site and what it declares, verified by a fetch. Free, from the registry page.
139,788 homepages measured 3–6 October 2026: 6.3% block an AI crawler, 15.4% publish llms.txt, the median homepage is 3.6% text. Live defect rates inside.
Google's three-domain Googlebot rule accepts any Google Cloud server: 198 of 200 sampled addresses passed. Two tools tested, the fix, and how to reproduce.
422 paired tests: dropping a rating from 4.8 to 4.1 stars pushed a business down an AI ranking every time. Review count and area served came next.