CrawlCheck
Free check for your website

Can AI read your site?

See what your website serves to AI crawler request profiles. CrawlCheck compares real responses from its own network, names what went wrong, and shows you what to fix first.

First scan free with your email, then 3 credits ($3) a scan. Nothing installed, nothing changed on your site. 52 sections, read as 16 named crawler request profiles and 3 comparison controls.

Example report · a public site, checked daily www.denverpost.com C81 / 100Score81 / 1008 things to fixShare of the page text each crawler profile receivedGPTBot100%ClaudeBot100%Googlebot100%Bytespider403 AI assistants like ChatGPT can read this site, but find little they can quote from its pages. See the full report

Where scans come from. Every dot is a city or country a visitor has run a scan from, shown at city level. Open the globe

570,959sites measured
9,160scans run
52sections in every audit
16 + 3crawler profiles and controls
107findings published
READ ASGPTBotClaudeBotOAI-SearchBotClaude-SearchBotPerplexityBotChatGPT-UserClaude-UserPerplexity-UserGooglebotGooglebot SmartphoneBingbotApplebotAmazonbotBytespiderMeta-ExternalAgentCCBot+ controls

01 · How it works

See exactly what each request received.

We compare what your site serves to search crawlers, training crawlers, user-triggered fetchers and controls from CrawlCheck's network. That includes separate desktop and smartphone Googlebot requests. We also read your robots.txt, machine files and declared agent permissions. The answer is graded by published rules and signed, so anyone can check it offline.

The full definition

CrawlCheck is the verification layer for the agentic web. It measures what a site actually serves — tested with named crawler request profiles, including desktop and smartphone Googlebot, alongside its machine files, crawl policy, declared rights and capabilities — decides findings by published rules, and signs the answer so an agent can verify it offline before it reads, cites, connects or acts. Site owners get the same measurement as a free A–F audit with every defect named.

01

We ask as each crawler

One address, the same page, requested as each of the 16 named crawler profiles plus a desktop browser, a smartphone Googlebot and plain controls.

02

We compare what came back

Status, size, content and cache state for each of the 19 replies, side by side, with your robots.txt, machine files and declared agent permissions read alongside.

03

We name what to fix first

A grade from A to F across 52 sections, every finding shown in full with its fix steps, and the one thing to fix first at the top.

Paying never changes a reading. The scan, step by step

02 · Independent research

Being readable beats being talked about.

In 37,927 AI agent sessions about 1,056 real businesses, agents did better on every measure when they could read the business’s own site.

Source: Finder, Elovic, Shalev and Yosef, “AX is the New AEO”, arXiv:2609.34951, 29 September 2026. CrawlCheck was not involved in the study.

1.9×as often clearly recommended
78%of answers built from the business’s own pages, against 56%
41%more accurate answers
7–10%of the answer came from the model’s memory

03 · Four doors

Who it is for.

04 · Questions

What people ask before they check.

Can ChatGPT read my website?

ChatGPT reaches the web through more than one crawler: OAI-SearchBot finds pages for ChatGPT search, and GPTBot collects content for model training. Allowing GPTBot is not needed to appear in ChatGPT search, and blocking it does not remove you. Many sites block one or both in robots.txt or behind a bot wall without knowing. That is the access we can test, not the only way an AI service may reach your content. The free check fetches your homepage as OAI-SearchBot, GPTBot and fourteen other crawlers from our own network and shows exactly what each one got.

Why doesn’t my business show up in AI answers?

Three usual reasons: the crawler is blocked, the page it gets has no readable text, or there is no clear fact to quote: a name, address, hours and prices that match everywhere. The report names which one applies to you.

Is the check free, and does it change anything on my site?

Your first scan is free with your email, then 3 credits ($3) a scan; reports already run stay free to open. Every request is a read, the same GET a crawler makes. Nothing is installed, submitted or changed.

What does registering my site do?

It tells every agent that asks who runs the site and what it declares, verified by a fetch. Free, from the registry page.