Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
How to check whether AI crawlers can read your site
Five requests, ten minutes, no account required. What each one catches, and why opening the page in your browser is the one test that cannot find any of it.
Four sites we operate: same city, same kind of business, same person writing the copy. The tree site carries nearly twice the visible text in a third of the bytes — the gap is build weight, not writing. Scan any of these yourself; the numbers move when the sites change.
Nearly every way of checking this is a variation on the same mistake: loading the page in a browser, seeing it work, and concluding it works. A browser is the one client that reliably does not have the problem. It runs JavaScript, it clears challenges, it carries cookies from your last visit, and it is often coming from an address your own firewall already trusts. It is the least representative reader you own.
What follows is what a crawler does instead. Five requests, in order of how often they find something. The failure rate beside each one is measured across the scans in the public dataset here, so you can see how likely each check is to fire before you run it.
The five checks
| # | Check | Passes when | Fires on |
|---|---|---|---|
| 1 | Content type of robots.txt | text/plain | 3.9% of scans |
| 2 | The bare URL, no cache-buster | edge and origin agree | 30.9% of scans |
| 3 | Crawler UA versus browser UA | same status, same size | 3.4% of scans |
| 4 | A path that cannot exist | 404, and unlike the homepage | 1.3% of scans |
| 5 | Share of payload that is text | 20% and up | 13.7% of scans |
1. Read the content type, not the status code
A robots.txt that answers 200 OK with a bot-challenge page is read as allow-everything by most crawlers, because a parser that receives HTML finds no directives, and no directives means no restrictions. That is the exact opposite of what the file said.
curl -sI https://yoursite.com/robots.txt | grep -i content-type
It must say text/plain. If it says text/html, something is answering in place of your file and the 200 next to it means nothing. Run the same check on your sitemap and your llms.txt. In this corpus, 3.9% of scans find robots.txt not answering 200 at all, 1.9% find a challenge page served at 200, and 0.4% find HTML sitting where the file should be. A file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth.
2. Fetch the bare URL first, the cache-buster second
This is the check people get backwards, and it is the single most common defect in the entire dataset. Appending ?cb=123 to a URL asks the origin a fresh question and bypasses the cached object — so the file comes back perfect and you conclude the file is fine. The crawler never appends anything. It requests the bare URL and gets whatever the edge is holding.
STALE_CACHE_SERVED fires on 30.9% of scans here — 321 of 1,038. Roughly one site in three is serving crawlers a cached copy of something that no longer matches what its own origin says. Fetch the bare URL, note the size and the age or cf-cache-status header, then fetch it again with a cache-buster and compare. A difference between those two responses is the difference between what you think you publish and what is actually being read.
3. Ask twice, as two different clients
Send the same request as a browser and as a crawler, and compare status and byte count.
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0" https://yoursite.com/curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "GPTBot" https://yoursite.com/
Different status codes mean an edge rule is deciding who gets in, and it is very rarely a rule anyone remembers writing. On one site measured here, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally, with nothing in robots.txt to explain it. ANSWER_ENGINE_REFUSED fires on 3.4% of scans. Same status but a materially smaller body is a different problem with the same effect: the crawler is being handed a thinner page than the person.
One caveat that matters, and most write-ups skip it. Your request is sending a crawler's name from an address that operator does not publish. An edge that verifies properly should refuse it. So a refusal here is not proof your firewall is blocking OpenAI — it is proof that something at your edge treats that identity differently, which is worth knowing either way. Your server logs are what settle which one it is.
4. Ask for something that cannot exist
Request a random path nobody would ever create, and compare it to the homepage.
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/zzq-does-not-exist-4831
It should 404, and it should not look like your homepage. If a URL that cannot exist answers with the same status and roughly the same size as the page you care about, you are not measuring a website — you are measuring a wall in front of one, and every reading you take through it is meaningless. No real site answers a URL that does not exist the same way it answers its homepage. This control fires on 1.3% of scans, and it exists because a scanner without it will confidently grade a firewall and hand you a number about your site that describes someone else's product.
5. Weigh the payload
Take the bytes on the wire and work out how much of it is visible text a machine could quote. The rest is markup, inline styles and script that a crawler downloads and cannot use.
The heaviest page measured in this corpus delivered 476,540 bytes and 2.3% visible text. It rendered quickly, it read well, and no validator had an opinion about it. PAGE_IS_MOSTLY_CODE fires on 13.7% of scans. Inline CSS is the usual cause and it is the worst kind of weight, because it cannot be cached between pages — a crawler pays for it again on every URL it fetches.
What this does not tell you
All five checks answer one question: can the thing that fetches your site read what came back. None of them tells you whether an assistant will cite you. Those are separate problems, and the order matters, because the second one is unanswerable until the first is clean. Nothing you write reaches an engine that is being handed a challenge page.
And one reading is one reading. A deterministic block reproduces perfectly, so running the same request twice from the same machine is not corroboration — it is the same measurement, taken twice. Check from a second network before you conclude anything, and treat a defect that appears on Tuesday and not on Wednesday as a defect that appears on Tuesday.
Why once is not enough
Across 48 sites measured every day for fifteen days, 68.8% carried a defect on the final day. 93.8% carried one at least once during the window. That is a 25-point gap, and 12 sites look clean now that did not look clean on at least one day in that period. A single audit can only ever produce the first number. The second one requires having watched.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
How do I check if ChatGPT can read my website?
Fetch your robots.txt from outside your own network and read the content type rather than the status code: it must be text/plain. Then request your homepage twice, once with a browser user-agent and once as GPTBot, and compare the status and byte count. A difference in either means an edge rule is deciding who gets served.
Why does my site look fine in a browser but fail these checks?
A browser runs JavaScript, clears bot challenges, carries your existing cookies and often comes from an address your firewall already trusts. A crawler does none of that. The browser is the least representative reader of your own site.
What is the most common problem these checks find?
A stale cached copy served on the bare URL. It fires on 30.9% of scans in this dataset. It hides from most testing because adding a cache-buster to the URL bypasses the cached object and returns the correct file, while a crawler always requests the bare URL.