Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 0 views

How to check whether AI crawlers and agents can read your site

Five requests, ten minutes, no account required. What each one catches, and why opening the page in your browser is the one test that cannot find any of it.

Nearly every way of checking this is a variation on the same mistake: loading the page in a browser, seeing it work, and concluding it works. A browser is the one client that reliably does not have the problem. It runs JavaScript, it clears challenges, it carries cookies from your last visit, and it is often coming from an address your own firewall already trusts. It is the least representative reader you own.

What follows is what a crawler does instead. Five requests, in order of how often they find something. The failure rate beside each one is measured across the scans in the public dataset here, so you can see how likely each check is to fire before you run it.

The five checks #

1. Read the content type, not the status code #

A robots.txt that answers 200 OK with a bot-challenge page is read as allow-everything by most crawlers, because a parser that receives HTML finds no directives, and no directives means no restrictions. That is the exact opposite of what the file said.

curl -sI https://yoursite.com/robots.txt | grep -i content-type

It must say text/plain. If it says text/html, something is answering in place of your file and the 200 next to it means nothing. Run the same check on your sitemap and your llms.txt. In this corpus, 5.9% of scans find robots.txt not answering 200 at all, 0.6% find a challenge page served at 200, and 0.2% find HTML sitting where the file should be. A file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth.

2. Fetch the bare URL first, the cache-buster second #

This is the check people get backwards, and it is the single most common defect in the entire dataset. Appending ?cb=123 to a URL asks the origin a fresh question and bypasses the cached object — so the file comes back perfect and you conclude the file is fine. The crawler never appends anything. It requests the bare URL and gets whatever the edge is holding.

STALE_CACHE_SERVED fires on 17.4% of scans here. That share of sites is serving crawlers a cached copy of something that no longer matches what its own origin says. Fetch the bare URL, note the size and the age or cf-cache-status header, then fetch it again with a cache-buster and compare. A difference between those two responses is the difference between what you think you publish and what is actually being read.

3. Ask twice, as two different clients #

Send the same request as a browser and as a crawler, and compare status and byte count.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "GPTBot" https://yoursite.com/

Different status codes mean an edge rule is deciding who gets in, and it is very rarely a rule anyone remembers writing. On one site measured here, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally, with nothing in robots.txt to explain it. ANSWER_ENGINE_REFUSED fires on 5.8% of scans. Same status but a materially smaller body is a different problem with the same effect: the crawler is being handed a thinner page than the person.

One caveat that matters, and most write-ups skip it. Your request is sending a crawler's name from an address that operator does not publish. An edge that verifies properly should refuse it. So a refusal here is not proof your firewall is blocking OpenAI — it is proof that something at your edge treats that identity differently, which is worth knowing either way. Your server logs are what settle which one it is.

4. Ask for something that cannot exist #

Request a random path nobody would ever create, and compare it to the homepage.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/zzq-does-not-exist-4831

It should 404, and it should not look like your homepage. If a URL that cannot exist answers with the same status and roughly the same size as the page you care about, you are not measuring a website — you are measuring a wall in front of one, and every reading you take through it is meaningless. No real site answers a URL that does not exist the same way it answers its homepage. This control fires on 6.8% of scans, and it exists because a scanner without it will confidently grade a firewall and hand you a number about your site that describes someone else's product. That rate was measured before the control was tightened on 1 September 2026; the rate measured since is on the dataset page.

5. Weigh the payload #

Take the bytes on the wire and work out how much of it is visible text a machine could quote. The rest is markup, inline styles and script that a crawler downloads and cannot use.

The heaviest page measured in this corpus delivered 476,540 bytes and 2.3% visible text. It rendered quickly, it read well, and no validator had an opinion about it. PAGE_IS_MOSTLY_CODE fires on 27.6% of scans. Inline CSS is the usual cause and it is the worst kind of weight, because it cannot be cached between pages — a crawler pays for it again on every URL it fetches.

What this does not tell you #

All five checks answer one question: can the thing that fetches your site read what came back. None of them tells you whether an assistant will cite you. Those are separate problems, and the order matters, because the second one is unanswerable until the first is clean. Nothing you write reaches an engine that is being handed a challenge page.

And one reading is one reading. A deterministic block reproduces perfectly, so running the same request twice from the same machine is not corroboration — it is the same measurement, taken twice. Check from a second network before you conclude anything, and treat a defect that appears on Tuesday and not on Wednesday as a defect that appears on Tuesday.

Why once is not enough #

The same sites, measured every day, produce two numbers a single check cannot separate — the live pair is below.

That is a 25-point gap, and 12 sites look clean now that did not look clean on at least one day in that period. A single audit can only ever produce the first number. The second one requires having watched.

Add the agent checks

The checks above answer whether a crawler can read the page. An agent acting for a person needs two more things to finish what it came for.

A free CrawlCheck audit covers the first three rows from outside. The last two are a two-minute test in an ordinary browser: open the page signed out, in a private window, and see whether the content and the form are reachable without clearing anything first.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• Your new cache rule did not fix the object it was written for
• Firewall or website? When an AI visibility tool grades your WAF
• Site audits go stale: the day you audited it is not the finding
• What answer engines receive from local business sites: mostly not text

Questions this post answers

How do I check if ChatGPT can read my website?

Fetch your robots.txt from outside your own network and read the content type rather than the status code: it must be text/plain. Then request your homepage twice, once with a browser user-agent and once as GPTBot, and compare the status and byte count. A difference in either means an edge rule is deciding who gets served.

Why does my site look fine in a browser but fail these checks?

A browser runs JavaScript, clears bot challenges, carries your existing cookies and often comes from an address your firewall already trusts. A crawler does none of that. The browser is the least representative reader of your own site.

What is the most common problem these checks find?

A stale cached copy served on the bare URL. It is the most common finding in this dataset, and the live share is published on the dataset page. It hides from most testing because adding a cache-buster to the URL bypasses the cached object and returns the correct file, while a crawler always requests the bare URL.

How do I check if AI agents can use my website?

Run the crawler checks first: robots.txt, content in the server response, and no challenge page served to datacentre addresses. Then open the page in a private browser window and confirm the content and your contact form are reachable without a consent wall or interstitial, because that is what a browser-driving agent meets.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works