Library · Crawler access and robots.txt
Crawler access and robots.txt
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
Whether a crawler is let in, and whether the server agrees with what robots.txt says.
2026-11-02 · Case studies6 min
robots.txt returns 200 but no crawler can read it
Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it.
CHALLENGE_SERVED_2000.4%UNIFORM_REFUSAL11.7%
Read the measurements →
2026-11-027 min
How to block AI crawlers like GPTBot and ClaudeBot
Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that usually get made as one, and the one people regret is rarely the one they intended.
AI_OPTOUT_SET2.9%ROBOTS_RULES_SHADOWED13%ANSWER_ENGINE_REFUSED6.1%+2
Read the measurements →
2026-11-025 min
How to check if AI crawlers can read your site
Five requests from outside your network, ten minutes, no account. What each one catches, and why your browser is the one test that cannot find any of it.
ROBOTS_NOT_2005.6%STALE_CACHE_SERVED15.4%ANSWER_ENGINE_REFUSED6.1%+3
Read the measurements →
2026-11-027 min
Cloudflare AI Crawl Control vs CrawlCheck: block vs measure
One controls AI crawler access at Cloudflare’s edge; the other measures, from outside, what each crawler is actually served. What each does, where each is weaker, and the routine that uses both.
ANSWER_ENGINE_REFUSED6.1%
Read the measurements →
2026-11-029 min
How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared
Policy testers tell you whether your file allows a crawler; only a delivery check tells you whether the crawler gets the page. Five tools compared, the two-minute test, and the mistakes we see most.
ANSWER_ENGINE_REFUSED6.1%
Read the measurements →
2026-11-025 min
Is Cloudflare blocking AI crawlers on your site?
Cloudflare’s defaults changed on 15 September 2026 and reached hardest the sites that never opened the setting. How to tell which side of it you are on.
ANSWER_ENGINE_REFUSED6.1%AI_OPTOUT_SET2.9%CHALLENGE_SERVED_2000.4%+3
Read the measurements →
2026-11-024 min
Content-Signal in robots.txt: search, ai-input, ai-train
Few top sites use Content-Signal yet: 4.3% of the top 10,000 in our census carry the line. What each value means, why it blocks nothing, and the one setting that costs you citations.
AI_OPTOUT_SET2.9%
Read the measurements →
2026-11-022 min
Google-Extended and Applebot-Extended: block AI training
Neither is a crawler. Both are robots.txt names that decide how pages Google and Apple already crawl may be used for AI, and neither affects search ranking. How to use them, and how many top sites already do.
Read the measurements →
2026-11-02 · Case studies4 min
Naming a crawler cancels your * rules: Semrush's robots.txt
We checked twenty-one audit tools against their own robots.txt. Three carry a group that voids the rules above it. One carried a joke, and CrawlCheck marked it down for that, which was our defect, not theirs.
Read the measurements →
2026-11-025 min
Do AI crawlers and agents respect robots.txt?
The named crawlers largely do, and we checked who fetched our own file and who broke it. The costlier problem is a file that cannot be read, or an edge that refuses what the file invites.
ROBOTS_NOT_2005.6%CHALLENGE_SERVED_2000.4%ROBOTS_IS_HTML0.1%+9
Read the measurements →
2026-11-027 min
Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexing
Three mechanisms share a name and answer three different questions at three different moments. Most robots failures are a directive placed in a layer that cannot read it, starting with the noindex behind a Disallow.
ROBOTS_RULES_SHADOWED13%ROBOTS_DISALLOW_ALL1.1%ROBOTS_IS_HTML0.1%+2
Read the measurements →
2026-11-026 min
robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers
The status code on robots.txt decides how a crawler treats the whole site before it reads a single rule. A 404 means crawl everything. A 503 means crawl nothing. A 403 is where crawlers disagree.
ROBOTS_IS_CATCHALL0.6%ROBOTS_IS_HTML0.1%ROBOTS_CTYPE0.1%+2
Read the measurements →
2026-11-027 min
Naming a crawler in robots.txt drops your Disallow rules
A crawler obeys exactly one group in robots.txt, the most specific one that names it. Add a group for GPTBot with Allow: / and every Disallow under User-agent: * stops applying to GPTBot. How to see it, how to write the file so it cannot happen, and the full-block shape that only looks like the mistake.
ROBOTS_RULES_SHADOWED13%ROBOTS_IS_CATCHALL0.6%
Read the measurements →
2026-11-02 · Research · Exclusive5 min
402 Payment Required: what top sites send AI crawlers
Three of the top 1,000 sites answer AI crawlers with HTTP 402. None quotes a price: one is a refusal, one a toll, one contradicts its own robots.txt.
Read the measurements →