Free tools
Six free checks, no signup
In one sentence
The free tools page answers two questions in seconds without an account: is a request in your access logs that claims to be GPTBot, ClaudeBot, Googlebot or another named crawler genuinely from that operator (verified against the IP ranges the operator publishes — verified, spoofed or unverifiable, never two outcomes), and can a machine read your robots.txt, sitemap and llms.txt as the right content type, or is a cache serving something your server stopped producing.
Type a domain. No signup, nothing installed. Can a machine read your files, is that crawler in your logs real, what does your robots.txt actually permit each AI agent, a first draft of your llms.txt from the pages you already publish, how you measure against a competitor, and what Google can anchor on in any YouTube video.
Can a machine read your files?
Four requests to the domain you name: robots.txt, sitemap, llms.txt and the homepage, read by content type and cache age.
Is that crawler in your logs real?
Paste raw access-log lines. Each one that names a known crawler is checked against the address ranges that operator actually publishes.
This is the check almost nobody runs, and it inverts conclusions. A log full of GPTBot 403s reads like your host is blocking OpenAI — until you find every one of them came from a single address wearing seven different crawler names. We have measured exactly that on live sites. Group by address before you conclude anything.
What does your robots.txt let each AI crawler do?
One fetch of the file, then every named answer engine, search index and training crawler resolved against it the way a crawler resolves it: its own group only, longest match, allow wins a tie.
The rule people get wrong: a Disallow under User-agent: * never applies to an agent that has its own group. Name GPTBot once anywhere in the file and every * rule stops applying to it. This shows what actually applies, per agent, at the path you choose.
Draft an llms.txt from what you already publish
Reads your homepage and up to 25 declared pages for their own titles and descriptions, and writes a first draft in the conventional shape.
It is a draft on purpose. A generated file is a link dump with titles as descriptions; the value of the file is the choosing. Cut it to the pages someone would ask an assistant about, write the descriptions yourself, then serve it as text/plain and check the bare URL — the check above will tell you whether it arrived.
How do you compare to a competitor?
Two homepages, the same scan on both: every section an answer engine needs, side by side, with the findings only one of you has.
What can Google anchor on in your video?
Paste a YouTube URL: chapters, caption kind, description, language and topics, each row against what a search result can anchor to. Whole channels and the site tie-in are in Watch.
Three outcomes, never two
Verified
The address is inside the range the operator publishes. The request is what it says it is.
Spoofed
The operator publishes ranges and this address is outside all of them. Someone else is wearing the name.
Unverifiable
The operator publishes no feed at all, so the claim can be neither confirmed nor accused. Amazonbot and Meta-ExternalAgent live here permanently.
Counting the unverifiable as forged would inflate the rate; counting them as genuine would understate it. So the forgery rate divides by verified plus spoofed only, and the response states that denominator rather than leaving you to assume one.
What the two checks catch
The machine-file check fetches your robots.txt, llms.txt and sitemap the way a crawler does and reads the content type and body, not just the status. The failure it exists for: a bot shield or challenge page answering HTTP 200 with HTML where a text file should be. Every uptime monitor calls that healthy. A validator parses it as "no directives", which reads as everything allowed — the exact opposite of what your file says. We found this serving a verification page as robots.txt to every crawler including Googlebot, cached at the edge, for hours. The check takes one request per file and reports what a machine would actually parse.
The purge check answers a question your CDN dashboard will not: did the purge you just ran actually evict anything? It fetches the URL before and after with cache-busting disabled, compares the bytes, and reads the age and cache-status headers. The failure it exists for: a purge API that returns 200 against one cache layer while another layer — Varnish in front of an origin, or an edge rule pinning a TTL — keeps serving the old bytes. A 200 from a purge endpoint is a statement about the API call, not about the cache. We shipped an edit, purged, got a 200, and the anonymous URL served the old file for another hour; that hour is why this script exists.
When to run them: after any edit to a machine file, after any cache or CDN configuration change, and any time a crawler-facing fix "should" be live. The one habit that matters: check the bare URL from outside your own session, because your logged-in browser is the least representative client your site has.
Take the scripts
The same two checks, for a terminal. These are the ones we run.
Neither is a product. They are the checks that caught real failures on this project, published because a post describing a tool you cannot have is an advertisement.
cachecheck.sh
Asks whether a cache is serving a document your origin no longer produces. It ignores status codes entirely: a PURGE that returns 200 has proved something answered, not that anything was evicted. It fetches the bare URL and a cache-busted one and compares them, and refuses to judge when either reading is a challenge page.
View or download it → · curl -sO https://crawlcheck.io/tools/cachecheck.sh
verify.sh
Checks whether a change you deployed is actually on the page a stranger receives. It counts your markup outside style and script tags, so a class name sitting in a stylesheet cannot pass as an element on the page; it fetches three times so anything conditional shows up; and it states that it measures the anonymous view.
View or download it → · curl -sO https://crawlcheck.io/tools/verify.sh
Use them, change them, no attribution needed. They make no network request anywhere except the URL you hand them.
Want the check run for you, twice a day?
These two run when you remember to run them. The same checks on a schedule, with a note the moment a result changes, is what monitoring is. Leave an address and we will tell you when it is ready — no account, and nothing sent in the meantime.
One address, one purpose. The scripts above stay free and need none of this.