Free tools for the agent answer, the crawler claim and the machine files.
Type a domain. No account, nothing installed. What an agent is told about your site before it acts, whether a machine can read your files, whether that crawler in your logs is real, what your robots.txt actually permits each AI agent, a first draft of your llms.txt from the pages you already publish, how you measure against a competitor, what Google can anchor on in a video, and whether a signed bot request comes from the bot it names. Every check is also a public endpoint an agent can call. A side-by-side of what each named crawler is served is agent lens. The per-crawler robots.txt reading is the robots.txt check for AI crawlers.
In one sentence
The free tools page answers eight questions without an account, each also callable as a public endpoint by an agent: what an AI agent is told about a domain by the signed Resolve answer (a decision per action, what the site declares, what it was served, what is absent); whether a machine can read its robots.txt, sitemap and llms.txt as the right content type or a cache is serving something stale; whether a request in your access logs claiming to be GPTBot, ClaudeBot, Googlebot or another named crawler came from that operator’s published IP ranges (verified, spoofed or unverifiable, never two outcomes); what a robots.txt actually permits each AI crawler at a path; a first draft of llms.txt from pages already published; a side-by-side against a competitor; what a search result can anchor on in a video; and whether a Web Bot Auth signed request comes from the bot it names.
- 01What is an agent told about your site?the signed Resolve answer, rendered
- 02Can a machine read your files?robots.txt · sitemap · llms.txt · homepage
- 03Is that crawler in your logs real?verified · spoofed · unverifiable
- 04What does your robots.txt let each AI crawler do?per agent, at the path you choose
- 05Draft an llms.txt from what you already publishhomepage + up to 25 declared pages
- 06How do you compare to a competitor?two homepages, one scan each
- 07Is that signed bot request real?Web Bot Auth, every step shown
- 08What can Google anchor on in your video?chapters · captions · description · topics
01
What is an agent told about your site?
The public, signed answer Resolve gives any agent that asks about a domain before it reads, cites, connects or transacts, rendered for a person: a decision per action, what the site declares, what it was served, what is absent.
Agents do not read your homepage and form an impression. They ask one endpoint and act on the fields. This shows you those fields for your own domain, and if there is no record yet, says so the way an agent is told: not measured, nothing guessed.
Agents: GET /api/v1/resolve?domain=
02
Can a machine read your files?
Four requests to the domain you name: robots.txt, sitemap, llms.txt and the homepage, read by content type and cache age.
Agents: GET /api/tool/files?domain=
03
Is that crawler in your logs real?
Paste raw access-log lines. Each one that names a known crawler is checked against the address ranges that operator actually publishes.
This is the check almost nobody runs, and it inverts conclusions. A log full of GPTBot 403s reads like your host is blocking OpenAI — until you find every one of them came from a single address wearing seven different crawler names. We have measured exactly that on live sites. Group by address before you conclude anything.
Agents: GET /api/v1/visitor?ip=&ua=
04
What does your robots.txt let each AI crawler do?
One fetch of the file, then every named answer engine, search index and training crawler resolved against it the way a crawler resolves it: its own group only, longest match, allow wins a tie.
The rule people get wrong: a Disallow under User-agent: * never applies to an agent that has its own group. Name GPTBot once anywhere in the file and every * rule stops applying to it. This shows what actually applies, per agent, at the path you choose.
Agents: GET /api/tool/robots?domain=&path=
05
Draft an llms.txt from what you already publish
Reads your homepage and up to 25 declared pages for their own titles and descriptions, and writes a first draft in the conventional shape.
It is a draft on purpose. A generated file is a link dump with titles as descriptions; the value of the file is the choosing. Cut it to the pages someone would ask an assistant about, write the descriptions yourself, then serve it as text/plain and check the bare URL — the check above will tell you whether it arrived.
06
How do you compare to a competitor?
Two homepages, the same scan on both: every section an answer engine needs, side by side, with the findings only one of you has.
07
Is that signed bot request real?
Web Bot Auth lets a bot sign every request with a key it publishes. Type the bot’s host, or paste a signed request’s headers, and every step of the check is shown: the key directory, the signatures it carries, each key’s thumbprint, the covered components and the time window.
A user-agent is a claim anyone can type; a signature over your own hostname is not. A key counts only if it signed the directory it sits in, so a copied or edited key list proves nothing. It works for any signing agent, not only ours.
Agents: GET /api/tool/wba?agent=
08
What can Google anchor on in your video?
Paste a YouTube URL: chapters, caption kind, description, language and topics, each row against what a search result can anchor to. Whole channels and the site tie-in are in Watch.
Three outcomes, never two
Counting the unverifiable as forged would inflate the rate; counting them as genuine would understate it. So the forgery rate divides by verified plus spoofed only, and the response states that denominator rather than leaving you to assume one.
Verified
The address is inside the range the operator publishes. The request is what it says it is.
Spoofed
The operator publishes ranges and this address is outside all of them. Someone else is wearing the name.
Unverifiable
The operator publishes no feed at all, so the claim can be neither confirmed nor accused. Meta-ExternalAgent sits here until its operator publishes one.
What the two checks catch
When to run them: after any edit to a machine file, after any cache or CDN configuration change, and any time a crawler-facing fix "should" be live. Check the bare URL from outside your own session, because your logged-in browser is the least representative client your site has.
The machine-file check
Fetches your robots.txt, llms.txt and sitemap the way a crawler does and reads the content type and body, not just the status. The failure it exists for: a bot shield answering HTTP 200 with HTML where a text file should be. Every uptime monitor calls that healthy; a validator reads it as everything allowed.
The purge check
Did the purge you just ran actually evict anything? It fetches the URL before and after with cache-busting disabled, compares the bytes, and reads the age and cache-status headers. A 200 from a purge endpoint is a statement about the API call, not about the cache.
The same two checks, for a terminal.
Neither is a product. They are the checks that caught real failures on this project, published because a post describing a tool you cannot have is an advertisement. Use them, change them, no attribution needed. They make no network request anywhere except the URL you hand them.
cachecheck.sh
Asks whether a cache is serving a document your origin no longer produces. It ignores status codes entirely: a PURGE that returns 200 has proved something answered, not that anything was evicted. It fetches the bare URL and a cache-busted one and compares them, and refuses to judge when either reading is a challenge page.
View or download it → · curl -sO https://crawlcheck.io/tools/cachecheck.sh
verify.sh
Checks whether a change you deployed is actually on the page a stranger receives. It counts your markup outside style and script tags, so a class name sitting in a stylesheet cannot pass as an element on the page; it fetches three times so anything conditional shows up; and it states that it measures the anonymous view.
View or download it → · curl -sO https://crawlcheck.io/tools/verify.sh
Want these run for you, twice a day?
These run when you remember to run them. Watch re-measures what every named AI crawler is served, twice a day, on three domains, and writes to you only when a result changes: the answer an agent gets about your site, kept current without anyone remembering to check.
The eight checks above stay free and need no account.
Questions about this page
QWhat does an AI agent see about my site before it acts?
QCan an agent call these tools directly?
QHow do I check that a signed bot request is real?
QHow does the crawler log verifier decide whether a request is real?
QIs anything I paste stored?
QWhat does the domain check look for?
QWhy does a purge that returned 200 still serve stale bytes?
Start from what you are checking
Free browser extension
Every check on this page, on any site you open.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.