What a website served, on a date, provably.
In one sentence
CrawlCheck evidence is the record the verification layer keeps of what a website served on a given day — to fifteen named crawler and browser identities, with the status, size and SHA-256 of each response, the machine files it published, and the changes since the last reading — reduced to its measurements, hashed, folded into one Merkle root per day that is anchored to the Bitcoin blockchain through OpenTimestamps, signed with a published Ed25519 key, and checkable on anyone’s machine from a downloadable bundle without asking CrawlCheck whether it is genuine; it proves that the record existed in exactly that form by that day and has not been altered since, and it does not prove that the measurement was correct.
A dispute about a website usually turns on one question: what did it serve, to whom, and when? A screenshot answers none of that. CrawlCheck keeps a dated record of what a site served to fifteen named crawler and browser identities, hashes it, seals each day’s records into one root anchored to the Bitcoin blockchain, signs every answer, and lets anyone check the record on their own machine without asking us whether it is genuine.
What one record holds
Every field is a measurement, not a description. Wording and layout are left out of the digest, so an editorial change can never break a proof.
Per-identity readings
The same URL requested as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Googlebot, Bingbot and the rest, and as a browser: status, bytes, and the SHA-256 of what each one received. Different bytes to different names is the finding a cloaking claim rests on.
Machine files, bodies hashed
robots.txt, the sitemap, llms.txt, the entity map and the security file as served that day, with their body hashes — so a rule that was added or removed later cannot be argued into the past.
Delivery parity
Whether an AI crawler was refused where a browser was served, and whether it got the same bytes. From CrawlCheck’s network, and — through the browser extension — from a residential address, so a block by name and address verification are told apart.
The change history
A watched domain is re-read twice a day and every finding carries a state: new, persisted, worsened, resolved. “It broke on the 14th” is a dated row, not a recollection.
How a record becomes a proof
No wallet, no fee, no key held by us. The proof is a file, not a coin.
- 1
Measurements only
Host, timestamp, every path and status fetched, every identity and what it received. Canonicalised with sorted keys and hashed with SHA-256.
- 2
One root a day
The day’s digests are sorted and folded into a Merkle tree. Arrival order cannot change the root.
- 3
Anchored to Bitcoin
The root goes to the OpenTimestamps calendars, which commit it into a Bitcoin transaction, usually within a day. The .ots file is published with the root.
- 4
Signed, and checked offline
Every answer carries an Ed25519 signature over its canonical content; the key is in a published directory. The verifier runs from the bundle’s bytes on the reader’s machine.
Proves
A record with this digest existed in exactly this form by that day, and has not been edited since. That claim is settled by mathematics, not by trusting CrawlCheck.
Does not prove
That the measurement was correct, or that the real crawler saw the same thing. CrawlCheck sends the named user-agent from its own network and from a browser; it never uses an operator’s address range, and a refusal is recorded as refused from here, never as blocked.
Four questions a record can answer
Each one is a question an expert has had to answer from screenshots and memory.
Was the site cloaking?
Per-identity hashes on the dates in question: whether Googlebot and a browser received the same bytes, and when that changed. A search engine’s terms-of-service claim stands or falls on this.
Was a crawler blocked, and by whom?
robots.txt as served that day, with every named group; whether the server refused the name; whether it refused only from a data centre. Negative-SEO and “you blocked us” claims become a dated table.
Did an AI crawler read the site after the opt-out?
For AI-licensing disputes, an evidence packet is built from the site’s own access logs: each hit that names an AI crawler is checked against the operator’s published address ranges, bound to what CrawlCheck read on those days, and sealed as its own leaf. The packet carries the hash of the log it was built from.
What did it look like before the complaint?
The census reads the 991 most-visited sites every day and the top 10,000 weekly, in public, since October 2026 — a third party’s reading taken before anyone had a reason to take one.
Three ways in
The public records are free. What costs money is depth and the record over time.
A record that already exists · free
Any report at its address, the census records, the daily seals and the verifier. If CrawlCheck read the site on the date in question, the record is public and checkable now.
A record from today on · Watch, $29 a month
The domain in question re-read twice a day with every finding’s state, from the day you start. The 90-day snapshot archive ($199 a month) adds every public signed answer behind the public window.
A matter · ask first
Send the domain and the dates. CrawlCheck says what the record holds for them — which days were read, by which identities, what was sealed — before anything is paid. An evidence packet for an AI-licensing dispute is built from the site’s logs on the API.
CrawlCheck measures and records; it does not give legal opinions, and this page is not legal advice. What a record proves and does not prove is stated above, and on every proof page.