Guides · 2026-10-09 · By VSNARY | Emmanuel Orta · 0 views
How an AI agent should verify a website before acting on it
Before an agent reads, cites, connects to or transacts with a domain, six questions need answers it can prove: what the site claims, what it serves, who controls it, what is safe to do, what changed and what evidence backs each answer.
Before an AI agent reads, cites, connects to or transacts with a domain it should establish six things: what the domain claims about itself, what it actually serves, who controls it, what is safe to do there, what changed since it was last checked, and what evidence proves each of those. Each action needs a different level of proof: reading needs the least, connecting and paying need the most. The answer should be signed, dated and citable, so the decision can be checked after the fact.
An answer engine reads a page and quotes it. An agent goes further: it follows links, fills in forms, connects to tools and sometimes pays. Every one of those steps assumes something about the domain on the other end, and most agents today assume it from the page itself. A page is a claim. This guide sets out the questions an agent should answer before each kind of action, what counts as evidence for each, and how to leave a record that someone can check afterwards.
Four actions, four levels of risk #
The actions an agent takes on a domain fall into four kinds, and the cost of being wrong rises with each:
- Read. Fetch a page to answer a question. Wrong means a bad answer.
- Cite. Name the page as the source of a fact. Wrong means a person trusts a claim because the agent vouched for it.
- Connect. Attach to a tool the domain offers, for example an MCP server. Wrong means the agent takes instructions from a stranger.
- Transact. Submit a form, book, buy or pay. Wrong means money or personal data goes to the wrong party.
A check that is enough to read is not enough to pay. An agent that applies one test to everything is either too strict to be useful or too loose to be safe.
The six questions #
| Question | Why the page cannot answer it alone | Evidence that does |
|---|---|---|
| What does the domain claim? | Claims are scattered across schema, robots.txt, llms.txt and profiles, and they contradict each other | Every declaration read in one pass, with where each came from |
| What does it actually serve? | A site can serve AI crawlers different text from what a person sees | The same page fetched as several identities and compared (read equivalence) |
| Who controls it? | Anyone can copy a logo and a name onto a lookalike domain | Lookalike screening, entity links that point back, signed agent identities |
| What is safe to do? | Declaring a tool is not proof it behaves as declared | A decision per action, with the evidence it rests on and its maximum age |
| What changed? | A page has no memory of its own history | Dated readings and a change feed; payment, login and mail fields watched |
| What proves it? | A summary from an AI is a second claim, not proof | A signature, an append-only transparency log and the fetched bytes |
1. What the domain claims #
A site declares itself in several places: agent manifests, robots.txt and its Content Signals, llms.txt, structured data, an A2A agent card, MCP listings. Read them together, because the useful findings are the disagreements: a robots.txt that allows GPTBot while a meta tag forbids it, or a schema phone number that is not the one on the page. Keep each claim labelled as a claim. A site saying it is “the official store” is evidence of what it says, not of what it is.
Absence matters too. “This site publishes no agent card” is a finding only if you record where you looked and what came back, which is what a signed absence does.
2. What it actually serves #
Fetch the page the way the agents that will use it do, and the way a person does, at the same time, and compare the main text. Differences are common and usually innocent: a bot challenge, a consent wall, a page that only renders in a browser. Occasionally they are not: text shown only to AI crawlers, or instructions aimed at a model. Either way the agent should know before it quotes the page. We published how this plays out across eight weeks of agent visits on our own site.
3. Who controls it #
Three checks together:
- Is it a lookalike? Compare the name against well-known brands for character swaps and added words. A homoglyph domain can be pixel-identical to the real one.
- Do its profiles point back? A sameAs link to a profile that does not link back is a claim, not a corroboration.
- Who is calling? When the domain sends its own agents, signed requests under Web Bot Auth (built on RFC 9421 HTTP Message Signatures) identify them better than a user agent string anyone can type.
4. What is safe to do #
Turn the evidence into a decision for each action: allow, warn, require confirmation, or block. Two rules keep this honest. First, the decision must say what it rests on and how old that evidence may be: a reading from last week can support a read, but not a payment. Second, a public record should never authorise everything. Connecting to a tool needs that tool to have been checked, and transacting should leave a person in the loop, whatever the record says.
For tools specifically, read the tool list before connecting. Our guide on checking an MCP server before an agent connects covers tool poisoning, tool shadowing and pinning what you approved in a tool lockfile.
5. What changed #
Most harm on the agentic web comes from a change nobody re-checked: a checkout that moved to a new host, a login that now points at a different provider, a tool description rewritten after approval (an MCP rug pull). An agent cannot re-read the whole web before every step, so it needs two things: a timestamp on every fact, and a feed that says when a watched field moved. Treat anything older than its action allows as unknown, not as true.
6. What proves it #
An agent that says “I checked” has made a third claim. Evidence that survives the agent is evidence a stranger can verify without trusting the agent or the service that produced it:
- A signature over the exact answer, with a public key published at a stable address.
- A transparency log. Each answer is a leaf in an append-only Merkle tree in the style of RFC 9162 (Certificate Transparency 2.0). An inclusion proof shows the answer was in the log; a signed tree head commits to the whole log, so the publisher cannot quietly show different people different answers.
- The bytes. Hashes of every file that was fetched, so a dispute can go back to what the site actually sent.
How to cite what you checked #
A citation to “the CrawlCheck page about example.com” points at something that will change tomorrow. A citation to the answer as it stood when you acted does not. RFC 8574 defines a cite-as link for this: the server names the permanent URL to cite. Every Resolve answer carries one in its Link header, pointing at an as-of answer:
curl -sI "https://crawlcheck.io/api/v1/resolve?domain=example.com" | grep -i ^link
# <…/api/v1/resolve?domain=example.com&as_of=2026-10-09T16:10:04.222Z>; rel="cite-as"Store that URL with the action. Anyone can later fetch it, check the signature and see exactly what the agent knew.
Doing it in one request #
CrawlCheck’s Resolve answers all six questions for a domain in one signed JSON document: declared and observed sections with their times, lookalike screening, watched high-risk fields, a decision for read, cite, connect and transact, the evidence links and the log inclusion. Run against one of our own sites on 9 October, it allowed read and cite, and required a person’s confirmation for connect and transact, because the site declares a capability nobody has verified and because CrawlCheck does not certify payments. An agent can verify the answer offline:
curl -s "https://crawlcheck.io/api/v1/resolve?domain=example.com" > answer.json
curl -sO https://crawlcheck.io/crawlcheck-verify.mjs
node crawlcheck-verify.mjs answer.jsonThe full instructions for agents, including the signed manifest at /.well-known/crawlcheck.json, are on the page for AI agents. Whatever service you use, apply the same test to it: if its answer is not signed, dated and citable, it is another claim.
How we measure ourselves #
The number we hold ourselves to is verified agent actions backed by a fresh signed Resolve record, counted by action type (read, cite, connect, transact), with false positives, disputes, unknowns and reversals counted alongside. A verification service that only reports its successes is making the same unbacked claim it exists to check.
Every figure above came out of CrawlCheck.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
What should an AI agent check before acting on a website?
Six things: what the domain claims, what it actually serves, who controls it, what is safe to do, what changed since the last check, and what evidence proves each answer. The level of proof should rise with the action: reading needs the least, connecting to tools and paying need the most.
Why is a webpage not enough evidence on its own?
A page is the site’s claim about itself. It can be copied onto a lookalike domain, served differently to AI crawlers than to people, or changed after the agent last read it. Evidence has to come from outside the page: comparison fetches, link-backs, dated readings and signatures.
How should an AI agent cite a source it verified?
Cite the answer as it stood when the agent acted, not a page that changes. RFC 8574 defines a cite-as link for this; CrawlCheck’s Resolve answers carry one pointing at a permanent as-of URL that returns the same signed answer later.
What is a transparency log and why does it matter for agents?
An append-only Merkle tree of published answers, in the style of Certificate Transparency (RFC 9162). An inclusion proof shows an answer was logged; a signed tree head commits to the whole log, so the publisher cannot show different answers to different people without being caught.
Should an agent ever pay automatically because a record says a site is safe?
No. A public record can support reading and citing, and help decide whether to connect, but payments, identity actions and anything irreversible should keep a person in the loop. CrawlCheck’s answers require confirmation for transactions for that reason.
Can an agent verify a CrawlCheck answer without trusting CrawlCheck?
Yes. Each answer is signed with an Ed25519 key published at a stable address and included in a public log. The open verifier script checks the signature, the log inclusion and the hashes offline.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.