Guides · 2026-10-08 · By VSNARY | Emmanuel Orta · 0 views
The agentic web for site owners: permission, delivery, identity and tools
AI systems are starting to act on websites, not just read them. What that changes for a site, the four questions an agent needs answered before it acts, and what you can check this week.
The agentic web is the part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. For a site owner it adds four checkable questions: permission (what an agent may fetch), delivery (whether the content is in what it was served), identity (who the site and the requester are) and tools (whether declared actions do what they say).
For two years the question was whether AI could read your site: whether GPTBot or ClaudeBot could fetch it, and whether an answer engine would cite it. The agentic web adds a second question. An agent working for a person does not stop at reading. It opens your page because someone asked it to, compares you with two competitors, fills in your quote form, or calls a tool your site exposes. Reading is the first step of that task, not the whole of it.
That traffic is still small and it is real. On crawlcheck.io, between 12 August and 8 October 2026, 613 of 25,718 verified requests from named AI and search agents (2.4%) came from user-triggered fetchers, the clients an assistant sends when a person asks about a page right now. The rest were crawlers. The full breakdown shows who they were and how many were forged.
Three kinds of AI client reach your site
| Client | Who decides to fetch | Examples | What it needs from you |
|---|---|---|---|
| Crawler | The operator, on its own schedule | GPTBot, ClaudeBot, PerplexityBot, Googlebot | Permission and readable content |
| User-triggered fetcher | A person asking an assistant | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent | The same, at the moment someone asks |
| Browser-driving agent | A person delegating a task | Assistants operating a real browser | A page and a form it can finish without a human-only step |
The crawler and agent list covers every name and how each is verified. The split that matters here is who decides: a crawler is governed well by robots.txt, an agent acting for a person only partly, because two of the three largest operators say their user-triggered fetchers may not follow it.
The four layers, and how to check each
| Layer | The question | How to check it | Guide |
|---|---|---|---|
| Permission | May an agent fetch this, and for what use? | Read your robots.txt group for each name; check Content Signals; fetch as each name and compare the status | Content Signals |
| Delivery | Is the content in what it was served? | Fetch the raw response and look for your headline, prices and address before any script runs | Rendering |
| Identity | Is this site who it says, and is this request who it says? | Your entity declarations agree everywhere; incoming requests checked by address range or signature | Entity SEO, signed agents |
| Tools | Does a declared action do what it says? | Read your MCP tool descriptions as an instruction to a model, because that is what they are | MCP check |
Permission
robots.txt still decides what crawlers fetch, and naming a crawler in a group changes which rules it reads, which is the most common way sites void their own Disallow lines. For agents, permission is increasingly enforced at the edge by verified identity, because a Disallow line is a request.
Delivery
What an agent is served is not always what a visitor sees. A firewall that challenges datacentre addresses serves a challenge page with a 200 status; a page assembled by script serves a shell. Browser-driving agents add a third failure: a consent wall or interstitial that a person clears in a second and an agent may not.
Identity
Two directions. Your site has to resolve to one business with one phone, address and service area, because an agent calls the number it resolves. And a request claiming to be an agent has to be checked: the name in a user agent is a claim anyone can type, and on this site 40% of Claude-User claims and 86% of Perplexity-User claims were forged.
Tools
Sites and services now expose tools to agents directly through MCP servers. A tool description is text a model reads as guidance. In the census of 25,983 registry endpoints, eleven servers carried tool text aimed at the model rather than the user, four of them telling it to keep something from the user.
What to check this week
- Fetch your homepage as GPTBot, ClaudeBot and PerplexityBot and confirm each gets a 200 with your content in the bytes. A free CrawlCheck audit does this for twelve crawler identities at once; repeat it for a user-triggered fetcher with
curl -A "ChatGPT-User". - Read your robots.txt groups for every name you mention, and confirm each named group repeats the Disallow lines you meant for everyone.
- Open your contact page in a private window and run the two-minute agent test.
- Check that your phone, address and service area are identical in your schema, your footer and your Google Business Profile.
- If you run an MCP server, read every tool description aloud as if it were an instruction. If it would be odd for a person to hear, it is odd for a model.
CrawlCheck is built for exactly this list: one signed answer about a site that an agent can verify before it reads, cites or acts, and a free audit a site owner can run on the same measurements.
Every figure above came out of CrawlCheck.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
What is the agentic web?
The part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. A site needs to be readable, trustworthy and usable by those agents, not only citable.
How is an AI agent different from an AI crawler?
A crawler fetches pages on its operator’s schedule to train or index. An agent fetches because a person asked it to do something now. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User are the agent-side clients sites see most often.
How much website traffic comes from AI agents?
Still a small share. On crawlcheck.io between 12 August and 8 October 2026, 613 of 25,718 verified requests from named AI and search agents (2.4%) came from user-triggered fetchers; the rest were crawlers.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.