Guides · 2026-10-08 · By VSNARY | Emmanuel Orta · 1 view
What is the agentic web? What it means for your website
AI systems now act on websites, not only read them. Which agents actually arrived on our own site, which followed the rules, and the four things you can check this week.
The agentic web is the part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. For a site owner it adds four checkable questions: permission (what an agent may fetch), delivery (whether the content is in what it was served), identity (who the site and the requester really are) and tools (whether declared actions do what they say).
For two years the question was whether AI could read your site: whether GPTBot or ClaudeBot could fetch it, and whether an answer engine would cite it. The agentic web adds a second question. An agent working for a person opens your page because someone asked it to, compares you with two competitors, fills in your quote form, or calls a tool your site exposes. Reading is the first step of that task, not the whole of it.
That traffic is still small and it is real. On crawlcheck.io, between 12 August and 8 October 2026, 613 of 25,718 verified requests from named AI and search agents (2.4%) came from user-triggered fetchers, the clients an assistant sends when a person asks about a page right now. The rest were crawlers. The full breakdown shows who they were.
Which kinds of AI client reach your site?
| Client | Who decides to fetch | Examples | What it needs from you |
|---|---|---|---|
| Crawler | The operator, on its own schedule | GPTBot, ClaudeBot, PerplexityBot, Googlebot | Permission and readable content |
| User-triggered fetcher | A person asking an assistant | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent | The same, at the moment someone asks |
| Browser-driving agent | A person delegating a task | Assistants operating a real browser | A page and a form it can finish without a human-only step |
The split that matters is who decides. A crawler is governed well by robots.txt; an agent acting for a person only partly, because two of the three largest operators say their user-triggered fetchers may not follow it. The crawler and agent list covers every name and how each is verified.
Which agents actually showed up?
From 13 August to 17 October 2026 we logged every visit to our own site from known AI and search crawlers, and counted a visit only when it came from the network addresses its operator publishes. The largest verified visitors:
| Agent | Operator | Verified requests |
|---|---|---|
| PerplexityBot | Perplexity | 8,336 |
| Googlebot | 6,666 | |
| Applebot | Apple | 4,624 |
| ClaudeBot | Anthropic | 4,287 |
| GPTBot | OpenAI | 4,084 |
| OAI-SearchBot | OpenAI | 1,980 |
| ChatGPT-User (a person asked) | OpenAI | 665 |
Some requests used one of these names without coming from the operator’s network. Anyone can type “GPTBot” into a request; only its operator can send it from its own addresses. The live forged share on this site:
Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate. Sentence form, as of 2026-11-22: 11.2% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25688 of 228894 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was Google-Extended at 100%.
Do agents follow the rules a site sets?
Our robots.txt names every major AI crawler and blocks three paths for all of them. Checked request by request over the same weeks, Googlebot, Applebot, Bingbot, OAI-SearchBot and ChatGPT-User read the file and stayed out. GPTBot stayed out too, though it never fetched the file under its own name, so we cannot credit it with reading it.
One crawler did not stay out every time. On 16 October ClaudeBot fetched a blocked path three times in about ten minutes, from Anthropic’s own published network: three requests out of 4,287. That is not a crawler ignoring the rules wholesale, but “almost always” is not “always”. The running record is in the October conduct report.
The four layers, and how to check each
| Layer | The question | How to check it | Guide |
|---|---|---|---|
| Permission | May an agent fetch this, and for what use? | Read your robots.txt group for each name; check Content Signals; fetch as each name and compare the status | Content Signals |
| Delivery | Is the content in what it was served? | Fetch the raw response and look for your headline, prices and address before any script runs | Rendering |
| Identity | Is this site who it says, and is this request who it says? | Your entity declarations agree everywhere; incoming requests checked by address range or signature | Entity SEO, signed agents |
| Tools | Does a declared action do what it says? | Read your MCP tool descriptions as an instruction to a model, because that is what they are | MCP check |
Permission
robots.txt still decides what crawlers fetch, and naming a crawler in a group changes which rules it reads, which is the most common way sites void their own Disallow lines. For agents, permission is increasingly enforced at the edge by verified identity, because a Disallow line is a request.
Delivery
What an agent is served is not always what a visitor sees. A firewall that challenges datacentre addresses serves a challenge page with a 200 status; a page assembled by script serves a shell. Browser-driving agents add a third failure: a consent wall or interstitial that a person clears in a second and an agent may not.
Identity
Two directions. Your site has to resolve to one business with one phone, address and service area, because an agent calls the number it resolves. And a request claiming to be an agent has to be checked against its operator’s published ranges or a signature, because the name alone is a costume. How much crawler traffic is forged has the live figures per name.
Tools
Sites and services now expose tools to agents directly through MCP servers, and a tool description is text a model reads as guidance. In our census of 25,983 registry endpoints, eleven servers carried tool text aimed at the model rather than the user, four of them telling it to keep something from the user.
Many agentic-web explainers show a block like this and call it an MCP schema. It is a generic function-calling definition, the format chat APIs use:
{
"name": "searchTickets",
"description": "Searches support tickets by status and priority",
"parameters": {
"type": "object",
"properties": {
"status": { "type": "string", "enum": ["open", "closed", "pending"] },
"priority": { "type": "string", "enum": ["low", "medium", "high"] }
},
"required": ["status"]
}
}The Model Context Protocol describes the same tool with inputSchema instead of parameters, and can say whether calling it changes anything, which is what a client reads before deciding to call a tool automatically:
{
"name": "searchTickets",
"description": "Searches support tickets by status and priority",
"inputSchema": {
"type": "object",
"properties": {
"status": { "type": "string", "enum": ["open", "closed", "pending"] },
"priority": { "type": "string", "enum": ["low", "medium", "high"] }
},
"required": ["status"]
},
"annotations": { "readOnlyHint": true }
}What does an agent do with a business it cannot check?
Agents are increasingly asked to pick a business: a plumber, a fence company, a dentist. In 422 paired rankings, lowering a fictional business’s star rating from 4.8 to 4.1 pushed it down the list in every test, outweighing licence numbers, insurance and years in business. An agent rewards whatever a page says, so the facts on yours have to be true, current and in plain text. In the first run of our trap test across four models, the dangers models missed were the ones a page cannot show: a receipt revoked since, or a fact that had gone out of date.
What should you check this week?
- Fetch your homepage as GPTBot, ClaudeBot and PerplexityBot and confirm each gets a 200 with your content in the bytes. A free CrawlCheck audit does this for the main crawler identities at once; repeat it for a user-triggered fetcher with
curl -A "ChatGPT-User". - Read your robots.txt groups for every name you mention, and confirm each named group repeats the Disallow lines you meant for everyone.
- Open your contact page in a private window and run the two-minute agent test.
- Check that your rating, review count, phone, address and service area are on the page as plain text and identical in your schema, your footer and your Google Business Profile.
- If you run an MCP server, read every tool description aloud as if it were an instruction. If it would be odd for a person to hear, it is odd for a model.
The crawler figures come from one website, ours, not a sample of the web. Published by CrawlCheck.
CrawlCheck’s dataset when this page loaded: 585,216 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is the agentic web?
The part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. A site needs to be readable, trustworthy and usable by those agents, not only citable.
How is an AI agent different from an AI crawler?
A crawler fetches pages on its operator’s schedule to train or index. An agent fetches because a person asked it to do something now. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User are the agent-side clients sites see most often.
How much website traffic comes from AI agents?
Still a small share. On crawlcheck.io most verified requests from named AI and search agents come from crawlers; user-triggered fetchers, sent when a person asks about a page, are a small minority.
Do AI agents follow robots.txt?
Crawlers mostly do, though not perfectly every time. User-triggered fetchers are a different case: some large operators say those fetchers may not follow robots.txt, because a person asked for the page.
Can I trust a visitor that says it is GPTBot or ClaudeBot?
Not from the name alone. A user agent is text anyone can send. Check the request’s IP address against the operator’s published ranges, or verify a signature where the agent signs its requests.
What is the first thing a site owner should check for AI agents?
Whether AI crawlers get a 200 with your real content in the response. Access and delivery come before identity and tools, because an agent that is refused or served a challenge page never reaches anything else.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.