Agent view · Guides · 2026-10-08
What is the agentic web? What it means for your website
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/what-the-agentic-web-means-for-a-site-owner: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/what-the-agentic-web-means-for-a-site-owner>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 19,697 |
| words | 2,772 |
| sha-256 | e89a34fffd7426f555ca2ff2dac7b0143109d00ac0178dd372f5c0a8c47a940e |
| repr-digest | sha-256=:6Jo0//10JvVVyi/y2sewFDEJ0ArAF43TcvXAqMR6lA4=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795378016;expires=1795381616;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="gXavk3InhdaL0bs56_b2gg";tag="crawlcheck-page" |
| signature | sig1=:DVw+RmEx7pazG8rMUSD32q0zggzZFYtQoCgRR47WGGEIOQCJkKFqB3copuiPU0pXRy6KtDvvygq/GiThTUQfDg==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-08 · By [VSNARY | Emmanuel Orta](/author) · 1 view
# What is the agentic web? What it means for your website
AI systems now act on websites, not only read them. Which agents actually arrived on our own site, which followed the rules, and the four things you can check this week.
The agentic web is the part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. For a site owner it adds four checkable questions: permission (what an agent may fetch), delivery (whether the content is in what it was served), identity (who the site and the requester really are) and tools (whether declared actions do what they say).
Check your own site free
Check my site
About a minute. First scan free with your email, nothing installed.
On this page
- Which kinds of AI client reach your site?
- Which agents actually showed up?
- Do agents follow the rules a site sets?
- The four layers, and how to check each
- What does an agent do with a business it cannot check?
- What should you check this week?
For two years the question was whether AI could *read* your site: whether GPTBot or ClaudeBot could fetch it, and whether an [answer engine](/glossary/answer-engine) would cite it. The [agentic web](/glossary/agentic-web) adds a second question. An agent working for a person opens your page because someone asked it to, compares you with two competitors, fills in your quote form, or calls a tool your site exposes. Reading is the first step of that task, not the whole of it.
Every layer can be checked from outside the site. None of them requires knowing what a model concluded.
That traffic is still small and it is real. On crawlcheck.io, between 12 August and 8 October 2026, **613 of 25,718 verified requests from named AI and search agents (2.4%)** came from user-triggered fetchers, the clients an assistant sends when a person asks about a page right now. The rest were crawlers. [The full breakdown](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt) shows who they were.
## Which kinds of AI client reach your site?
| Client | Who decides to fetch | Examples | What it needs from you |
| --- | --- | --- | --- |
| Crawler | The operator, on its own schedule | GPTBot, ClaudeBot, PerplexityBot, Googlebot | Permission and readable content |
| User-triggered fetcher | A person asking an assistant | ChatGPT-User, Claude-User, Perplexity-User, [Google-Agent](/glossary/google-agent) | The same, at the moment someone asks |
| Browser-driving agent | A person delegating a task | Assistants operating a real browser | A page and a form it can finish without a human-only step |
The split that matters is who decides. A crawler is governed well by [robots.txt](/glossary/robots-txt); an agent acting for a person only partly, because [two of the three largest operators say their user-triggered fetchers may not follow it](/guides/do-ai-crawlers-respect-robots-txt#agents). The [crawler and agent list](/guides/ai-crawler-list-what-each-one-does) covers every name and how each is verified.
## Which agents actually showed up?
From 13 August to 17 October 2026 we logged every visit to our own site from known AI and search crawlers, and counted a visit only when it came from the network addresses its operator publishes. The largest verified visitors:
| Agent | Operator | Verified requests |
| --- | --- | --- |
| PerplexityBot | Perplexity | 8,336 |
| Googlebot | Google | 6,666 |
| Applebot | Apple | 4,624 |
| ClaudeBot | Anthropic | 4,287 |
| GPTBot | OpenAI | 4,084 |
| OAI-SearchBot | OpenAI | 1,980 |
| ChatGPT-User (a person asked) | OpenAI | 665 |
Some requests used one of these names without coming from the operator’s network. Anyone can type “GPTBot” into a request; only its operator can send it from its own addresses. The live forged share on this site:
**Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate`. Sentence form, as of 2026-11-22: *11.2% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25688 of 228894 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was [Google-Extended](/glossary/google-extended) at 100%.*
## Do agents follow the rules a site sets?
Our robots.txt names every major AI crawler and blocks three paths for all of them. Checked request by request over the same weeks, Googlebot, Applebot, Bingbot, OAI-SearchBot and ChatGPT-User read the file and stayed out. GPTBot stayed out too, though it never fetched the file under its own name, so we cannot credit it with reading it.
One crawler did not stay out every time. On 16 October ClaudeBot fetched a blocked path three times in about ten minutes, from Anthropic’s own published network: three requests out of 4,287. That is not a crawler ignoring the rules wholesale, but “almost always” is not “always”. The running record is in [the October conduct report](/blog/agent-conduct-october-2026).
## The four layers, and how to check each
| Layer | The question | How to check it | Guide |
| --- | --- | --- | --- |
| Permission | May an agent fetch this, and for what use? | Read your robots.txt group for each name; check [Content Signals](/glossary/content-signals); fetch as each name and compare the status | [Content Signals](/guides/content-signals-in-robots-txt-search-ai-input-ai-train) |
| Delivery | Is the content in what it was served? | Fetch the raw response and look for your headline, prices and address before any script runs | [Rendering](/guides/do-ai-crawlers-render-javascript) |
| Identity | Is this site who it says, and is this request who it says? | Your entity declarations agree everywhere; incoming requests checked by address range or signature | [Entity SEO](/guides/what-is-entity-seo), [signed agents](/guides/web-bot-auth-how-signed-crawler-requests-work) |
| Tools | Does a declared action do what it says? | Read your [MCP tool](/glossary/mcp-tool) descriptions as an instruction to a model, because that is what they are | [MCP check](/guides/how-to-check-an-mcp-server-before-an-agent-connects) |
### Permission
robots.txt still decides what crawlers fetch, and naming a crawler in a group changes which rules it reads, which is [the most common way sites void their own Disallow lines](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules). For agents, permission is increasingly enforced at the edge by verified identity, because a Disallow line is a request.
### Delivery
What an agent is served is not always what a visitor sees. A firewall that challenges datacentre addresses serves a [challenge page with a 200 status](/blog/a-file-that-answered-200-and-could-not-be-read); a page assembled by script serves a shell. Browser-driving agents add a third failure: a consent wall or interstitial that a person clears in a second and an agent may not.
### Identity
Two directions. Your site has to resolve to one business with one phone, address and [service area](/glossary/service-area), because an agent [calls the number it resolves](/guides/what-is-entity-seo#agents). And a request claiming to be an agent has to be checked against its operator’s published ranges or a signature, because the name alone is a costume. [How much crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged) has the live figures per name.
### Tools
Sites and services now expose tools to agents directly through [MCP](/glossary/mcp) servers, and a tool description is text a model reads as guidance. In our [census of 25,983 registry endpoints](/blog/mcp-registry-census-1851-servers-what-they-answer), eleven servers carried tool text aimed at the model rather than the user, four of them telling it to keep something from the user.
Many agentic-web explainers show a block like this and call it an MCP schema. It is a generic function-calling definition, the format chat APIs use:
```
{
"name": "searchTickets",
"description": "Searches support tickets by status and priority",
"parameters": {
"type": "object",
"properties": {
"status": { "type": "string", "enum": ["open", "closed", "pending"] },
"priority": { "type": "string", "enum": ["low", "medium", "high"] }
},
"required": ["status"]
}
}
```
The Model Context Protocol describes the same tool with **inputSchema** instead of **parameters**, and can say whether calling it changes anything, which is what a client reads before deciding to call a tool automatically:
```
{
"name": "searchTickets",
"description": "Searches support tickets by status and priority",
"inputSchema": {
"type": "object",
"properties": {
"status": { "type": "string", "enum": ["open", "closed", "pending"] },
"priority": { "type": "string", "enum": ["low", "medium", "high"] }
},
"required": ["status"]
},
"annotations": { "readOnlyHint": true }
}
```
## What does an agent do with a business it cannot check?
Agents are increasingly asked to pick a business: a plumber, a fence company, a dentist. In [422 paired rankings](/blog/ai-picked-a-local-contractor-422-times-the-rating-decided-it), lowering a fictional business’s star rating from 4.8 to 4.1 pushed it down the list in every test, outweighing licence numbers, insurance and years in business. An agent rewards whatever a page says, so the facts on yours have to be true, current and in plain text. In the first run of our [trap test across four models](/blog/do-ai-agents-stop-before-acting-on-a-lookalike-site), the dangers models missed were the ones a page cannot show: a receipt revoked since, or a fact that had gone out of date.
## What should you check this week?
- Fetch your homepage as GPTBot, ClaudeBot and PerplexityBot and confirm each gets a 200 with your content in the bytes. A [free CrawlCheck audit](/) does this for the main crawler identities at once; repeat it for a user-triggered fetcher with `curl -A "ChatGPT-User"`.
- Read your robots.txt groups for every name you mention, and confirm each named group repeats the Disallow lines you meant for everyone.
- Open your contact page in a private window and [run the two-minute agent test](/guides/can-an-ai-agent-use-your-contact-form).
- Check that your rating, review count, phone, address and service area are on the page as plain text and identical in your schema, your footer and your [Google Business Profile](/glossary/google-business-profile).
- If you run an [MCP server](/glossary/mcp-server), read every tool description aloud as if it were an instruction. If it would be odd for a person to hear, it is odd for a model.
*The crawler figures come from one website, ours, not a sample of the web. Published by CrawlCheck.*
CrawlCheck’s dataset when this page loaded: 585,216 sites measured.
Written by
[Emmanuel Orta](/author) · VSNARY
I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.
[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)
Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fwhat-the-agentic-web-means-for-a-site-owner%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fwhat-the-agentic-web-means-for-a-site-owner%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=What%20is%20the%20agentic%20web%3F%20What%20it%20means%20for%20your%20website)[Email](mailto:?subject=What%20is%20the%20agentic%20web%3F%20What%20it%20means%20for%20your%20website&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fwhat-the-agentic-web-means-for-a-site-owner)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)
## Questions this post answers
### What is the agentic web?
The part of the web used by AI agents that act for a person rather than only answering: opening pages, comparing options, filling in forms and calling tools. A site needs to be readable, trustworthy and usable by those agents, not only citable.
### How is an AI agent different from an AI crawler?
A crawler fetches pages on its operator’s schedule to train or index. An agent fetches because a person asked it to do something now. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User are the agent-side clients sites see most often.
### How much website traffic comes from AI agents?
Still a small share. On crawlcheck.io most verified requests from named AI and search agents come from crawlers; user-triggered fetchers, sent when a person asks about a page, are a small minority.
### Do AI agents follow robots.txt?
Crawlers mostly do, though not perfectly every time. User-triggered fetchers are a different case: some large operators say those fetchers may not follow robots.txt, because a person asked for the page.
### Can I trust a visitor that says it is GPTBot or ClaudeBot?
Not from the name alone. A user agent is text anyone can send. Check the request’s IP address against the operator’s published ranges, or verify a signature where the agent signs its requests.
### What is the first thing a site owner should check for AI agents?
Whether AI crawlers get a 200 with your real content in the response. Access and delivery come before identity and tools, because an agent that is refused or served a challenge page never reaches anything else.
## Related findings
[2026-10-15 Robots.txt allows it, the server refuses: 57 top sites At 57 of 418 top sites, the server refused GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot even though robots.txt allows it. How](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)[2026-10-08 AI agents acting for people: 613 visits, one read robots.txt ChatGPT-User made 526 verified visits, Claude-User 79, Perplexity-User 8. Only Claude-User read robots.txt; none fetched a disallo](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt)[2026-11-15 Top 1,000 websites: every outside tracker we logged We opened the 1,000 most visited homepages in a browser and logged every outside company. 685 measured; Tag Manager on 38%, Double](/blog/top-1000-websites-third-party-trackers-november-2026)
The main product
Found this on your own site? **We fix it for $100.**
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)
Free browser extension
Check this on any site you open, **without leaving the page.**
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)
Write for the library
Measured something yourself? **Submit an article.**
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week
Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).
## Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.
Send for review
Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.
[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)
We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)
#### How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).
**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.
**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.
**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [jammiwoo.gr](/registry/jammiwoo.gr) · [patreon.com](/registry/patreon.com) · [whatsapp.com](/registry/whatsapp.com) · [agentalog.com](/registry/capabilities/agentalog.com) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) | **Datasets:** [Idaho](/datasets/state/idaho) · [Technologies: Divi](/datasets/technologies/divi) · [Datasets](/datasets) | **Blog:** [areaServed vs service area pages: 20 towns declared, 1 page](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them)
[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/what-the-agentic-web-means-for-a-site-owner.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.