CrawlCheck
Data · census · weekly

Agentic web census, week 2026-W40

Sealed 2026-10-09. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.

top 1,000, daily

Run 3 (finished 2026-10-04), 991 sites, no earlier run to compare with.

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 991		51.5%——
Serves a robots.txt (200, not HTML)
of 510		79%——
robots.txt names a sitemap
of 403		69.7%——
robots.txt carries a Content-Signal line
of 403		3.7%——
Content-Signal says ai-train=no
of 403		2.2%——
Content-Signal says ai-input=no or search=no
of 403		1%——
robots.txt closes / to at least one of the 12 AI agents
of 403		24.6%——
robots.txt closes / to GPTBot
of 403		16.6%——
robots.txt closes / to OAI-SearchBot
of 403		8.2%——
robots.txt closes / to ChatGPT-User
of 403		10.4%——
robots.txt closes / to ClaudeBot
of 403		15.4%——
robots.txt closes / to Claude-SearchBot
of 403		8.4%——
robots.txt closes / to PerplexityBot
of 403		12.4%——
robots.txt closes / to Google-Extended
of 403		13.9%——
robots.txt closes / to Applebot-Extended
of 403		11.9%——
robots.txt closes / to CCBot
of 403		19.6%——
robots.txt closes / to Bytespider
of 403		18.9%——
robots.txt closes / to Meta-ExternalAgent
of 403		15.1%——
robots.txt closes / to Amazonbot
of 403		12.9%——
Serves /llms.txt
of 510		20.6%——
Serves /agents.md
of 510		2.7%——
Publishes an agent-skills index
of 510		2%——
Publishes an MCP server card
of 510		2.4%——
Publishes an API catalog (RFC 9727)
of 510		2.2%——
Publishes an A2A agent card
of 510		0.4%——
Declares any agent capability (any of the four above)
of 510		2.7%——
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 510		17.1%——
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 510		3.7%——
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 0		—%——
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 0		—%——
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 0		—%——
The A2A agent card names an endpoint url
of 0		—%——
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 0		—%——
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 0		—%——
GPTBot refused where the browser got 2xx
of 510		14.7%——
ClaudeBot refused where the browser got 2xx
of 510		13.9%——
OAI-SearchBot refused where the browser got 2xx
of 510		9.4%——
PerplexityBot refused where the browser got 2xx
of 510		10%——

top 10,000, weekly

Run 1 (finished 2026-10-03), 9776 sites, no earlier run to compare with.

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 9776		57.7%——
Serves a robots.txt (200, not HTML)
of 5643		80.8%——
robots.txt names a sitemap
of 4562		69.7%——
robots.txt carries a Content-Signal line
of 4562		3.2%——
Content-Signal says ai-train=no
of 4562		2%——
Content-Signal says ai-input=no or search=no
of 4562		0.5%——
robots.txt closes / to at least one of the 12 AI agents
of 4562		19.1%——
robots.txt closes / to GPTBot
of 4562		13.7%——
robots.txt closes / to OAI-SearchBot
of 4562		6.4%——
robots.txt closes / to ChatGPT-User
of 4562		8.4%——
robots.txt closes / to ClaudeBot
of 4562		12.3%——
robots.txt closes / to Claude-SearchBot
of 4562		5.9%——
robots.txt closes / to PerplexityBot
of 4562		9.4%——
robots.txt closes / to Google-Extended
of 4562		10.4%——
robots.txt closes / to Applebot-Extended
of 4562		10.1%——
robots.txt closes / to CCBot
of 4562		14.9%——
robots.txt closes / to Bytespider
of 4562		14%——
robots.txt closes / to Meta-ExternalAgent
of 4562		11.1%——
robots.txt closes / to Amazonbot
of 4562		10.2%——
Serves /llms.txt
of 5643		15.4%——
Serves /agents.md
of 5643		1.3%——
Publishes an agent-skills index
of 5643		0.9%——
Publishes an MCP server card
of 5643		0.9%——
Publishes an API catalog (RFC 9727)
of 5643		1.1%——
Publishes an A2A agent card
of 5643		0.3%——
Declares any agent capability (any of the four above)
of 5643		1.3%——
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 5643		22.9%——
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 5643		2.4%——
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 0		—%——
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 0		—%——
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 0		—%——
The A2A agent card names an endpoint url
of 0		—%——
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 0		—%——
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 0		—%——
GPTBot refused where the browser got 2xx
of 5643		19.6%——
ClaudeBot refused where the browser got 2xx
of 5643		19.8%——
OAI-SearchBot refused where the browser got 2xx
of 5643		12.3%——
PerplexityBot refused where the browser got 2xx
of 5643		12.7%——

Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.

Signed JSON: /api/v1/census/weekly?week=2026-W40 · feed: Atom · method: /api/census. Editions: 2026-W41 · 2026-W40.