CrawlCheck
Data · census · weekly

Agentic web census, week 2026-W41

Sealed 2026-10-12. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.

top 1,000, daily

Run 10 (finished 2026-10-11), 991 sites, compared with run 3 (2026-10-04).

Biggest moves: GPTBot refused where the browser got 2xx +1 pt (6 gained, 1 lost); Serves a robots.txt (200, not HTML) +1 pt (5 gained, 0 lost); At least one AI identity is refused (non-2xx) where the browser got 2xx +0.6 pt (6 gained, 3 lost); Homepage answers a browser (2xx after redirects) +0.5 pt (16 gained, 11 lost); ClaudeBot refused where the browser got 2xx -0.4 pt (2 gained, 4 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 991		52%+0.5 pt+16 / −11
Serves a robots.txt (200, not HTML)
of 515		80.4%+1 pt+5 / −0
robots.txt names a sitemap
of 414		69.6%-0.3 pt+0 / −1
robots.txt carries a Content-Signal line
of 414		3.6%0 pt+0 / −0
Content-Signal says ai-train=no
of 414		2.2%0 pt+0 / −0
Content-Signal says ai-input=no or search=no
of 414		1%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 414		24.4%0 pt+0 / −0
robots.txt closes / to GPTBot
of 414		16.2%-0.3 pt+0 / −1
robots.txt closes / to OAI-SearchBot
of 414		8%0 pt+0 / −0
robots.txt closes / to ChatGPT-User
of 414		10.1%0 pt+0 / −0
robots.txt closes / to ClaudeBot
of 414		15.2%-0.3 pt+0 / −1
robots.txt closes / to Claude-SearchBot
of 414		8.5%0 pt+0 / −0
robots.txt closes / to PerplexityBot
of 414		11.8%-0.3 pt+0 / −1
robots.txt closes / to Google-Extended
of 414		13.3%-0.3 pt+0 / −1
robots.txt closes / to Applebot-Extended
of 414		11.6%-0.3 pt+0 / −1
robots.txt closes / to CCBot
of 414		19.3%0 pt+0 / −0
robots.txt closes / to Bytespider
of 414		18.6%0 pt+0 / −0
robots.txt closes / to Meta-ExternalAgent
of 414		15%0 pt+0 / −0
robots.txt closes / to Amazonbot
of 414		12.8%-0.3 pt+0 / −1
Serves /llms.txt
of 515		20.8%+0.4 pt+3 / −1
Serves /agents.md
of 515		2.7%0 pt+0 / −0
Publishes an agent-skills index
of 515		2.3%+0.2 pt+1 / −0
Publishes an MCP server card
of 515		2.3%0 pt+0 / −0
Publishes an API catalog (RFC 9727)
of 515		2.1%0 pt+0 / −0
Publishes an A2A agent card
of 515		0.4%0 pt+0 / −0
Declares any agent capability (any of the four above)
of 515		2.9%0 pt+0 / −0
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 515		18.4%+0.6 pt+6 / −3
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 515		4.1%+0.2 pt+2 / −1
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 515		10.9%——
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 515		1.4%——
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 56		69.6%——
The A2A agent card names an endpoint url
of 515		0.2%——
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 1		0%——
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 515		0.2%——
GPTBot refused where the browser got 2xx
of 515		16.1%+1 pt+6 / −1
ClaudeBot refused where the browser got 2xx
of 515		14.4%-0.4 pt+2 / −4
OAI-SearchBot refused where the browser got 2xx
of 515		10.3%+0.2 pt+2 / −1
PerplexityBot refused where the browser got 2xx
of 515		10.5%0 pt+2 / −2

top 10,000, weekly

Run 2 (finished 2026-10-10), 9776 sites, compared with run 1 (2026-10-03).

Biggest moves: Homepage answers a browser (2xx after redirects) -0.9 pt (151 gained, 235 lost); robots.txt closes / to CCBot -0.5 pt (3 gained, 26 lost); robots.txt closes / to Meta-ExternalAgent -0.5 pt (2 gained, 24 lost); OAI-SearchBot refused where the browser got 2xx +0.4 pt (43 gained, 22 lost); PerplexityBot refused where the browser got 2xx +0.3 pt (46 gained, 28 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 9776		56.9%-0.9 pt+151 / −235
Serves a robots.txt (200, not HTML)
of 5559		81.2%0 pt+34 / −33
robots.txt names a sitemap
of 4512		69.7%0 pt+7 / −6
robots.txt carries a Content-Signal line
of 4512		3.1%0 pt+2 / −0
Content-Signal says ai-train=no
of 4512		1.9%0 pt+2 / −1
Content-Signal says ai-input=no or search=no
of 4512		0.5%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 4512		18.8%0 pt+3 / −2
robots.txt closes / to GPTBot
of 4512		13.3%-0.1 pt+3 / −7
robots.txt closes / to OAI-SearchBot
of 4512		6.3%0 pt+2 / −4
robots.txt closes / to ChatGPT-User
of 4512		8.3%0 pt+3 / −4
robots.txt closes / to ClaudeBot
of 4512		12%-0.1 pt+3 / −7
robots.txt closes / to Claude-SearchBot
of 4512		5.8%0 pt+2 / −2
robots.txt closes / to PerplexityBot
of 4512		9.1%-0.1 pt+1 / −5
robots.txt closes / to Google-Extended
of 4512		10.2%-0.1 pt+3 / −8
robots.txt closes / to Applebot-Extended
of 4512		9.8%-0.1 pt+3 / −6
robots.txt closes / to CCBot
of 4512		14%-0.5 pt+3 / −26
robots.txt closes / to Bytespider
of 4512		13.5%0 pt+3 / −3
robots.txt closes / to Meta-ExternalAgent
of 4512		10.4%-0.5 pt+2 / −24
robots.txt closes / to Amazonbot
of 4512		9.9%-0.1 pt+2 / −5
Serves /llms.txt
of 5559		15.7%+0.3 pt+20 / −5
Serves /agents.md
of 5559		1.3%0 pt+3 / −2
Publishes an agent-skills index
of 5559		1%0 pt+3 / −2
Publishes an MCP server card
of 5559		1%0 pt+2 / −2
Publishes an API catalog (RFC 9727)
of 5559		1.2%+0.1 pt+4 / −1
Publishes an A2A agent card
of 5559		0.3%0 pt+2 / −1
Declares any agent capability (any of the four above)
of 5559		1.4%0 pt+2 / −2
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 5559		23.2%+0.1 pt+70 / −65
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 5559		2.6%+0.1 pt+19 / −14
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 5559		5.3%——
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 5559		0.9%——
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 295		77.6%——
The A2A agent card names an endpoint url
of 5559		0.1%——
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 8		37.5%——
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 5559		0.1%——
GPTBot refused where the browser got 2xx
of 5559		19.7%+0.1 pt+44 / −39
ClaudeBot refused where the browser got 2xx
of 5559		19.9%0 pt+49 / −47
OAI-SearchBot refused where the browser got 2xx
of 5559		12.8%+0.4 pt+43 / −22
PerplexityBot refused where the browser got 2xx
of 5559		13.2%+0.3 pt+46 / −28

Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.

Signed JSON: /api/v1/census/weekly?week=2026-W41 · feed: Atom · method: /api/census. Editions: 2026-W41 · 2026-W40.