Guides · 2026-10-26 · By VSNARY | Emmanuel Orta
Website crawlers in 2026: who crawls your site, how to verify them, and which tool to check with
Search engines, AI crawlers and the audit tools you run yourself all fetch your pages. What each kind is for, how often the top 10,000 sites block the AI ones, how to tell a real crawler from an impostor, and four tools compared.
Three kinds of crawler reach a site in 2026: search index crawlers such as Googlebot and Bingbot, AI crawlers (training crawlers like GPTBot and CCBot, AI search crawlers like OAI-SearchBot, and fetchers that act for a person, like ChatGPT-User), and the audit crawlers you run yourself. A user-agent string is only a claim: a real crawler is confirmed by checking its source IP against the ranges its operator publishes. In CrawlCheck's October 2026 census of the top 10,000 sites, 14.3% of sites serving a robots.txt closed it to GPTBot, but 20.0% of sites refused a client declaring GPTBot at the door.
Every site is read by more machines than people. Some build a search index, some collect text for AI models, some fetch a page because a person asked an assistant about it, and some are audit tools you point at your own site. They behave differently, they are controlled differently, and a few of them are not crawlers at all. This guide covers all of them, with figures measured across the top 10,000 sites and a comparison of four tools that check your site.
Disclosure: CrawlCheck publishes this guide and is one of the four tools compared in it. Every figure about another company comes from that company’s own pages, linked where it appears, read on 26 October 2026. Where CrawlCheck does less than another tool, the table says so.
Who crawls your site #
| Kind | Examples (operator) | What it is for | Can a request be verified? |
|---|---|---|---|
| Search index crawler | Googlebot (Google), Bingbot (Microsoft) | Builds the search index | Yes: published IP ranges and reverse DNS |
| AI training crawler | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) | Collects pages that may be used to train models | GPTBot and ClaudeBot: published IP ranges |
| AI search crawler | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity) | Indexes pages an assistant can cite in answers | Yes: published IP ranges |
| Fetcher acting for a person | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity) | Fetches a page because a user asked about it | Yes: published IP ranges |
| Control token, not a crawler | Google-Extended (Google), Applebot-Extended (Apple) | A robots.txt name that decides how already-crawled pages may be used for AI | Nothing to verify: no request ever carries it |
| Crawler with no published ranges | Bytespider (ByteDance) | Collects pages for its operator | No: the name in the user agent cannot be checked |
| Audit crawler you run | Ahrefs Site Audit, Screaming Frog SEO Spider | Finds technical problems on your own site | It is your own crawl |
CrawlCheck’s crawler catalogue lists 114 named agents with the verification method for each, free as JSON. Two at a time: crawlers compared.
How often the top 10,000 sites block AI crawlers #
CrawlCheck’s weekly census reads the robots.txt of the Tranco top 10,000 and also requests each homepage as a browser and as named AI crawlers. In the run that finished on 24 October 2026, 5,553 homepages answered a browser and 4,501 of those sites served a robots.txt.
| Crawler | robots.txt closes the homepage (of 4,501 sites with a robots.txt) | Refused at the door (of 5,553 sites that answered a browser) |
|---|---|---|
| GPTBot | 644 (14.3%) | 1,108 (20.0%) |
| ClaudeBot | 591 (13.1%) | 1,124 (20.2%) |
| PerplexityBot | 409 (9.1%) | 733 (13.2%) |
| OAI-SearchBot | 280 (6.2%) | 702 (12.6%) |
| CCBot | 679 (15.1%) | not requested |
| Google-Extended | 516 (11.5%) | a control token: never requested |
More sites refuse an AI crawler at the door than say so in robots.txt: a firewall or bot rule answers before robots.txt is ever read. “Refused” here means a non-2xx answer to a client declaring that crawler’s name, sent from CrawlCheck’s own addresses, where the browser got 2xx; the real crawler, coming from its operator’s addresses, may be treated differently. Every crawler on one page: the radar.
How to tell a real crawler from an impostor #
Anyone can send User-agent: GPTBot. The name is a claim; the source address is the evidence. Operators that publish their ranges let you check it: compare the request’s IP with the operator’s file. CrawlCheck reads 17 of those files every day and keeps every change at /data/ip-ranges. Google and Microsoft also support reverse DNS: the IP resolves to a hostname under the operator’s domain, and that hostname resolves back to the same IP. A crawler whose operator publishes neither, such as Bytespider, cannot be verified at all. More: how to verify Googlebot, Bingbot and AI crawlers.
How to control them #
robots.txt is read per crawler: each crawler obeys the most specific User-agent group that names it and ignores * entirely. That catches most sites out. Name GPTBot with only Allow: / and every Disallow rule in your * group stops applying to it. Repeat the rules you want it to keep in its own group. Control tokens work the same way in robots.txt but change only how content is used, not what is crawled: see Google-Extended and Applebot-Extended.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/That example keeps OpenAI’s training crawler out while letting its search crawler index the site, and keeps /account/ closed to the search crawler because naming it dropped the * rules.
Four tools that check your site, compared #
| Ahrefs Site Audit | Screaming Frog SEO Spider | aicrawlercheck | CrawlCheck | |
|---|---|---|---|---|
| Built for | Technical SEO audits of a whole site in the cloud | Deep technical SEO crawls on your own computer | Whether AI bots are allowed to read a site | What AI crawlers are actually served, and whether that matches what the site declares |
| Pages per audit | Whole sites; the vendor states up to 170k URLs a minute | 500 URLs free; more with a licence | One URL, or 20 in its batch checker | The homepage and machine files, plus sitemap sampling; not a deep crawl |
| How it reads your site | As its own audit crawler | As one user agent you choose per crawl, with optional JavaScript rendering | Reads robots.txt, meta robots, X-Robots-Tag and llms.txt, then evaluates each bot’s access | Fetches as a browser and as 12 named crawler identities in one scan, and compares what each received |
| Crawler coverage | 170+ technical SEO issues; its crawler guide does not cover AI crawlers | Any user agent you type; its pricing page lists crawling with OpenAI and Gemini | 265 bots in its directory; 42 checks scored out of 100 | 114 agents in its catalogue; 17 operator IP files read daily to verify crawler hits |
| Top-sites research | — | — | Scout Radar reads robots.txt across the Tranco top 10,000 | The radar compares robots.txt with what each homepage serves, top 10,000 weekly, top 1,000 daily |
| Price | Part of Ahrefs plans; free for your own verified sites in Ahrefs Free (formerly Webmaster Tools), 5,000 crawl credits a month per project | Free to 500 URLs; £199 a year per licence | Free | Scan free with no account; paid plans for monitoring |
Sources: Ahrefs, “Crawl Me Maybe? How Website Crawlers Work”; Screaming Frog SEO Spider pricing; Ahrefs Free; aicrawlercheck.com. Read on 26 October 2026.
Which to use #
For a deep technical crawl of every page, use Ahrefs Site Audit or Screaming Frog: CrawlCheck does not crawl hundreds of thousands of pages and is not trying to. For a quick read of what your robots.txt allows each AI bot, aicrawlercheck and CrawlCheck both answer it free, and aicrawlercheck’s directory names more bots. To find out whether AI crawlers are actually served what your robots.txt promises, which is where most sites in the census go wrong, run a CrawlCheck scan: it requests the page as each crawler and shows you the difference.
The main product
Found this on your own site? We fix it for $500.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Questions this post answers
What are the main types of website crawlers?
Search index crawlers such as Googlebot and Bingbot; AI crawlers, which split into training crawlers (GPTBot, ClaudeBot, CCBot), AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and fetchers acting for a person (ChatGPT-User, Claude-User); and audit crawlers you run on your own site, such as Ahrefs Site Audit or Screaming Frog.
How do I know if a crawler visiting my site is real?
Check the request's source IP against the IP ranges the operator publishes, or for Google and Microsoft use reverse DNS that resolves back to the same IP. The user-agent name alone proves nothing. A crawler whose operator publishes neither cannot be verified.
Can Screaming Frog or Ahrefs check AI crawler access?
Screaming Frog can crawl as any user agent you set, one per crawl. Ahrefs Site Audit checks technical SEO issues as its own crawler. Neither compares what several AI crawlers are served against a browser in one scan, which is what CrawlCheck does.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.