CCBot
The Common Crawl crawler. Its open corpus is downloaded and used by many model builders, so a page can enter training sets without ever being fetched by any model operator's own crawler.
What CCBot means
That is the case that breaks the sentence we blocked GPTBot, so we are out of the training data: the copy that trained the model may have been taken by a nonprofit archive years earlier.
What CCBot does not show
CCBot honours robots.txt and publishes its identity, so exclusion is possible; it is only retroactive removal that is not.
Where CCBot comes up on this site
- Content-Signal in robots.txt: search, ai-input, ai-trainIf you refuse training, back it with a Disallow: / group for the crawlers whose operators say they train, such as GPTBot and CCBot, plus Google's Google-Extended token, and make sure those groups do not drop your other rules, as explained in how named…
- Google-Extended and Applebot-Extended: block AI trainingGPTBot, ClaudeBot, CCBot and the rest each need their own group; how often top sites block each one .
- Google-Extended has no user agent. 1,582 requests used itCCBot and Claude-SearchBot are in the same position from where we stand: never verified once, across sixteen sites and seven days.
- Linked from every page, our llms.txt was fetched 8 timesHow often each kind of URL was fetched The sixteen names counted are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Amazonbot,…
- State of AI visibility, October 2026: 139,788 homepagesGPTBot is the most blocked at 3.7%, followed by CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%).
Terms this definition uses
Corpus · training data · robots.txt
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about CCBot
What is CCBot?
The Common Crawl crawler. Its open corpus is downloaded and used by many model builders, so a page can enter training sets without ever being fetched by any model operator's own crawler.
What does CCBot not show or guarantee?
Its open corpus is downloaded and used by many model builders, so a page can enter training sets without ever being fetched by any model operator's own crawler. CCBot honours robots.txt and publishes its identity, so exclusion is possible; it is only retroactive removal that is not.
Which area of the glossary does CCBot belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← PerplexityBot vs Perplexity-User · Bytespider →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.