CrawlCheck

AI crawlers · Training crawlers

CCBot

CCBot is the training crawler run by Common Crawl: Common Crawl. Its operator publishes the IP ranges it crawls from (https://index.commoncrawl.org/ccbot.json), so a request carrying the name can be checked against that list.

User-agent tokenCCBot
OperatorCommon Crawl
What it doesCommon Crawl
Kindtraining crawler
Can a request be verified?Yes — published ip feed
Verification sourcehttps://index.commoncrawl.org/ccbot.json
In a CrawlCheck scanSent as a request on every scan; reported, never scored

Allow or block it in robots.txt

User-agent: CCBot
Disallow: /
User-agent: CCBot
Allow: /

Add a group that names it: "User-agent: CCBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names CCBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

See what CCBot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about CCBot

QHow do I block CCBot in robots.txt?
Add a group that names it: "User-agent: CCBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names CCBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
QHow can I tell if a request from CCBot is real?
Its operator publishes the IP ranges it crawls from (https://index.commoncrawl.org/ccbot.json), so a request carrying the name can be checked against that list.
QDoes blocking CCBot affect whether my site appears in AI answers?
No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.
QDoes CrawlCheck test my site as CCBot?
Yes. Every CrawlCheck scan fetches the homepage as CCBot and compares the response with what a browser receives. Rules for it are reported and never scored, because refusing it is a policy choice.