CrawlCheck

AI crawlers · Answer engines

YouBot

YouBot is the answer-engine crawler run by You.com: You.com index. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.

User-agent tokenYouBot
OperatorYou.com
What it doesYou.com index
Kindanswer-engine crawler
Can a request be verified?No — user agent string only
In a CrawlCheck scanRead from robots.txt; a refusal is scored

Allow or block it in robots.txt

User-agent: YouBot
Disallow: /
User-agent: YouBot
Allow: /

Add a group that names it: "User-agent: YouBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names YouBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

Yes. It is how You.com reaches a page to answer a question or to build the index its answers read, so refusing it stops that engine reading the site.

See what YouBot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about YouBot

QHow do I block YouBot in robots.txt?
Add a group that names it: "User-agent: YouBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names YouBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
QHow can I tell if a request from YouBot is real?
Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.
QDoes blocking YouBot affect whether my site appears in AI answers?
Yes. It is how You.com reaches a page to answer a question or to build the index its answers read, so refusing it stops that engine reading the site.
QDoes CrawlCheck test my site as YouBot?
Not as a request. CrawlCheck reads the rules your robots.txt sets for YouBot and reports them. A refusal of it counts toward the grade, because it costs the site visibility.