AI crawlers · Search indexes

`Yandex`

Yandex is the search index crawler run by Yandex: Yandex (umbrella token). Its operator documents how to verify it: reverse DNS to yandex.ru, yandex.net or yandex.com, then forward DNS back to the same address. A reverse DNS lookup on the source address must return the operator's domain, and a forward lookup on that name must return the same address.

User-agent token	`Yandex`
Operator	Yandex
What it does	Yandex (umbrella token)
Kind	search index crawler
Can a request be verified?	Yes — reverse dns
Verification source	reverse DNS to yandex.ru, yandex.net or yandex.com, then forward DNS back to the same address
In a CrawlCheck scan	Read from robots.txt; a refusal is scored

Allow or block it in robots.txt

User-agent: Yandex
Disallow: /

User-agent: Yandex
Allow: /

Add a group that names it: "User-agent: Yandex" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Yandex, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

Indirectly. It feeds Yandex (umbrella token). A refusal removes pages from that index, and most AI answer surfaces still draw on search indexes.

Other agents from Yandex

YandexBot — indexes pages for Yandex search

See what Yandex and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Yandex

QHow do I block Yandex in robots.txt?

QHow can I tell if a request from Yandex is real?

Its operator documents how to verify it: reverse DNS to yandex.ru, yandex.net or yandex.com, then forward DNS back to the same address. A reverse DNS lookup on the source address must return the operator's domain, and a forward lookup on that name must return the same address.

QDoes blocking Yandex affect whether my site appears in AI answers?

Indirectly. It feeds Yandex (umbrella token). A refusal removes pages from that index, and most AI answer surfaces still draw on search indexes.

QDoes CrawlCheck test my site as Yandex?

Not as a request. CrawlCheck reads the rules your robots.txt sets for Yandex and reports them. A refusal of it counts toward the grade, because it costs the site visibility.