CrawlCheck

AI crawlers · Training crawlers

Omgilibot

Omgilibot is the training crawler run by Webz.io: collects pages for Webz.io's web data feeds, which are sold for AI training. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.

User-agent tokenOmgilibot
OperatorWebz.io
What it doescollects pages for Webz.io's web data feeds, which are sold for AI training
Kindtraining crawler
Can a request be verified?No — user agent string only
In a CrawlCheck scanRead from robots.txt; reported, never scored

Allow or block it in robots.txt

User-agent: Omgilibot
Disallow: /
User-agent: Omgilibot
Allow: /

Add a group that names it: "User-agent: Omgilibot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Omgilibot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

Other agents from Webz.io

See what Omgilibot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Omgilibot

QHow do I block Omgilibot in robots.txt?
Add a group that names it: "User-agent: Omgilibot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Omgilibot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
QHow can I tell if a request from Omgilibot is real?
Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.
QDoes blocking Omgilibot affect whether my site appears in AI answers?
No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.
QDoes CrawlCheck test my site as Omgilibot?
Not as a request. CrawlCheck reads the rules your robots.txt sets for Omgilibot and reports them. Rules for it are reported and never scored, because refusing it is a policy choice.