AI crawlers · Training crawlers
SemrushBot-OCOB
SemrushBot-OCOB is the training crawler run by Semrush: Semrush AI corpus. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.
| User-agent token | SemrushBot-OCOB |
| Operator | Semrush |
| What it does | Semrush AI corpus |
| Kind | training crawler |
| Can a request be verified? | No — user agent string only |
| In a CrawlCheck scan | Read from robots.txt; reported, never scored |
Allow or block it in robots.txt
User-agent: SemrushBot-OCOB Disallow: /
User-agent: SemrushBot-OCOB Allow: /
Add a group that names it: "User-agent: SemrushBot-OCOB" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names SemrushBot-OCOB, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
What blocking it changes
No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.
Other agents from Semrush
SemrushBot— Semrush crawler
See what SemrushBot-OCOB and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.