AI crawlers · SEO and research crawlers
AhrefsBot
AhrefsBot is the tool or developer fetch run by Ahrefs: Ahrefs link index. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.
| User-agent token | AhrefsBot |
| Operator | Ahrefs |
| What it does | Ahrefs link index |
| Kind | tool or developer fetch |
| Can a request be verified? | No — user agent string only |
| In a CrawlCheck scan | Read from robots.txt; reported, never scored |
Allow or block it in robots.txt
User-agent: AhrefsBot Disallow: /
User-agent: AhrefsBot Allow: /
Add a group that names it: "User-agent: AhrefsBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names AhrefsBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
What blocking it changes
No. It is a third-party tool's crawler. Blocking it changes what that tool knows about your site, not anyone's AI answers.
What it actually did on crawlcheck.io
| Verified requests | 10,912 |
| Requests using its name from outside its operator's ranges | 3 (0% of claims) |
| robots.txt fetched | 212 times |
| Fetches of paths robots.txt disallows | 12 |
| Observed | 2026-08-14 to 2026-11-12 |
Measured on this site’s own traffic, verified against the operator’s published ranges. Every agent’s record: agent conduct.
See what AhrefsBot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.