Directory · Site crawler
Seobility — what a crawler receives from seobility.net
Answer engines can technically reach this site, but get little they can use.
AI visibility 73/100, measured 2026-09-02 08:17 UTC from outside its network, as fifteen named crawler identities and one unnamed client. An engine can technically reach this site and would struggle to get anything quotable out of it.
Held at C A medium-severity finding sets the letter.
4
findings
✓
llms.txt on its host
✗
entity map on its host
3
scans on record since 2026-08-18
The three stages, in the order an engine hits a site
Each stage gates the next. An engine that cannot reach a page never reads it; one that cannot read it never quotes it. That is why reach is a ceiling and not just a weight.
Stage 1 · 40% of the score
Reach
Can a named answer-engine crawler get your pages at all?
At least one answer engine was refused, challenged or served less than a browser.
Stage 2 · 30% of the score
Read
Once it has the bytes, can it find the words?
The content is buried in markup, or the files that guide a crawler are missing.
Stage 3 · 30% of the score
Quote
Is there a specific fact it can state and attribute?
An assistant would have to paraphrase your page instead of quoting a fact.
The stage to fix first is read, because the three run in order.
What to fix, in order
Findings first: each one holds the grade down until it is gone (the top one sets the letter). Then headroom: each one lifts the score by the points shown. Open a card for what we saw, why it matters and how to fix it.
Files, generated from this scan
Each file says at its top exactly what was changed or filled and what was left as a TODO. Publish, then re-scan: done is measured.
1www and the apex both serve the site instead of one redirectingwww.seobility.net answers 200 and stays there: 127,268 bytes against 127,268 on the apex. Every page exists at two addresses.Medium
What we saw
www.seobility.net answers 200 and stays there: 127,268 bytes against 127,268 on the apex. Every page exists at two addresses.
Why it matters
Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page.
How to fix it
Pick one host (apex or www) and redirect the other to it with a 301 so there is one copy of every page.
HOST_DUPLICATE_200 · full transcript
2robots.txt rules do not apply to named agents58 named User-agent groups (58 agents) declare no Disallow rules while the `*` group carries 16. A crawler obeys only its own most-specific group and ignores `*Medium
What we saw
58 named User-agent groups (58 agents) declare no Disallow rules while the `*` group carries 16. A crawler obeys only its own most-specific group and ignores `*` entirely, so those rules do not apply to any of the named agents. Worst group: `googlebot` bypasses /wp-admin/, /*/keywordcheck/check/*, /*/keywordchecker/check/*, /*/seocompare/check/*, /*/comparador-web-seo/check/*, /*/rankingcheck/check/*, /*/serpchecker/check/*, /*/seocheck/pdfexport*, /*/seochecker/pdfexport*, /*/seocheck/check*, /*/seochecker/check*, /en/keyword-research-tool/check, /de/keyword-recherche-tool/check, /es/buscador-palabras-clave/check, /pdfexample/, /docs/ai/. This includes a SEARCH crawler (googlebot) — the bypassed paths are being crawled and can enter the index. /robots.txt
Why it matters
A crawler reads only the User-agent group that matches it best and ignores every other group, including `*`. Naming an agent and giving it only `Allow: /` therefore deletes all of your `*` Disallow rules for that agent. The file still parses and the agent still reaches your homepage, so this is invisible on inspection — but the paths you meant to keep out of search are open, and search engines will crawl and may index them.
How to fix it
Every User-agent group that names a crawler must repeat the Disallow rules you want applied. A named group with no rules means 'allow everything' for that crawler.
ROBOTS_RULES_SHADOWED · full transcript
3An answer engine's crawler was refused while a browser was served (unconfirmed from this vantage)OAI-SearchBot 429 (a Cloudflare managed challenge), PerplexityBot 429 (a Cloudflare managed challenge), Googlebot 429 (a Cloudflare managed challenge), Bingbot Low
What we saw
OAI-SearchBot 429 (a Cloudflare managed challenge), PerplexityBot 429 (a Cloudflare managed challenge), Googlebot 429 (a Cloudflare managed challenge), Bingbot 429 (a Cloudflare managed challenge), Applebot 429 (a Cloudflare managed challenge) — while an unnamed client got 200 with 1299 words. Not treated as a block: the mobile-browser identity was refused as well (HTTP 429), so this origin is refusing this client, not answer engines in particular. Confirm from your own address with the extension, or in the access logs against the operator's published ranges
Why it matters
The same URL, the same minute: an ordinary client was served the page and a named answer-engine crawler was not. That is the edge rather than a robots.txt decision. One caveat this site's own guides state and this finding must too: the scan sent the crawler's name from an address that operator does not publish, and an edge that verifies identity by IP is right to refuse it. So this proves the edge treats the identity differently - not that it refuses the real crawler. Your own logs, checked against the operator's published ranges, settle which. If the edge is refusing the real crawler, nothing else in this report reaches that engine, however good it is.
How to fix it
Allow the named answer-engine crawlers through the security layer (Cloudflare bot rules, WAF, rate limits), or verify their published IP ranges instead of blocking by name.
ANSWER_ENGINE_REFUSED · full transcript
4Visitors and crawlers are being served an old copy of this pagethe copy served to us was built 6 hours 5 min agoLow
What we saw
the copy served to us was built 6 hours 5 min ago
Why it matters
The page handed to us carried a cache age older than an hour. Nothing is broken today — the copy matches what the origin produces. It matters the moment you change something: an edit, a new price, a corrected phone number stays invisible to every visitor and every crawler for about that long, and nothing in your dashboard says so.
How to fix it
Lower the cache lifetime on HTML (an hour or less), or purge the edge and origin caches after every publish so crawlers see the current copy.
STALE_CACHE_SERVED · full transcript
Headroom +25 points if every row passes
Nothing here is broken. Each row is a measure that currently fails its optimal range, and what fixing it is worth to the score.
Watch this domain
Twice-daily scans, the date each finding first appeared, and a change receipt when something moves. Free for 30 days, no card.
Start watching seobility.netGet this fixed for you
Every finding above fixed on your site, then re-measured - done is measured, not asserted. One-time, from $750.
See the fix serviceThe chain a crawler follows
The chain breaks at entity graph. Everything after that point is only reachable by a crawler guessing the conventional path.
A machine file that does not parse is worth less than one that is absent, because it looks present. Rows behind this.
Where the bytes go
Of the 127,203 decompressed bytes the homepage delivers, 6.9% is text a reader or a model can actually use. Most of what a crawler downloads here is not words.
- Readable text 6.9% · 8,821 B
- Markup & attributes 49.7% · 63,210 B
- Structured data (JSON-LD) 1.1% · 1,460 B
- Inline CSS 14.9% · 18,906 B
- Inline JavaScript 14.4% · 18,360 B
- Inline SVG 8.8% · 11,234 B
- HTML comments 4.1% · 5,212 B
What changed
- 2026-09-02score moved 83 → 75score-down
- 2026-09-02the homepage no longer read as mostly markupresolved
- 2026-09-02a stale edge copy was served againreturned
- 2026-09-02an answer engine was refusedappeared
- 2026-09-01the stale edge copy clearedresolved
- 2026-09-01resolved: the cache is still serving a file the origin no longer hasresolved
- 2026-09-01a named-agent group shadowed the wildcard rulesappeared
- 2026-09-01a new finding appeared: www and the apex both serve the site instead of one redirectingappeared
3 scans on record, first 2026-08-18. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.
Is this your site?
This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.
Claim the record for seobility.netDispute or re-scan
If a reading here is wrong, it is our instrument that is wrong, and we want to know. Email hello@crawlcheck.io and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.
Category “Site crawler” is our label for what the product is primarily sold as; it is not scored. Back to the directory.