CrawlCheck

Directory · Site crawler

Siteliner — what a crawler receives from siteliner.com

Website measurement, not a product review. This page records what machine clients received from siteliner.com. It does not evaluate Siteliner software, features, pricing or support.

53
Fgrade

Answer engines are being served something other than this site.

AI visibility 53/100, measured 2026-10-18 18:09 UTC from outside its network, as each named AI and search crawler and as a browser. An answer engine cannot use this site as it is served today.

Held at F A critical-severity finding sets the letter.

Open the full report siteliner.com ↗

531. ReachCan a named answer-engine crawler get your pages at all?532. ReadOnce it has the bytes, can it find the words?523. QuoteIs there a specific fact it can state and attribute?

9

findings

✗

llms.txt on its host

✗

entity map on its host

24

scans on record since 2026-09-01

The three stages, in the order an engine hits a site

Each stage gates the next: an engine that cannot reach a page never reads it, and one that cannot read it never quotes it.

53

Stage 1 · 40% of the score

Reach

Can a named answer-engine crawler get your pages at all?

At least one answer engine was refused, challenged or served less than a browser.

53

Stage 2 · 30% of the score

Read

Once it has the bytes, can it find the words?

The content is buried in markup, or the files that guide a crawler are missing.

52

Stage 3 · 30% of the score

Quote

Is there a specific fact it can state and attribute?

An assistant would have to paraphrase your page instead of quoting a fact.

The stage to fix first is quote, because the three run in order.

What to fix

9 findings on this site, 7 of them scored serious. Every one is named below; the top one is shown in full, free. What we saw, why each matters, how to fix it and the bytes behind it are what a licence buys — or we implement the whole list for you.

4 Critical 3 Medium 2 Note

1robots.txt is HTML, not texttext/html; charset=utf-8, 84,906 bytes.CriticalSetting

What we saw

text/html; charset=utf-8, 84,906 bytes. /robots.txt

Why it matters

robots.txt must be plain text. Served as HTML it cannot be parsed, so no crawl directives and no sitemap reference are read.

How to fix it

robots.txt is returning an HTML page. Serve a real text file at /robots.txt, and check that no catch-all route is answering instead.

Setting: a setting on the server, CDN, DNS or a plugin — no code and no writing

Paste this

A minimal valid file. Serve it at https://siteliner.com/robots.txt as text/plain with a 200, and let it through any bot or firewall rule.

User-agent: *
Disallow:

Sitemap: https://siteliner.com/sitemap.xml

The whole corrected robots.txt for siteliner.com · built from the live file, every change listed at the top.

Fix it with a coding agent

Paste into Claude Code, Cursor or Codex, opened on the code that serves this site.

Show prompt
You are working in the codebase (or server/CDN configuration) that serves siteliner.com. A crawler audit found the problem below. Fix it at the source, keep the change as small as possible, and do not change anything a visitor sees unless the fix requires it.

Finding: robots.txt is HTML, not text (ROBOTS_IS_HTML, critical severity)
Page: https://siteliner.com/robots.txt
What was measured: text/html; charset=utf-8, 84,906 bytes.
Why it matters: robots.txt must be plain text. Served as HTML it cannot be parsed, so no crawl directives and no sitemap reference are read.
How to fix it: robots.txt is returning an HTML page. Serve a real text file at /robots.txt, and check that no catch-all route is answering instead.

Rule and examples: https://crawlcheck.io/rules/ROBOTS_IS_HTML
Full report (markdown): https://crawlcheck.io/r/khf0qyynkq.md

Evidence (bytes the crawler received):
```

<HTML>
	
	<HEAD>        
                    <BASE href="https://www.siteliner.com/" />
            <META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
            <LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D413F9" />
            <link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC10" />
  	    <script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
            <SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.j
```

When you are done:
1. Tell me which files or settings you changed and why.
2. If the fix is a server, CDN or cache setting you cannot change from the code, say exactly where it is set.
3. After deploying, re-scan at https://crawlcheck.io/?url=siteliner.com . The fix counts when ROBOTS_IS_HTML is gone from the new report.
Why this was decided
RuleROBOTS_IS_HTML · rule pack sv32 · scope core
Evaluated2026-10-18 18:09:44 UTC
Decisiond1:bf50507c5efd98fca765 · finding f1:5464eea0f7bfc1b1f23e resolve either at /api/graph/<id> with a key
Observercrawlcheck-worker build b3cf8c17 · audit v4

What it rests on

IdentityStatusBytessha256Kept
read: /robots.txt20084,90686aa54c0dd38write_unconfirmed

21 more in the manifest.

Artifacts: manifest:d7e8a08ca00871268c2f78030d6023980d7b718a4bc1bd1565f0386639f9303c

Evidence bytes

<HTML>
	
	<HEAD>        
                    <BASE href="https://www.siteliner.com/" />
            <META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
            <LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D413F9" />
            <link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC10" />
  	    <script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
            <SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.j

Grade effect Caps the letter at F together with 3 other critical-severity findings; the letter rises only when all of them are gone. Moves no points.

Rule ROBOTS_IS_HTML under scoring rules v32 · fingerprint ROBOTS_IS_HTML|/robots.txt · persisting (28 scans)

Verify the fix Re-scan after the change. This finding is cleared when ROBOTS_IS_HTML is no longer reported at /robots.txt; if it comes back after clearing, it is marked a regression, not a new finding.

ROBOTS_IS_HTML · full transcript

2The sitemap returns HTML, not XML/sitemap.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Critical
3The sitemap returns HTML, not XML/sitemap_index.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Critical
4The sitemap returns HTML, not XML/wp-sitemap.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Critical
5www and the apex both serve the site instead of one redirecting🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Medium
6robots.txt declares no sitemap/robots.txt🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Medium
7Almost nothing a machine receives from this page is readable text🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Medium
8No llms.txt/llms.txt🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Note
9HTTPS is served without a Strict-Transport-Security header🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Note

Open the other 8

What we saw, why it matters, how to fix it, and the evidence bytes for every finding — on this report and every re-scan. From $29 a month.

Unlock the findings

Or have them fixed

We implement the list on your site and re-measure, with a sealed before and after. One site, $749 once.

Get this fixed — $749

The chain a crawler follows

The chain breaks at robots.txt. Everything after that point is only reachable by a crawler guessing the conventional path.

✗robots.txtEvery crawler reads this first
✗sitemapNamed in robots.txt and resolves
✗llms.txtPoints onward, links stay on this host
✗entity graphParses as JSON and points home

A machine file that does not parse is worth less than one that is absent, because it looks present. Rows behind this.

Where the bytes go

Of the 87,524 decompressed bytes the homepage delivers, 0.4% is text a reader or a model can actually use. Most of what a crawler downloads here is not words.

  • Readable text 0.4% · 312 B
  • Markup & attributes 97.3% · 85,124 B
  • Inline JavaScript 2.4% · 2,088 B

What changed

  1. 2026-10-02score moved 48 → 47score-down
  2. 2026-10-02resolved: the machine files cost more than the pageresolved
  3. 2026-09-16a new finding appeared: the machine files cost more than the pageappeared
  4. 2026-09-05score moved 53 → 51score-down
  5. 2026-09-03a new finding appeared: https is served without a strict-transport-security headerappeared
  6. 2026-09-03a new finding appeared: www and the apex both serve the site instead of one redirecting againreturned
  7. 2026-09-02resolved: www and the apex both serve the site instead of one redirectingresolved

24 scans on record, first 2026-09-01. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.

Is this your site?

This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.

Claim the record for siteliner.com

Is this your company?

This listing came from a fetch, not a submission, so nobody was asked. If siteliner.com is yours you can prove it in a minute and for nothing — a DNS TXT record or one meta tag on the home page, both checkable by anyone.

Claiming marks you as the verified owner and lets you correct the category and the company name. It moves no score, no rank and no position.

Claim this listing — free or ask about a verified profile

Dispute or re-scan

If a reading here is wrong, it is our instrument that is wrong, and we want to know. Tell us what is wrong and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.

By email instead: hello@crawlcheck.io. Category “Site crawler” is our label for what the product is primarily sold as; it is not scored. Back to the directory.