Directory · Site crawler
Siteliner — what a crawler receives from siteliner.com
Answer engines are being served something other than this site.
AI visibility 44/100, measured 2026-09-02 09:17 UTC from outside its network, as fifteen named crawler identities and one unnamed client. An answer engine cannot use this site. Whatever else is true of the marketing, the machine layer is shut.
Held at F A critical-severity finding sets the letter.
7
findings
✗
llms.txt on its host
✗
entity map on its host
7
scans on record since 2026-08-18
The three stages, in the order an engine hits a site
Each stage gates the next. An engine that cannot reach a page never reads it; one that cannot read it never quotes it. That is why reach is a ceiling and not just a weight.
Stage 1 · 40% of the score
Reach
Can a named answer-engine crawler get your pages at all?
At least one answer engine was refused, challenged or served less than a browser.
Stage 2 · 30% of the score
Read
Once it has the bytes, can it find the words?
The content is buried in markup, or the files that guide a crawler are missing.
Stage 3 · 30% of the score
Quote
Is there a specific fact it can state and attribute?
An assistant would have to paraphrase your page instead of quoting a fact.
The stage to fix first is read, because the three run in order.
What to fix, in order
Findings first: each one holds the grade down until it is gone (the top one sets the letter). Then headroom: each one lifts the score by the points shown. Open a card for what we saw, why it matters and how to fix it.
Files, generated from this scan
Each file says at its top exactly what was changed or filled and what was left as a TODO. Publish, then re-scan: done is measured.
1robots.txt is HTML, not texttext/html; charset=utf-8, 82,745 bytes.Critical
What we saw
text/html; charset=utf-8, 82,745 bytes. /robots.txt
Why it matters
robots.txt must be plain text. Served as HTML it cannot be parsed, so no crawl directives and no sitemap reference are read.
How to fix it
robots.txt is returning an HTML page. Serve a real text file at /robots.txt, and check that no catch-all route is answering instead.
Evidence bytes
<HTML>
<HEAD>
<BASE href="https://www.siteliner.com/" />
<META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D41402" />
<link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC25" />
<script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
<SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.jROBOTS_IS_HTML · full transcript
2The sitemap returns HTML, not XMLtext/html; charset=utf-8, 87,863 bytes.Critical
What we saw
text/html; charset=utf-8, 87,863 bytes. /sitemap.xml
Why it matters
A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered.
How to fix it
The sitemap URL is returning an HTML page. Serve XML at that path, or update robots.txt to point at the sitemap that actually exists.
Evidence bytes
<HTML>
<HEAD>
<BASE href="https://www.siteliner.com/" />
<META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D413F9" />
<link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC10" />
<script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
<SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.jSITEMAP_IS_HTML · full transcript
3The sitemap returns HTML, not XMLtext/html; charset=utf-8, 87,143 bytes.Critical
What we saw
text/html; charset=utf-8, 87,143 bytes. /sitemap_index.xml
Why it matters
A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered.
How to fix it
The sitemap URL is returning an HTML page. Serve XML at that path, or update robots.txt to point at the sitemap that actually exists.
Evidence bytes
<HTML>
<HEAD>
<BASE href="https://www.siteliner.com/" />
<META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D413F9" />
<link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC10" />
<script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
<SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.jSITEMAP_IS_HTML · full transcript
4The sitemap returns HTML, not XMLtext/html; charset=utf-8, 85,979 bytes.Critical
What we saw
text/html; charset=utf-8, 85,979 bytes. /wp-sitemap.xml
Why it matters
A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered.
How to fix it
The sitemap URL is returning an HTML page. Serve XML at that path, or update robots.txt to point at the sitemap that actually exists.
Evidence bytes
<HTML>
<HEAD>
<BASE href="https://www.siteliner.com/" />
<META http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<LINK rel="stylesheet" type="text/css" href="https://www.siteliner.com/static/sat_premium.css?v=D413F9" />
<link rel="shortcut icon" href="https://www.siteliner.com/static/images/siteliner_favicon.ico?v=08DC10" />
<script language="javascript" type="text/javascript" src="https://www.flotcharts.org/flot/excanvas.min.js"></script>
<SCRIPT src="https://code.jquery.com/jquery-1.8.0.min.jSITEMAP_IS_HTML · full transcript
5robots.txt declares no sitemapNo Sitemap: directive present.Medium
What we saw
No Sitemap: directive present. /robots.txt
Why it matters
Adding one Sitemap: line lets crawlers enumerate the site instead of guessing at it.
How to fix it
Add a Sitemap: line to robots.txt pointing at the full sitemap URL, so crawlers find it without guessing.
ROBOTS_NO_SITEMAP · full transcript
6Almost nothing a machine receives from this page is readable text0.3% of 89,533 decompressed bytes is visible textMedium
What we saw
0.3% of 89,533 decompressed bytes is visible text
Why it matters
A crawler pays for every byte it fetches and can only quote the text. Under 5% readable content means an answer engine downloads the whole page and comes away with almost nothing it can use — the difference between being quotable and being skipped. The content-ratio row above passes at 20%, so without this a page at 2% and a page at 19% were scored identically.
How to fix it
Move inline CSS and JavaScript into external files, drop unused page-builder styling, and make sure the words a visitor reads are in the HTML itself.
PAGE_IS_MOSTLY_CODE · full transcript
7No llms.txtNot present.Note
What we saw
Not present. /llms.txt
Why it matters
Not a defect. llms.txt is an emerging convention for telling AI systems what a site is and which pages matter.
How to fix it
Publish /llms.txt: a short Markdown file naming the business, what it does, and the pages worth reading, served as text/plain or text/markdown.
NO_LLMS_TXT · full transcript
Headroom +47 points if every row passes
Nothing here is broken. Each row is a measure that currently fails its optimal range, and what fixing it is worth to the score.
Watch this domain
Twice-daily scans, the date each finding first appeared, and a change receipt when something moves. Free for 30 days, no card.
Start watching siteliner.comGet this fixed for you
Every finding above fixed on your site, then re-measured - done is measured, not asserted. One-time, from $750.
See the fix serviceThe chain a crawler follows
The chain breaks at robots.txt. Everything after that point is only reachable by a crawler guessing the conventional path.
A machine file that does not parse is worth less than one that is absent, because it looks present. Rows behind this.
Where the bytes go
Of the 89,533 decompressed bytes the homepage delivers, 0.3% is text a reader or a model can actually use. Most of what a crawler downloads here is not words.
- Readable text 0.3% · 312 B
- Markup & attributes 97.3% · 87,133 B
- Inline JavaScript 2.3% · 2,088 B
What changed
- 2026-09-02resolved: www and the apex both serve the site instead of one redirectingresolved
- 2026-09-01a sitemap became readableresolved
- 2026-09-01no llms.txt was foundappeared
- 2026-09-01a new finding appeared: www and the apex both serve the site instead of one redirectingappeared
7 scans on record, first 2026-08-18. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.
Is this your site?
This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.
Claim the record for siteliner.comDispute or re-scan
If a reading here is wrong, it is our instrument that is wrong, and we want to know. Email hello@crawlcheck.io and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.
Category “Site crawler” is our label for what the product is primarily sold as; it is not scored. Back to the directory.