Directory · Site crawler
Screaming Frog — what a crawler receives from screamingfrog.co.uk
Website measurement, not a product review. This page records what machine clients received from screamingfrog.co.uk. It does not evaluate Screaming Frog software, features, pricing or support.
Answer engines can reach and read this site. It needs more it can quote.
AI visibility 81/100, measured 2026-10-18 17:02 UTC from outside its network, as each named AI and search crawler and as a browser. An engine can reach and read this site. What is left is giving it something specific to say.
Held at C A medium-severity finding sets the letter.
6
findings
✗
llms.txt on its host
✗
entity map on its host
24
scans on record since 2026-09-01
The three stages, in the order an engine hits a site
Each stage gates the next: an engine that cannot reach a page never reads it, and one that cannot read it never quotes it.
Stage 1 · 40% of the score
Reach
Can a named answer-engine crawler get your pages at all?
At least one answer engine was refused, challenged or served less than a browser.
Stage 2 · 30% of the score
Read
Once it has the bytes, can it find the words?
The content is buried in markup, or the files that guide a crawler are missing.
Stage 3 · 30% of the score
Quote
Is there a specific fact it can state and attribute?
The facts an assistant is asked for are declared and consistent.
The stage to fix first is read, because the three run in order.
What to fix
6 findings on this site, 2 of them scored serious. Every one is named below; the top one is shown in full, free. What we saw, why each matters, how to fix it and the bytes behind it are what a licence buys — or we implement the whole list for you.
2 Medium 4 Note
1robots.txt declares no sitemapNo Sitemap: directive present.MediumSetting
What we saw
No Sitemap: directive present. /robots.txt
Why it matters
Adding one Sitemap: line lets crawlers enumerate the site instead of guessing at it.
How to fix it
Add a Sitemap: line to robots.txt pointing at the full sitemap URL, so crawlers find it without guessing.
Setting: a setting on the server, CDN, DNS or a plugin — no code and no writing
On WordPress (detected)
- WordPress serves a virtual robots.txt. Edit it in your SEO plugin: Yoast SEO → Tools → File editor; Rank Math → General Settings → Edit robots.txt; All in One SEO → Tools → Robots.txt Editor.
- A real robots.txt file in the site root (the folder with wp-config.php) overrides every plugin. If one exists, edit that file instead, through your host's file manager or SFTP.
- Keep one group per crawler: a crawler that has its own User-agent group ignores the * group entirely, so repeat any Disallow lines it still needs.
In Cloudflare (this site is behind it)
- If Cloudflare's managed robots.txt is on (Security → Bots, or AI Crawl Control → robots.txt), Cloudflare adds its own lines above yours. Turn it off, or keep it knowing those lines are there.
- Robots.txt is cached at the edge: purge it (Caching → Configuration → Custom purge → the robots.txt URL) after you change it.
On WordPress (detected)
- WordPress 5.5 and later serve /wp-sitemap.xml; Yoast and Rank Math replace it with /sitemap_index.xml. Keep exactly one turned on.
- Core's wp-sitemap.xml has no lastmod dates; Yoast and Rank Math sitemaps do. If the finding is missing lastmod, use the plugin sitemap.
- Add the sitemap's address to robots.txt as Sitemap: https://yourdomain/sitemap_index.xml.
Paste this
Add this line anywhere in robots.txt.
Sitemap: https://www.screamingfrog.co.uk/sitemap.xmlThe whole corrected robots.txt for www.screamingfrog.co.uk · built from the live file, every change listed at the top.
Fix it with a coding agent
Paste into Claude Code, Cursor or Codex, opened on the code that serves this site.
Show prompt
You are working in the codebase (or server/CDN configuration) that serves screamingfrog.co.uk. A crawler audit found the problem below. Fix it at the source, keep the change as small as possible, and do not change anything a visitor sees unless the fix requires it. Finding: robots.txt declares no sitemap (ROBOTS_NO_SITEMAP, medium severity) Page: https://screamingfrog.co.uk/robots.txt What was measured: No Sitemap: directive present. Why it matters: Adding one Sitemap: line lets crawlers enumerate the site instead of guessing at it. How to fix it: Add a Sitemap: line to robots.txt pointing at the full sitemap URL, so crawlers find it without guessing. Detected platform: WordPress. Make the change where this platform controls it. Rule and examples: https://crawlcheck.io/rules/ROBOTS_NO_SITEMAP Full report (markdown): https://crawlcheck.io/r/ftvczknh2m.md When you are done: 1. Tell me which files or settings you changed and why. 2. If the fix is a server, CDN or cache setting you cannot change from the code, say exactly where it is set. 3. After deploying, re-scan at https://crawlcheck.io/?url=screamingfrog.co.uk . The fix counts when ROBOTS_NO_SITEMAP is gone from the new report.
Why this was decided
| Rule | ROBOTS_NO_SITEMAP · rule pack sv32 · scope core |
|---|---|
| Evaluated | 2026-10-18 17:02:38 UTC |
| Decision | d1:46f7db265eb323fbdb26 · finding f1:cbe9479a4451c37add0a resolve either at /api/graph/<id> with a key |
| Observer | crawlcheck-worker build b3cf8c17 · audit v4 |
What it rests on
| Identity | Status | Bytes | sha256 | Kept |
|---|---|---|---|---|
| read: /robots.txt | 200 | 1,883 | 4daae490a5b5 | already_held |
23 more in the manifest.
Artifacts: manifest:a5325dca461eb28dd58f3ae9fbb3da450793fd6a2b8ed0c5fcfcb040e3829a35
Grade effect Caps the letter at C together with 1 other medium-severity finding; the letter rises only when all of them are gone. Moves no points.
Rule ROBOTS_NO_SITEMAP under scoring rules v32 · fingerprint ROBOTS_NO_SITEMAP|/robots.txt · persisting (30 scans)
Verify the fix Re-scan after the change. This finding is cleared when ROBOTS_NO_SITEMAP is no longer reported at /robots.txt; if it comes back after clearing, it is marked a regression, not a new finding.
ROBOTS_NO_SITEMAP · full transcript
Open the other 5
What we saw, why it matters, how to fix it, and the evidence bytes for every finding — on this report and every re-scan. From $29 a month.
Unlock the findingsOr have them fixed
We implement the list on your site and re-measure, with a sealed before and after. One site, $749 once.
Get this fixed — $749The main product
We fix the 6 findings on screamingfrog.co.uk. $749, once.
2 of them are scored serious. One site. You keep the grade for free — this is the part where the list stops being a list.
- Every finding implemented on the site itself: machine files, schema, NAP, the lot
- An entity map authored for your business, not a template with the name swapped
- Measured again afterwards — the before and the after, sealed, so it is evidenced rather than claimed
- Thirty days of monitoring included
- Work starts within two business days
What it covers · one payment, nothing recurring
The chain a crawler follows
The chain breaks at llms.txt. Everything after that point is only reachable by a crawler guessing the conventional path.
A machine file that does not parse is worth less than one that is absent, because it looks present. Rows behind this.
Where the bytes go
Of the 200,077 decompressed bytes the homepage delivers, 4.8% is text a reader or a model can actually use. Most of what a crawler downloads here is not words.
- Readable text 4.8% · 9,689 B
- Markup & attributes 41.7% · 83,342 B
- Structured data (JSON-LD) 1.2% · 2,340 B
- Inline CSS 7.8% · 15,565 B
- Inline JavaScript 40.4% · 80,769 B
- Inline SVG 3.5% · 6,997 B
- HTML comments 0.7% · 1,375 B
The individual blocks that weigh the most, so the fix is a search rather than a hunt:
- Inline JavaScript · 51,537 B (25.8%) —
#wc-country-select-js-extraopensvar wc_country_select_params = {"countries":"{\"AF\":[],\"AL\":{\"AL-0 - Inline JavaScript · 10,943 B (5.5%) —
#wc-address-i18n-js-extraopensvar wc_address_i18n_params = {"locale":"{\"AE\":{\"postcode\":{\"requi - Inline CSS · 10,286 B (5.1%) — WordPress theme or block stylesopens
:root{--wp--preset--aspect-ratio--square: 1;--wp--preset--aspect-ratio - Inline SVG · 6,972 B (3.5%) —
#Layer_1opens<g> <rect x="-32.6" y="-14" class="st0" width="2044.1" height="388.6" - Inline JavaScript · 3,426 B (1.7%) — unnamedopens
var sv_wc_payment_gateway_payment_form_params = {"order_button_text":"
What changed
- 2026-10-18a stale edge copy was served againreturned
- 2026-10-15the stale edge copy clearedresolved
- 2026-10-15the homepage began taking more than one redirect to reachappeared
- 2026-09-17score moved 75 → 76score-up
- 2026-09-17a stale edge copy was served againreturned
- 2026-09-16the stale edge copy clearedresolved
- 2026-09-16a new finding appeared: the machine files cost more than the pageappeared
- 2026-09-14score moved 75 → 76score-up
- 2026-09-14a stale edge copy was served againreturned
- 2026-09-13score moved 76 → 75score-down
- 2026-09-12the shadowed robots rules clearedresolved
- 2026-09-12the stale edge copy clearedresolved
24 scans on record, first 2026-09-01. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.
Is this your site?
This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.
Claim the record for screamingfrog.co.ukIs this your company?
This listing came from a fetch, not a submission, so nobody was asked. If screamingfrog.co.uk is yours you can prove it in a minute and for nothing — a DNS TXT record or one meta tag on the home page, both checkable by anyone.
Claiming marks you as the verified owner and lets you correct the category and the company name. It moves no score, no rank and no position.
Claim this listing — free or ask about a verified profile
Dispute or re-scan
If a reading here is wrong, it is our instrument that is wrong, and we want to know. Tell us what is wrong and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.
By email instead: hello@crawlcheck.io. Category “Site crawler” is our label for what the product is primarily sold as; it is not scored. Back to the directory.