Sixty-three faults, ten free tools, one answer key.

How it checks. Fetches the homepage as 13 named crawlers plus browser and mobile controls and compares what each one is served; applies robots.txt per crawler; reads robots.txt, the sitemap tree, llms.txt and security.txt; a separate free 41-page crawl reads every page's title, directives, canonical, links and redirects.

What it fetched from the lab (from the lab's request log): 541 requests, 103 distinct paths, sent as 53 user agents.

Caught (26). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything; R2 /assets/ (the stylesheet and the script every page needs) is disallowed for all crawlers; R3 Googlebot-Image is disallowed from the whole site; R4 Storebot-Google is disallowed from the whole site; S1 PerplexityBot is answered with HTTP 403 on every page; S2 Applebot gets HTTP 200 with a 'Just a moment...' challenge page instead of content; S4 ClaudeBot is redirected (302) from every page to /unavailable; S5 GPTBot gets HTTP 429 on every request; I1 /services/gutter-guards carries meta robots noindex; I2 /pricing is sent with an X-Robots-Tag: noindex header; I4 /services/downspout-repair has a canonical pointing to a different service page; I5 The homepage carries meta robots nosnippet; J3 /reviews content is rendered entirely by JavaScript into an empty container; D1 The homepage JSON-LD does not parse (trailing comma); M1 llms.txt does not start with an H1 title; X2 The sitemap lists a noindex URL (/services/gutter-guards); X3 The sitemap lists a redirecting URL (/old-gutter-page); X4 The sitemap lists a URL that returns 404 (/services/roof-repair); L3 /promo-a and /promo-b redirect to each other (loop), linked from a blog post; L5 /areas/lakeside is in the sitemap but linked from no page (orphan); P1 /careers has no <title>; P2 /areas/millbrook and /areas/cedar-park have the same title; P4 /services/ice-dam-prevention has two <h1> elements; P5 The three homepage images have no alt text; T1 No Strict-Transport-Security header on any page; T4 The homepage sets a session cookie without Secure or HttpOnly

Partly (14). R6 The Sitemap line is a relative path; it must be an absolute URL (robots.txt names a sitemap | 1 Sitemap: line(s) | at least one | ✓ Passed); S3 OAI-SearchBot gets pages with the services, areas and prices sections removed (OAI-SearchBot | different | words (198 → 164) | a template or a cache variant, not a different page); S6 Googlebot user agents get an extra keyword block no other visitor sees (cloaking) (Googlebot | 200 | 220 | 0.91 | different body text); I3 /areas/riverbend has a canonical pointing to a URL that returns 404 (/areas/riverbend canonical elsewhere · listed in the sitemap); I6 /faq has two conflicting robots meta tags (index,follow and noindex) (Gutter FAQ | Northfield Gutter Co. | 200 | 1 | 65 | ... | self | noindex); J1 /pricing prices exist only after JavaScript runs (the HTML says 'Loading prices...') (/pricing | 106 → 132 | 20% | 14 → 14 | 0 → 0 | —); J2 The homepage links to the area pages only through JavaScript click handlers, not <a href> (7 elements with onclick and no tabindex or role (span.lnk, span.lnk, span.lnk)); J4 /faq answers are fetched from /api/faq.json after load and are not in the HTML (/faq | 65 → 94 | 31% | 14 → 14 | 0 → 0 | —); M5 security.txt is served as text/html (security.txt | ... | absent (200)); X5 Two lastmod values are not valid W3C dates (Sitemap URLs with a change date | ... | 36 of 38); X6 The sitemap lists a URL on another host (/services status 530 · listed in the sitemap); L1 The homepage links to /services/roof-repair, which returns 404 (/services/roof-repair Not found | 404 | 1 | 4 | ... | 1 | 0); L2 /old-gutter-page redirects three times before reaching a page (/old-gutter-page 2 hops: 301 permanent → 301 permanent → https://lab2.crawlcheck.io/services/ (200)); L4 /specials returns 200 with 'Page not found' text (soft 404), linked from the footer (/specials Page not found | 200 | 1 link-found | 13)

Where it said the opposite of the truth:

• I7 /about has a canonical to the http:// version of itself: “About Northfield Gutter Co. | 200 | 1 | 171 | 36.3% | 2 | none | 1 | self”

Missed (22) on pages or files it did read; never looked where the other 0 live.

On the clean twin it said something untrue:

• Business phone on the page: none found (one defect, three statements: "none found ✗ Failed", "structured data only, not shown to readers", "Phone numbers written on the page | 0") — The number is printed in the page body and footer. Cause: the page-side phone check read only tel: links. Fixed after grading (2026-11-04); the first run is what is graded.
• The sitemap lists URLs but carries no lastmod dates / Add <lastmod> dates — Every entry has a lastmod. The detail line is true (all 35 share one build-stamp date) but the explanation and fix shown were the no-lastmod text. Fixed after grading (2026-11-04).

Evidence quality 2 of 3. Free output names agents, URLs and observed values: per-identity status, words, bytes and sha256; machine-file transcript with status, content-type and bytes; redirect hops; JS raw-vs-rendered counts; top finding with rule link and fetch manifest. Not 3: most finding evidence is locked, robots verdicts carry no line numbers, and several rows contradict the report's own tables (ClaudeBot redirect, Googlebot split, internal links resolve, sitemap named). One redirect status (/services/ 200) was never observed.

See for yourself: Homepage report, faulty site · Homepage report, clean twin · 41-page crawl, faulty site · 41-page crawl, clean twin · its reports above are the raw output · grade sheet

How it checks. Evaluates robots.txt for 265 bot tokens, then fetches the homepage as 20 crawler user agents through six gates (robots, directives, status, redirects, interstitial, text without JavaScript); a GEO endpoint reads the page and llms.txt.

What it fetched from the lab (from the lab's request log): 42 requests, 10 distinct paths, sent as 22 user agents.

Caught (8). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything; R3 Googlebot-Image is disallowed from the whole site; R4 Storebot-Google is disallowed from the whole site; S1 PerplexityBot is answered with HTTP 403 on every page; S5 GPTBot gets HTTP 429 on every request; I5 The homepage carries meta robots nosnippet; D1 The homepage JSON-LD does not parse (trailing comma); P5 The three homepage images have no alt text

Partly (3). R2 /assets/ (the stylesheet and the script every page needs) is disallowed for all crawlers (Full content access; 2 path rules (/assets/, /quote)); R6 The Sitemap line is a relative path; it must be an absolute URL (Sitemap index with 2 child sitemaps at https://lab2.crawlcheck.io/sitemap.xml. Not declared in robots.txt.); S4 ClaudeBot is redirected (302) from every page to /unavailable (1 redirect, ending at https://lab2.crawlcheck.io/unavailable)

Missed (12) on pages or files it did read; never looked where the other 40 live.

On the clean twin: no untrue claims. Advisories only: llms-full.txt missing; no named training-bot groups; thin content; no question headings.

Evidence quality 3 of 3. Names the agent, the observed status (403/429), the robots line number and group, character counts per agent, and states its own method ("Tested with the crawler user agent from our server, not the crawler IP range").

Good at things the key does not reward. Robots verdicts for 265 user-agent tokens across 8 categories (acc_main_lab2.json totalBots 265). Live fetch of the homepage as 20 crawler user agents with six gates each (robots, directive, status, redirects, interstitial, JS) (acc_diag_lab2.json results). Honest caveat on fake-bot protection: "Our request does not come from Perplexity AI's network, so we cannot tell which." (diag results[PerplexityBot].evidence).

See for yourself: Run on the faulty site · Run on the clean twin · raw output: acc_main_lab2.json, acc_diag_lab2.json, acc_geo_lab2.json, acc_main_lab2-clean.json, acc_diag_lab2-clean.json, acc_geo_lab2-clean.json · grade sheet

How it checks. Fetches the homepage as six AI crawlers and in a browser, reads robots.txt for 34 bots and the sitemap as Googlebot; the llms.txt checker parses the file and follows every link in it.

What it fetched from the lab (from the lab's request log): 34 requests, 21 distinct paths, sent as 10 user agents.

Caught (6). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything; S1 PerplexityBot is answered with HTTP 403 on every page; S5 GPTBot gets HTTP 429 on every request; I5 The homepage carries meta robots nosnippet; M1 llms.txt does not start with an H1 title; M2 llms.txt links to two pages that return 404

Partly (2). S4 ClaudeBot is redirected (302) from every page to /unavailable (ClaudeBot (200 but only 4 words)); D1 The homepage JSON-LD does not parse (trailing comma) (FIX Schema.org structured data 8 pts — No JSON-LD schema.org markup found.)

Missed (22) on pages or files it did read; never looked where the other 33 live.

On the clean twin: no untrue claims. Advisories only: No Last-Modified on llms.txt; llms.txt links to HTML not .md; no llms-full.txt; few answer-friendly structures.

Evidence quality 3 of 3. Cites robots line numbers (line 11, llms line 1), observed status codes per bot UA (429, 403) and word counts from live probes it actually made under each UA, and counts resolving llms.txt links (2/4, 11/11, both correct per the log).

Good at things the key does not reward. Separates robots permission from server behaviour: "robots does not disallow it (possible UA filtering)" — the right diagnosis for UA-based blocking. Link-checks every llms.txt link: "2/4 links resolve, 2 dead" on lab2 and "11/11 links resolve" on clean, both exact per the log. Accurately labels optional items as optional: "WARNING No summary blockquote — Optional per spec." (true on lab2, not a key fault).

See for yourself: Scanner on the faulty site · Scanner on the clean twin · llms.txt checker · raw output: geotoolbox_scanner_lab2.txt, geotoolbox_scanner_lab2-clean.txt, geotoolbox_llms_lab2.txt, geotoolbox_llms_lab2-clean.txt · grade sheet

How it checks. About 26 weighted checks in seven groups: crawl permission, findability, readability without JavaScript, structured data, contact and price legibility, llms.txt, agent protocols. Fetches the homepage with its own agent, a headless browser and once as ClaudeBot.

What it fetched from the lab (from the lab's request log): 19 requests, 14 distinct paths, sent as 4 user agents.

Caught (3). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything; D1 The homepage JSON-LD does not parse (trailing comma); M1 llms.txt does not start with an H1 title

Partly (1). R6 The Sitemap line is a relative path; it must be an absolute URL (Declared: /sitemap.xml. ... The sitemap probe could not be completed, so this could not be assessed.)

Where it said the opposite of the truth:

• S4 ClaudeBot is redirected (302) from every page to /unavailable: “Not blocked by bot detection 6/6 pts”
• J1 /pricing prices exist only after JavaScript runs (the HTML says 'Loading prices...'): “Prices are reachable without JavaScript or login 4/4 ... A pricing page is linked from the homepage.”

Missed (17) on pages or files it did read; never looked where the other 40 live.

On the clean twin: no untrue claims. Advisories only: no Content-Signal; no markdown negotiation; no Link headers; no WebMCP.

Evidence quality 1 of 3. Findings are named ("Blocked by robots.txt: ClaudeBot.", missing "# " heading, no valid JSON-LD) without the rule, line, status code or raw response; observed values appear only on passing or advisory rows (byte counts, Content-Type for Accept: text/markdown).

Good at things the key does not reward. Actually tests markdown negotiation and reports the observed response: "Homepage returned 'text/html; charset=utf-8' for Accept: text/markdown." Probes a random URL for a real 404 and checks for an MCP server card. Measures readable text in raw HTML: "Raw HTML contains 1830 characters of readable text."

See for yourself: Run on the faulty site · Run on the clean twin · raw output: hostinger_lab2.txt, hostinger_lab2-clean.txt · grade sheet

How it checks. Requests the homepage once as each of 25 AI user agents and reports the HTTP status for each.

What it fetched from the lab (from the lab's request log): 25 requests, 1 distinct paths, sent as 24 user agents.

Caught (2). S1 PerplexityBot is answered with HTTP 403 on every page; S5 GPTBot gets HTTP 429 on every request

Partly (1). S4 ClaudeBot is redirected (302) from every page to /unavailable ({"status":"success","code":"302","message":"Found"})

Where it said the opposite of the truth:

• S2 Applebot gets HTTP 200 with a 'Just a moment...' challenge page instead of content: “{"status":"success","code":"200","message":"OK"}”

Missed (9) on pages or files it did read; never looked where the other 50 live.

On the clean twin: no untrue claims.

Evidence quality 2 of 3. Names each agent with the HTTP status code it got; no request detail, no redirect target, no explanation or standard.

Good at things the key does not reward. Live homepage fetch with 25 real crawler user-agent strings, including legacy tokens (anthropic-ai, claude-web), Diffbot, Timpibot, omgili, LinkedInBot and AI2Bot (request log 1922266-1927852). Fetches from Amazon AWS rather than a single CDN, one agent at a time. Clean twin: zero false alarms.

See for yourself: Result for the faulty site · Result for the clean twin · raw output: knowatoa_lab2.txt, knowatoa_lab2-clean.txt · grade sheet

How it checks. Probes 20+ agent-protocol endpoints (MCP server card, A2A card, OAuth discovery, API catalog, agent skills, payments), DNS records, markdown negotiation, robots.txt and the sitemap index; renders the homepage headless to detect WebMCP.

What it fetched from the lab (from the lab's request log): 33 requests, 27 distinct paths, sent as 2 user agents.

Caught (1). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything

Partly (2). R2 /assets/ (the stylesheet and the script every page needs) is disallowed for all crawlers ({"directive": "disallow", "pattern": "/assets/"}); R6 The Sitemap line is a relative path; it must be an absolute URL (No Sitemap directives found in robots.txt)

Missed (18) on pages or files it did read; never looked where the other 42 live.

On the clean twin: no untrue claims. Advisories only: no Link headers; no DNS-AID records; no markdown negotiation; no Content Signals.

Evidence quality 3 of 3. Every check carries the request it made, the response status, selected headers and a body preview (e.g. the full robots.txt), so findings are reproducible; but it catches almost none of the key.

Good at things the key does not reward. Probes 20+ agent-protocol endpoints (MCP server card, A2A agent card, OAuth discovery, api-catalog, agent skills, UCP/ACP, x402) with the request and status shown for each. DNS-AID lookups over DoH with the queries listed (checks.discoverability.dnsAid.details.queriesAttempted). Markdown content negotiation test (Accept: text/markdown).

See for yourself: Run on the faulty site · Run on the clean twin · raw output: iar_lab2.json, iar_lab2-clean.json · grade sheet

How it checks. Reads robots.txt for 8 AI crawlers and shows the matched rule, checks that llms.txt and a sitemap exist, counts structured-data blocks, reads title, description and canonical.

What it fetched from the lab (from the lab's request log): 4 requests, 4 distinct paths, sent as 1 user agents.

Caught (1). R1 ClaudeBot's robots group holds only a Crawl-delay, so its access depends on the parser: Google's reference parser merges it into the next group and blocks it site-wide (line 11); others read an empty group and allow everything

Partly (1). R6 The Sitemap line is a relative path; it must be an absolute URL (Sitemap Failed /sitemap.xml)

Missed (20) on pages or files it did read; never looked where the other 41 live.

On the clean twin: no untrue claims. Advisories only: no llms-full.txt.

Evidence quality 2 of 3. Names the agent and the matched robots rule ("ClaudeBot | No | disallow: /") and status codes for llms files, but no line numbers, raw responses or live crawler-UA requests.

Good at things the key does not reward. Shows the matched rule per crawler, so a reader can see why each verdict was reached. Reports title, description and canonical of the homepage. Clean twin: zero false alarms; all eight crawlers "allow: /" correctly.

See for yourself: Tool page (enter the address) · raw output: swetrix_lab2.txt, swetrix_lab2-clean.txt · grade sheet

How it checks. Fetches one URL as Google-InspectionTool, renders it, and tests its structured data for rich-result eligibility.

What it fetched from the lab (from the lab's request log): 6 requests, 6 distinct paths, sent as 2 user agents.

Caught (1). D1 The homepage JSON-LD does not parse (trailing comma)

Missed (19) on pages or files it did read; never looked where the other 43 live.

On the clean twin: no untrue claims. Advisories only: LocalBusiness non-critical issues.

Evidence quality 2 of 3. Names the issue at the tested URL with the parser's observed error message; the capture holds no line number or raw response (the tool UI highlights the line, but that is not in the capture).

Good at things the key does not reward. Renders the page with Google's own crawler, so its structured-data verdict is what Google itself sees. On lab2 Google-InspectionTool did not fetch /assets/site.css or /assets/site.js (it did on clean): it honoured R2's Disallow: /assets/. The tool's page-resources view probably reports this, but it is not in the capture, so R2 is graded missed. Clean twin: zero false alarms.

See for yourself: Saved result, faulty site · Saved result, clean twin · Re-run on the faulty site · raw output: google_rrt_lab2.txt, google_rrt_lab2-clean.txt · grade sheet

How it checks. Fetches one page with a rendering agent and parses its structured data.

What it fetched from the lab (from the lab's request log): 7 requests, 7 distinct paths, sent as 2 user agents.

Caught (1). D1 The homepage JSON-LD does not parse (trailing comma)

Missed (12) on pages or files it did read; never looked where the other 50 live.

On the clean twin: no untrue claims.

Evidence quality 3 of 3. Names the exact error class, gives the character offset, and returns the fetched HTML so the error can be located and reproduced.

Good at things the key does not reward. Returns the rendered HTML it validated (isRendered: true) with character offsets for every node and error. Fetches page subresources with a rendering agent, so script-injected JSON-LD would be seen.

See for yourself: Run on the faulty site · Run on the clean twin · raw output: sv_lab2.txt, sv_lab2-clean.txt · grade sheet

How it checks. Fetches six pages as its own bot and the homepage in a browser, compares word counts, and prints a 16-bot table of robots.txt verdicts with a “can read” column.

What it fetched from the lab (from the lab's request log): 11 requests, 10 distinct paths, sent as 3 user agents.

Caught (0). Nothing.

Partly (1). D1 The homepage JSON-LD does not parse (trailing comma) ("label": "Structured data (JSON-LD)", "value": "none", "status": "fail")

Where it said the opposite of the truth:

• S1 PerplexityBot is answered with HTTP 403 on every page: “"name": "PerplexityBot", ... "robots": "allow", "canRead": "yes"”
• S2 Applebot gets HTTP 200 with a 'Just a moment...' challenge page instead of content: “"name": "Applebot", ... "robots": "allow", "canRead": "yes"”
• S4 ClaudeBot is redirected (302) from every page to /unavailable: “"name": "ClaudeBot", ... "robots": "allow", "canRead": "yes"”
• S5 GPTBot gets HTTP 429 on every request: “"name": "GPTBot", ... "robots": "allow", "canRead": "yes"”
• S6 Googlebot user agents get an extra keyword block no other visitor sees (cloaking): “"dynamic": {"checked": true, "cloaking": false, "botWords": 169, "browserWords": 169}”
• J1 /pricing prices exist only after JavaScript runs (the HTML says 'Loading prices...'): “"url": "https://lab2.crawlcheck.io/pricing", "verdict": "readable", ... "mode": "Server-rendered"”
• J3 /reviews content is rendered entirely by JavaScript into an empty container: “"url": "https://lab2.crawlcheck.io/reviews", "verdict": "readable", "words": 39, ... "mode": "Server-rendered"”

Missed (20) on pages or files it did read; never looked where the other 35 live.

On the clean twin: no untrue claims. Advisories only: thin homepage; schema rated warn with no reason given; short page word count.

Evidence quality 1 of 3. Gives labels and counts (words, links, "canRead") with no status codes, agents actually used or rules; the per-bot verdicts are not backed by any fetch as those bots.

Good at things the key does not reward. Crawls 6 pages (homepage plus 5 linked pages) and reports words/links/render mode per page. Raw-HTML vs rendered word comparison and a bot-vs-browser cloaking check (against its own UA). Detects llms.txt and robots.txt presence.

See for yourself: Tool page (enter the address) · raw output: ws_lab2.json, ws_lab2-clean.json · grade sheet