How often CrawlCheck is right, check by check
Every number here comes with how many cases it rests on and how they were chosen. A small sample gets a wide range, and the range is printed. Where nothing has been measured, the page says so.
New checks start as advisory
A new finding is shown on reports but cannot lower a grade or change an agent decision until enough real cases have been checked by hand with no false positive. Each code leaves this list in a dated release.
| Finding | Advisory since | Hand-checked true | False positives | Needed |
|---|---|---|---|---|
SECRETS_EXPOSED | 2026-10-22 | 0 | 0 | 10 |
Agent decisions: the frozen benchmark
For each family of cases in the agent safety benchmark (suite v4), whether CrawlCheck’s own answer matched the case label. These cases were chosen to test whether an agent heeds a signal, so a full match shows the answers in the suite are right; it is not a measure of how often a detector is right across the web.
| Check | Cases | Hazards / controls | Seen live / built | Matched | Missed hazard | False stop | 95% range |
|---|---|---|---|---|---|---|---|
| Lookalike domain detection | 8 | 4 / 4 | 8 / 0 | 8 | 0 | 0 | 68%–100% |
| MCP tool poisoning scan | 8 | 4 / 4 | 5 / 3 | 8 | 0 | 0 | 68%–100% |
| Revocation and receipt check | 5 | 3 / 2 | 0 / 5 | 5 | 0 | 0 | 57%–100% |
| Freshness of a cached decision | 6 | 3 / 3 | 6 / 0 | 6 | 0 | 0 | 61%–100% |
| robots.txt permission for an action | 6 | 3 / 3 | 6 / 0 | 6 | 0 | 0 | 61%–100% |
| Identity corroboration | 6 | 3 / 3 | 3 / 3 | 6 | 0 | 0 | 61%–100% |
Report findings checked by hand
Each time a finding is checked against the site by hand, the result is added here, including the ones that were wrong and what changed because of it. This list starts on 22 October 2026; earlier checks were not recorded in this form.
| Check | Recorded | True | False | Share true (95% range) |
|---|---|---|---|---|
| Outside files that hold the page until they load | 1 | 1 | 0 | 100% (21%–100%) |
| Language versions declared (hreflang): none | 1 | 1 | 0 | 100% (21%–100%) |
| Cited when asked by name / without its name | 1 | 1 | 0 | 100% (21%–100%) |
| Cited by more than one engine | 1 | 0 | 1 | 0% (0%–79%) |
| Same business name on every listing | 1 | 0 | 1 | 0% (0%–79%) |
| Listings link back to the site | 1 | 0 | 1 | 0% (0%–79%) |
| Pages the answers cited | 1 | 0 | 1 | 0% (0%–79%) |
Every entry
| Date | Finding | Site | Result | Note |
|---|---|---|---|---|
| 2026-10-22 | Outside files that hold the page until they load | terrariumstation.com (ours) | true | two font stylesheets in <head> blocked rendering; confirmed in the delivered HTML |
| 2026-10-22 | Language versions declared (hreflang): none | an outside translation service | true | its homepage carries no hreflang links; confirmed by fetching it |
| 2026-10-22 | Cited when asked by name / without its name | tree-d.example (ours) | true | 3 of 4 and 1 of 3 matched line by line against the stored answer run |
| 2026-10-22 | Cited by more than one engine | tree-d.example (ours) | wrong (fixed in aicite3) | failed every site while only one engine was being asked; now not judged until two are |
| 2026-10-22 | Same business name on every listing | tree-d.example (ours) | wrong (fixed in lst1b) | a directory that bans 'LLC' read as a different name |
| 2026-10-22 | Listings link back to the site | tree-d.example (ours) | wrong (fixed in lst2) | a platform that shows no website on any profile was failed; such platforms are now named, not failed |
| 2026-10-22 | Pages the answers cited | tree-d.example (ours) | wrong (fixed in aicite2) | read empty although 50 answers cited the site (cited links were not resolved) |
When a site says we are wrong
Any site can dispute a finding at /disputes. Turnaround and how often a dispute found CrawlCheck wrong are published at /api/v1/disputes/metrics.
What this does not show yet
- The benchmark cases were chosen to test whether an agent heeds a signal; CrawlCheck's answer matching their labels shows the answers in the suite are right, not how often a detector is right across the web.
- Hand-checks are cases someone fetched and judged by hand; they are not a random sample.
- No check has yet been measured on a random sample of live sites with independent labels.
Machine-readable: /accuracy.json · how money is kept away from findings: /neutrality.