{"ok":true,"kind":"crawlcheck-rulebook","v":1,"generated_at":"2026-11-22T01:33:47.999Z","score_version":43,"schema":"https://crawlcheck.io/schemas/rulebook.json","counts":{"rules":111,"by_kind":{"scan":86,"answer_correlation":5,"capability_mismatch":6,"self_audit":12,"mcp_server":2},"by_family":{"site":18,"headers":4,"mostly_code":3,"machine_files":12,"other":9,"ard":4,"webmcp":2,"trust_path":2,"nap":3,"robots":17,"delivery":9,"jsonld":3,"answers":5,"capabilities":6,"self_audit":12,"mcp":1,"watch":1},"revised":10,"scored":82},"scans_counted":9304,"severity_scale":[{"level":"info"},{"level":"low"},{"level":"medium"},{"level":"high"},{"level":"critical"}],"method":{"revisions":"A rule that changes what it reports gets a new revision; a record keeps the revision that decided it, and a replay uses that revision, never today's.","share":"Share of counted scans that carried the code at least once, the same numbers /data publishes. Self-scans and opted-out sites are never counted.","grade":"Each finding has a level from info to critical. Every graded finding lowers the score of its section; an open finding at medium or above also caps the letter grade, so fix those first."},"rules":[{"code":"SHOPPING_BLOCKED_STOREBOT","kind":"scan","family":{"id":"other","label":"Other"},"title":"robots.txt keeps Google's Shopping crawler off the store","meaning":"The site sells online, and robots.txt disallows Storebot-Google on its homepage or product pages (that crawler obeys its own group, else the * group). Google says rules for Storebot-Google affect all surfaces of Google Shopping, so those products cannot be listed in the Shopping tab or other Shopping results from the crawl. Googlebot being allowed does not cover it. The usual cause is a Disallow: / under User-agent: * written to keep unknown bots out, or a block-all-bots preset.","severity":{"level":"high"},"scored":true,"measured_on":"every scan","revision":{"current":1,"revised_at":null},"fix":{"advice":"Add a User-agent: Storebot-Google group that allows the product pages (Allow: /), or remove the Disallow that covers them from the group it follows.","effort":"config","endpoint":null},"share_of_scans":{"pct":0,"scans":0,"of_scans":9304,"basis":"no counted scan has carried it yet"},"links":{"self":"https://crawlcheck.io/rules#SHOPPING_BLOCKED_STOREBOT","api":"https://crawlcheck.io/api/rules?code=SHOPPING_BLOCKED_STOREBOT","glossary":[],"workflow":null,"lineage":"https://crawlcheck.io/docs/lineage-coverage","explain_template":"https://crawlcheck.io/api/explain?id={report_id}&code=SHOPPING_BLOCKED_STOREBOT"}}],"filter":{"code":"SHOPPING_BLOCKED_STOREBOT"}}