{"url":"https://lab2.crawlcheck.io/","finalUrl":"https://lab2.crawlcheck.io/","checkedAt":"2026-11-04T05:57:19.157Z","ms":92,"control":{"status":200,"refused":false,"browserRefused":false,"error":null},"robots":{"state":"ok","note":"robots.txt at https://lab2.crawlcheck.io returned 200 with 4 groups","groups":4},"score":{"reachable":13,"blocked":1,"verify":2,"unverifiable":0,"total":16,"pct":81},"issues":[{"key":"robots:https://lab2.crawlcheck.io:named:/","title":"Blocked by robots.txt: the \"User-agent: ClaudeBot\" group, line 11: Disallow: /","blocks":["ClaudeBot"],"fix":"Edit the \"User-agent: ClaudeBot\" group: remove or narrow the Disallow rule that matches /, or add a longer \"Allow: /\" rule."},{"key":"verify:429:ai","title":"Firewall treats these crawlers differently from a browser: HTTP 429 when requested as GPTBot, while a normal browser gets HTTP 200","blocks":["GPTBot"],"fix":"Open your CDN or WAF bot settings and check the rule for GPTBot (OpenAI publishes its IP ranges at https://openai.com/gptbot.json, so an IP-verified allow rule is possible). If GPTBot is set to block or challenge, allow it."},{"key":"verify:403:ai","title":"Firewall treats these crawlers differently from a browser: HTTP 403 when requested as PerplexityBot, while a normal browser gets HTTP 200","blocks":["PerplexityBot"],"fix":"Open your CDN or WAF bot settings and check the rule for PerplexityBot (Perplexity AI publishes its IP ranges at https://www.perplexity.com/perplexitybot.json, so an IP-verified allow rule is possible). If PerplexityBot is set to block or challenge, allow it."}],"results":[{"token":"GPTBot","name":"GPTBot","operator":"OpenAI","purpose":"training","profile":"/directory/ai-bots/gptbot","verdict":"Firewall block: verify","evidence":"HTTP 429 when requested as GPTBot, while a normal browser gets HTTP 200. The site treats this user agent differently. Either your firewall blocks GPTBot (a real block), or it blocks requests that claim to be GPTBot from outside OpenAI's IP range (fake-bot protection, the real crawler gets through). Our request does not come from OpenAI's network, so we cannot tell which.","fix":"Open your CDN or WAF bot settings and check the rule for GPTBot (OpenAI publishes its IP ranges at https://openai.com/gptbot.json, so an IP-verified allow rule is possible). If GPTBot is set to block or challenge, allow it.","issueKey":"verify:429:ai","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":null,"detail":"Not evaluated (page not fetched)"},"status":{"ok":false,"detail":"HTTP 429","code":429},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":null,"detail":"Not evaluated"}}},{"token":"OAI-SearchBot","name":"OAI-SearchBot","operator":"OpenAI","purpose":"search","profile":"/directory/ai-bots/oai-searchbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"877 characters of text are present without JavaScript."}}},{"token":"ChatGPT-User","name":"ChatGPT-User","operator":"OpenAI","purpose":"user","profile":"/directory/ai-bots/chatgpt-user","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"ClaudeBot","name":"ClaudeBot","operator":"Anthropic","purpose":"training","profile":"/directory/ai-bots/claudebot","verdict":"Blocked by robots.txt","evidence":"the \"User-agent: ClaudeBot\" group, line 11: Disallow: /","fix":"Edit the \"User-agent: ClaudeBot\" group: remove or narrow the Disallow rule that matches /, or add a longer \"Allow: /\" rule.","issueKey":"robots:https://lab2.crawlcheck.io:named:/","gates":{"robots":{"ok":false,"detail":"the \"User-agent: ClaudeBot\" group, line 11: Disallow: /"},"directive":{"ok":true,"detail":"No blocking directive"},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"1 redirect, ending at https://lab2.crawlcheck.io/unavailable","hops":1},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"Only 37 characters of text are in the raw HTML. If the main content loads with JavaScript, AI fetchers will miss it."}}},{"token":"Claude-SearchBot","name":"Claude-SearchBot","operator":"Anthropic","purpose":"search","profile":"/directory/ai-bots/claude-searchbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Claude-User","name":"Claude-User","operator":"Anthropic","purpose":"user","profile":"/directory/ai-bots/claude-user","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"PerplexityBot","name":"PerplexityBot","operator":"Perplexity AI","purpose":"search","profile":"/directory/ai-bots/perplexitybot","verdict":"Firewall block: verify","evidence":"HTTP 403 when requested as PerplexityBot, while a normal browser gets HTTP 200. The site treats this user agent differently. Either your firewall blocks PerplexityBot (a real block), or it blocks requests that claim to be PerplexityBot from outside Perplexity AI's IP range (fake-bot protection, the real crawler gets through). Our request does not come from Perplexity AI's network, so we cannot tell which.","fix":"Open your CDN or WAF bot settings and check the rule for PerplexityBot (Perplexity AI publishes its IP ranges at https://www.perplexity.com/perplexitybot.json, so an IP-verified allow rule is possible). If PerplexityBot is set to block or challenge, allow it.","issueKey":"verify:403:ai","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":null,"detail":"Not evaluated (page not fetched)"},"status":{"ok":false,"detail":"HTTP 403","code":403},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":null,"detail":"Not evaluated"}}},{"token":"Perplexity-User","name":"Perplexity-User","operator":"Perplexity AI","purpose":"user","profile":"/directory/ai-bots/perplexity-user","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Google-Extended","name":"Google-Extended","operator":"Google","purpose":"training","profile":"/directory/ai-bots/google-extended","verdict":"Opt-out token: allowed","evidence":"Robots.txt control token only: it never fetches pages, so gates 2 to 6 do not apply. the \"User-agent: *\" group has no rule matching /.","fix":"Nothing to fix, unless you want to opt out: add a \"User-agent: Google-Extended\" group with \"Disallow: /\".","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Googlebot","name":"Googlebot","operator":"Google","purpose":"search-engine","profile":"/directory/search-engines/googlebot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,229 characters of text are present without JavaScript."}}},{"token":"bingbot","name":"Bingbot","operator":"Microsoft","purpose":"search-engine","profile":"/directory/search-engines/bingbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Applebot-Extended","name":"Applebot-Extended","operator":"Apple","purpose":"training","profile":"/directory/search-engines/applebot-extended","verdict":"Opt-out token: allowed","evidence":"Robots.txt control token only: it never fetches pages, so gates 2 to 6 do not apply. the \"User-agent: *\" group has no rule matching /.","fix":"Nothing to fix, unless you want to opt out: add a \"User-agent: Applebot-Extended\" group with \"Disallow: /\".","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Meta-ExternalAgent","name":"Meta-ExternalAgent","operator":"Meta","purpose":"training","profile":"/directory/ai-bots/meta-externalagent","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Amazonbot","name":"Amazonbot","operator":"Amazon","purpose":"search","profile":"/directory/cloud-services/amazonbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"Bytespider","name":"Bytespider","operator":"ByteDance","purpose":"training","profile":"/directory/scrapers/bytespider","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"CCBot","name":"CCBot","operator":"Common Crawl","purpose":"training","profile":"/directory/scrapers/ccbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"DuckAssistBot","name":"DuckAssistBot","operator":"DuckDuckGo","purpose":"user","profile":"/directory/ai-bots/duckassistbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"MistralAI-User","name":"MistralAI-User","operator":"Mistral AI","purpose":"user","profile":"/directory/ai-bots/mistralai-user","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"cohere-ai","name":"cohere-ai","operator":"Cohere","purpose":"user","profile":"/directory/ai-bots/cohere-ai","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}},{"token":"YouBot","name":"YouBot","operator":"You.com","purpose":"search","profile":"/directory/ai-bots/youbot","verdict":"Reachable","evidence":"Reachable. <meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it..","fix":"Nothing to fix.","issueKey":"","gates":{"robots":{"ok":true,"detail":"the \"User-agent: *\" group has no rule matching /"},"directive":{"ok":true,"detail":"<meta name=\"robots\"> says \"nosnippet\". The page can be fetched, but engines that honour it will not quote it."},"status":{"ok":true,"detail":"HTTP 200","code":200},"redirects":{"ok":true,"detail":"No redirects","hops":0},"interstitial":{"ok":true,"detail":"No challenge or block page detected"},"js":{"ok":true,"detail":"1,138 characters of text are present without JavaScript."}}}]}