Which crawlers denverpost.com has closed, and since when.
From 27 dated readings of denverpost.com’s robots.txt (2026-09-24 19:13 UTC to 2026-11-12 00:17 UTC). Each date links a proof bundle: the fetched bytes, signed and sealed in that day’s timestamped root, checkable offline with the verifier. Signed at /api/v1/optout.
41
crawlers closed
74
crawlers open
4
changed state in the record
| Crawler | Now | In this state since | Changes | Conduct |
|---|---|---|---|---|
| ChatGPT live fetch ChatGPT-User · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Claude search index Claude-SearchBot · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Claude live fetch Claude-User · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| DuckDuckGo AI assist DuckAssistBot · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Meta AI live fetch Meta-ExternalFetcher · answer | closed | between 2026-10-27 00:17 UTC and 2026-10-28 00:18 UTC proof · reading | 1 | — |
| Le Chat live fetch MistralAI-User · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| ChatGPT search index OAI-SearchBot · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Perplexity live fetch Perplexity-User · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Perplexity index PerplexityBot · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| You.com YouBot · answer | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Amazon Amazonbot · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| Amazon search eligibility, separate from Amazonbot Amzn-SearchBot · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Apple and Siri Applebot · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| Baidu Baiduspider · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | B |
| Huawei Petal PetalBot · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Yahoo Slurp Slurp · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Yandex YandexBot · search | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Allen Institute AI2Bot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Allen Institute Dolma Ai2Bot-Dolma · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Anthropic legacy agent anthropic-ai · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Apple training Applebot-Extended · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Apple ads corpus Applebot-Extended-Ads · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Internet Archive archive.org_bot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| ByteDance Bytespider · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Common Crawl CCBot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Anthropic legacy agent Claude-Web · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Anthropic training ClaudeBot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| Cohere training corpus cohere-training-data-crawler · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Diffbot knowledge graph Diffbot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Meta legacy FacebookBot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Gemini training Google-Extended · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| OpenAI training and index GPTBot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | B |
| NICT ICC ICC-Crawler · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Meta AI training meta-externalagent · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Meta web index meta-webindexer · train | closed | between 2026-10-27 00:17 UTC and 2026-10-28 00:18 UTC proof · reading | 1 | — |
| Webz.io omgili omgili · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Webz.io Omgilibot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Huawei PanGu PanguBot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Timpi Timpibot · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Velen VelenPublicWebCrawler · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Webz.io extended Webzio-Extended · train | closed | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Amazon live fetch for Alexa questions Amzn-User · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Microsoft Copilot fetch Copilot-User · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Gemini research agent Gemini-Deep-Research · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google user-triggered agent, acts on the web Google-Agent · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Gemini Notebook user-supplied URL fetch Google-GeminiNotebook · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Kagi Kagibot · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Phind PhindBot · answer | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google push delivery APIs-Google · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Bing and Copilot index bingbot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| Bing page preview BingPreview · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Coc Coc coccocbot-web · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| DuckDuckGo DuckDuckBot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | B |
| Exa AI search index, Web Bot Auth signed ExaSearchBot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google Search and AI Overviews Googlebot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Google image crawler Googlebot-Image · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google News Googlebot-News · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google video crawler Googlebot-Video · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google AdSense Mediapartners-Google · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Mistral search index for Vibe MistralAI-Index · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Microsoft legacy crawler msnbot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Neeva Neevabot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Seznam Seznambot · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Google Shopping Storebot-Google · search | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Yandex (umbrella token) Yandex · search | open | between 2026-11-02 00:17 UTC and 2026-11-03 00:17 UTC proof · reading | 1 | — |
| Awario AwarioBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Awario RSS AwarioRssBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Awario smart AwarioSmartBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Cohere cohere-ai · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| DeepSeek DeepSeekBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| FriendlyCrawler ML FriendlyCrawler · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Vertex AI agent build Google-CloudVertexBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google non-search fetch GoogleOther · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | A |
| Google common crawler, public image URLs GoogleOther-Image · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google common crawler, public video URLs GoogleOther-Video · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Internet Archive (legacy) ia_archiver · train | open | between 2026-11-02 00:17 UTC and 2026-11-03 00:17 UTC proof · reading | 1 | — |
| Imagesift ImagesiftBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| img2dataset image corpus img2dataset · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Magpie AI Magpie-crawler · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Mistral training crawl MistralAI-Training · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| MyCentral AI MyCentralAIScraperBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| ProRata ProRataInc · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| QuillBot QuillBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Generic scraper framework Scrapy · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Semrush AI corpus SemrushBot-OCOB · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| TikTok TikTokSpider · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Turnitin TurnitinBot · train | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| 360 Search 360Spider · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Microsoft Ads AdIdxBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Google Ads quality AdsBot-Google · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Ahrefs link index AhrefsBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | D |
| Amazon Ads AmazonAdBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| WebMeUp link index BLEXBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Cloudflare AutoRAG Cloudflare-AutoRAG · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Cloudflare Browser Run /crawl CloudflareBrowserRenderingCrawler · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| DataForSEO DataForSeoBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Moz link index DotBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Exalead Exabot · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Facebook link preview facebookexternalhit · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Meta link crawler Facebot · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Klaviyo Customer Agent knowledge-source crawler KlaviyoAIBot · tool | open | on or before 2026-11-10 00:17 UTC proof · reading | 0 | — |
| LinkedIn link preview LinkedInBot · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Majestic link index MJ12bot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| OpenAI ChatGPT ads page validation OAI-AdsBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Peer39 ad context Peer39_Crawler · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Pinterest Pinterestbot · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Moz site crawler rogerbot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Semrush crawler SemrushBot · tool | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Slack unfurl Slackbot-LinkExpanding · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Sogou Sogou web spider · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Soso Sosospider · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| X link preview Twitterbot · social | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Naver Yeti · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Yisou YisouSpider · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
| Youdao YoudaoBot · regional | open | on or before 2026-09-24 19:13 UTC proof · reading | 0 | — |
State = what robots.txt says for that crawler’s own group at the path tested. A date range means the change happened between those two readings; scan more often to narrow it. Conduct = how the crawler really behaved where verified requests are logged (grades); it is not measured on denverpost.com unless the site reports its traffic. Record covers 2026-09-24 onward. Generated 2026-11-12 16:58 UTC.