# Block SEO, advertising and monitoring crawlers # 27 crawlers, generated from https://crawlcheck.io/ai-crawlers # Paste above your existing rules. Every group here only adds Disallow: / for the crawler it names; # crawlers not named keep following your own rules. User-agent: KlaviyoAIBot User-agent: Mediapartners-Google User-agent: APIs-Google User-agent: BingPreview User-agent: Magpie-crawler User-agent: AwarioRssBot User-agent: AwarioSmartBot User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: rogerbot User-agent: DataForSeoBot User-agent: BLEXBot User-agent: VelenPublicWebCrawler User-agent: Google-CloudVertexBot User-agent: CloudflareBrowserRenderingCrawler User-agent: Cloudflare-AutoRAG User-agent: Cloudflare-AI-Search-External User-agent: Peer39_Crawler User-agent: AdsBot-Google User-agent: AmazonAdBot User-agent: AdIdxBot User-agent: Scrapy User-agent: SemrushBot-OCOB User-agent: AwarioBot User-agent: OAI-AdsBot Disallow: /