CrawlCheck

Guides · 2026-10-03 · By · 1 view

AI crawler monitoring vs uptime monitors: what each one catches

UptimeRobot, Visualping and CrawlCheck Watch compared: what each watches, how often, and the AI crawler failures a green uptime check never shows.

Uptime monitors such as UptimeRobot (free for 50 monitors, Solo $12/month) confirm a site answers; change monitors such as Visualping (free for 150 checks, Personal $14/month) flag visible page changes. Neither sees an AI crawler being refused while browsers are served. CrawlCheck Watch ($29/month for three domains) re-measures twice a day what GPTBot, ClaudeBot and PerplexityBot are served and alerts when it changes.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Most sites already have a monitor: an uptime check that pings the homepage, or a change monitor that emails when a page changes. Both watch the site as a browser sees it. AI crawler access breaks in ways neither sees: a CDN update starts challenging GPTBot, a security plugin refuses ClaudeBot, a robots.txt edit drops a rule for one crawler. The site stays up and looks the same, and AI systems quietly stop reading it.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

What each monitor watches #

UptimeRobotVisualpingCrawlCheck Watch
WatchesWhether the site answers, keywords present, SSL and DNSVisual or text changes on chosen pagesWhat each AI crawler identity is served, robots.txt per crawler, machine files, findings
Checks asA monitoring client; custom headers on paid plansA browserCrawler identities including GPTBot, ClaudeBot and PerplexityBot, plus a browser
FrequencyEvery 5 minutes free; down to 15 seconds on ScaleEvery 60 minutes free; down to 2 minutesTwice a day
Alerts on AI crawler refusalNoNoYes: webhook when something crawler-visible moves
Free plan50 monitors150 checks a month, 5 pagesUnlimited one-off scans
Entry paidSolo $12/moPersonal $14/mo$29/mo for 3 domains

Sources: UptimeRobot pricing, Visualping pricing, CrawlCheck pricing.

Failures only an AI crawler monitor catches #

FailureUptime monitorChange monitorShare of scanned sites (live)
AI crawler refused while browsers are servedSite reads as upNo visible change5.8%
Challenge page served with status 200Reads as upOnly if the challenge is visible to the monitor0.2%
A cache serving a stale copyReads as upOften missed13.5%
robots.txt groups dropping rules for named crawlersNot checkedNot checked14%

Why one audit is not enough: your site was clean the day you audited it.

How crawler access breaks when nobody touched the site #

The failures in the table above rarely come from someone deciding to block AI crawlers. They come from changes made for other reasons:

  • A CDN or firewall rule update. Bot-management rules are updated by the provider, and a new rule can start challenging a crawler that was allowed last week.
  • A security plugin update. WordPress security plugins block by user-agent and request rate, and an update can change both lists.
  • An SEO plugin rewriting robots.txt. A settings change regenerates the file, and a group for a named crawler can lose rules it used to inherit. See why naming a crawler drops your Disallow rules.
  • A cache holding an old copy. The site is fixed, but crawlers are served the cached version for hours or days.
  • A host migration. New servers, new firewall defaults, no one rechecks crawler access.

In each case the homepage still loads for a person and for an uptime check, and the visible page has not changed, so neither kind of monitor has anything to report.

A basic check you can run yourself #

Before paying for anything, a scheduled job can catch the worst case. Fetch the homepage and robots.txt with a browser user-agent and with each crawler’s, and alert when the status codes differ or the crawler’s copy is much smaller:

for ua in "Mozilla/5.0 Chrome/128" "GPTBot/1.2" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
  curl -s -o /dev/null -w "$ua %{http_code} %{size_download}\n" -A "$ua" https://example.com/
done

This has real limits. The strings above are shortened; the real crawlers send longer user-agents, and some rules match the full string. More importantly, a test from your own server is not the real crawler: a firewall that checks a crawler’s published IP ranges can allow your test and still refuse the real one, or the reverse. The same limit applies to CrawlCheck, which fetches from its own network under each crawler’s user-agent. The only direct evidence of what a verified crawler received is the site’s own access log. Where a site posts that log to CrawlCheck, the report shows verified, forged and unverifiable crawler requests side by side.

What a useful alert contains #

  • Which identity changed, not just “the site changed”: GPTBot refused while ClaudeBot is fine is a different fix from everything refused.
  • Before and after, with status codes, sizes and the robots.txt rule that applies.
  • When it changed, to the half day, so it can be matched to a deploy or a plugin update.
  • A receipt, so the change can be shown to a host or a CDN provider. CrawlCheck seals its change receipts daily so the time of the reading can be checked later.

Where CrawlCheck is weaker #

CrawlCheck re-measures twice a day, not every few seconds, so it is no replacement for uptime monitoring: if the site goes down at 2 PM you want to know at 2 PM. It also does not watch arbitrary pages for visual change. Keep UptimeRobot or similar for uptime, a change monitor for pages you care about, and add CrawlCheck Watch for AI crawler access, which neither covers.

A monitoring setup that covers all three #

  • Uptime: a free UptimeRobot monitor on the homepage and key pages.
  • Content: a change monitor on pricing and service pages.
  • AI access: CrawlCheck Watch on up to three domains for $29 a month, with webhook alerts and change receipts sealed daily.
Share of scanned sites with each finding in this articlerobots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%Bot challenge served as a 200 page0.2%Share of scanned sites with eachfinding in this articlerobots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%Bot challenge served as a 200 page0.2%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Can I monitor AI crawler access myself without a paid tool?

A scheduled job can catch the worst case: fetch the homepage and robots.txt with a browser user agent and with each crawler's, and alert when the status codes differ or the crawler's copy is much smaller. It has limits. Real crawlers send longer user-agent strings, and a firewall that checks a crawler's published IP ranges can treat your test differently from the real crawler. Only the site's access log shows what a verified crawler received.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works