CrawlCheck

Agent view · Guides · 2026-10-03

AI crawler monitoring vs uptime monitors: what each one catches

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes14,827
words2,096
sha-2566c39e0fe943589659dc499f1b46a4071276ad949a4e9dc87128ecc2aeed8aa2a
repr-digestsha-256=:bDng/pQ1iWWdxJnxtGpAcSdq2Umk6dyHEo7MKu7Yqio=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311493;expires=1795315093;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="5W5CK9IkZUrDkdpaqiDwWw";tag="crawlcheck-page"
signaturesig1=:KQLN2jV+KyWrvzsNWSfc/lJsc0usFZMguJhKzt5+45TEmkqRs3wJ6VXJjTcB2/Yl+TcBatiHYaDBO9zdt5i5CA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 1 view

# AI crawler monitoring vs uptime monitors: what each one catches

UptimeRobot, Visualping and CrawlCheck Watch compared: what each watches, how often, and the AI crawler failures a green uptime check never shows.

Uptime monitors such as UptimeRobot (free for 50 monitors, Solo $12/month) confirm a site answers; change monitors such as Visualping (free for 150 checks, Personal $14/month) flag visible page changes. Neither sees an AI crawler being refused while browsers are served. CrawlCheck Watch ($29/month for three domains) re-measures twice a day what GPTBot, ClaudeBot and PerplexityBot are served and alerts when it changes.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What each monitor watches
- Failures only an AI crawler monitor catches
- How crawler access breaks when nobody touched the site
- A basic check you can run yourself
- What a useful alert contains
- Where CrawlCheck is weaker
- A monitoring setup that covers all three

Most sites already have a monitor: an uptime check that pings the homepage, or a change monitor that emails when a page changes. Both watch the site as a browser sees it. AI crawler access breaks in ways neither sees: a [CDN](/glossary/cdn) update starts challenging GPTBot, a security plugin refuses ClaudeBot, a [robots.txt](/glossary/robots-txt) edit drops a rule for one crawler. The site stays up and looks the same, and AI systems quietly stop reading it.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

## What each monitor watches #

| | UptimeRobot | Visualping | CrawlCheck Watch |
| --- | --- | --- | --- |
| Watches | Whether the site answers, keywords present, SSL and DNS | Visual or text changes on chosen pages | What each AI crawler identity is served, robots.txt per crawler, machine files, findings |
| Checks as | A monitoring client; custom headers on paid plans | A browser | Crawler identities including GPTBot, ClaudeBot and PerplexityBot, plus a browser |
| Frequency | Every 5 minutes free; down to 15 seconds on Scale | Every 60 minutes free; down to 2 minutes | Twice a day |
| Alerts on AI crawler [refusal](/glossary/refusal) | No | No | Yes: webhook when something crawler-visible moves |
| Free plan | 50 monitors | 150 checks a month, 5 pages | Unlimited one-off scans |
| Entry paid | Solo $12/mo | Personal $14/mo | $29/mo for 3 domains |

Sources: [UptimeRobot pricing](https://uptimerobot.com/pricing/), [Visualping pricing](https://visualping.io/pricing), CrawlCheck [pricing](/pricing).

## Failures only an AI crawler monitor catches #

| Failure | Uptime monitor | Change monitor | Share of scanned sites (live) |
| --- | --- | --- | --- |
| AI crawler refused while browsers are served | Site reads as up | No visible change | 5.8% |
| Challenge page served with status 200 | Reads as up | Only if the challenge is visible to the monitor | 0.2% |
| A cache serving a stale copy | Reads as up | Often missed | 13.5% |
| robots.txt groups dropping rules for named crawlers | Not checked | Not checked | 14% |

Why one audit is not enough: [your site was clean the day you audited it](/blog/your-site-was-clean-the-day-you-audited-it).

## How crawler access breaks when nobody touched the site #

The failures in the table above rarely come from someone deciding to block AI crawlers. They come from changes made for other reasons:

- **A CDN or firewall rule update.** Bot-management rules are updated by the provider, and a new rule can start challenging a crawler that was allowed last week.
- **A security plugin update.** WordPress security plugins block by user-agent and request rate, and an update can change both lists.
- **An SEO plugin rewriting robots.txt.** A settings change regenerates the file, and a group for a named crawler can lose rules it used to inherit. See [why naming a crawler drops your Disallow rules](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules).
- **A cache holding an old copy.** The site is fixed, but crawlers are served the cached version for hours or days.
- **A host migration.** New servers, new firewall defaults, no one rechecks crawler access.

In each case the homepage still loads for a person and for an uptime check, and the visible page has not changed, so neither kind of monitor has anything to report.

## A basic check you can run yourself #

Before paying for anything, a scheduled job can catch the worst case. Fetch the homepage and robots.txt with a browser user-agent and with each crawler’s, and alert when the status codes differ or the crawler’s copy is much smaller:

```
for ua in "Mozilla/5.0 Chrome/128" "GPTBot/1.2" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
 curl -s -o /dev/null -w "$ua %{http_code} %{size_download}\n" -A "$ua" https://example.com/
done
```

This has real limits. The strings above are shortened; the real crawlers send longer user-agents, and some rules match the full string. More importantly, a test from your own server is not the real crawler: a firewall that checks a crawler’s published IP ranges can allow your test and still refuse the real one, or the reverse. The same limit applies to CrawlCheck, which fetches from its own network under each crawler’s user-agent. The only direct evidence of what a [verified crawler](/glossary/verified-crawler) received is the site’s own access log. Where a site posts that log to CrawlCheck, the report shows verified, forged and [unverifiable](/glossary/unverifiable) crawler requests side by side.

## What a useful alert contains #

- **Which identity changed,** not just “the site changed”: GPTBot refused while ClaudeBot is fine is a different fix from everything refused.
- **Before and after,** with status codes, sizes and the robots.txt rule that applies.
- **When it changed,** to the half day, so it can be matched to a deploy or a plugin update.
- **A receipt,** so the change can be shown to a host or a CDN provider. CrawlCheck seals its change receipts daily so the time of the reading can be checked later.

## Where CrawlCheck is weaker #

CrawlCheck re-measures twice a day, not every few seconds, so it is no replacement for uptime monitoring: if the site goes down at 2 PM you want to know at 2 PM. It also does not watch arbitrary pages for visual change. Keep UptimeRobot or similar for uptime, a change monitor for pages you care about, and add CrawlCheck Watch for AI crawler access, which neither covers.

## A monitoring setup that covers all three #

- Uptime: a free UptimeRobot monitor on the homepage and key pages.
- Content: a change monitor on pricing and service pages.
- AI access: CrawlCheck Watch on up to three domains for $29 a month, with webhook alerts and change receipts sealed daily.

Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmonitoring-ai-crawler-access-vs-uptime-and-change-monitors%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmonitoring-ai-crawler-access-vs-uptime-and-change-monitors%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=AI%20crawler%20monitoring%20vs%20uptime%20monitors%3A%20what%20each%20one%20catches)[Email](mailto:?subject=AI%20crawler%20monitoring%20vs%20uptime%20monitors%3A%20what%20each%20one%20catches&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmonitoring-ai-crawler-access-vs-uptime-and-change-monitors)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Can I monitor AI crawler access myself without a paid tool?

A scheduled job can catch the worst case: fetch the homepage and robots.txt with a browser user agent and with each crawler's, and alert when the status codes differ or the crawler's copy is much smaller. It has limits. Real crawlers send longer user-agent strings, and a firewall that checks a crawler's published IP ranges can treat your test differently from the real crawler. Only the site's access log shows what a verified crawler received.

## Related findings

[2026-10-03 Best llms.txt generators and validators compared llms.txt generators and validators compared: Yoast, Mintlify, Firecrawl and CrawlCheck, and the four failures no generator checks](/guides/best-llms-txt-generators-and-validators-compared)[2026-10-03 Agent readiness checkers: Is It Agent Ready? vs CrawlCheck Is It Agent Ready? vs CrawlCheck: agent standards checked, declared vs working capabilities, AI crawler access, scores that skip w](/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck)[2026-10-03 Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits Screaming Frog, Sitebulb, Semrush, Ahrefs and Lighthouse vs CrawlCheck: what each audit fetches, as whom, and which AI crawler pro](/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [Failure policy](/spec/failure-policy) | **Registry:** [foryouproperty.com](/registry/foryouproperty.com) · [localgtm.com](/registry/localgtm.com) · [slack.com](/registry/slack.com) · [Trade: Roofing](/registry/trade/roofing) · [stripe.dev](/registry/capabilities/stripe.dev) | **Datasets:** [Technologies: Calendly](/datasets/technologies/calendly) · [Problems: Image format mismatch](/datasets/problems/image-format-mismatch)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.