CrawlCheck
Crawl test · free, no signup

Crawl test: can search and AI crawlers fetch your site?

In one sentence

A crawl test requests a site the way search and AI crawlers do and records what each one was served: the status, the redirects, the robots.txt rules and the words. It shows whether a crawler can fetch the site at all, before any question of ranking or being quoted.

Type a domain. The free test fetches the homepage as Googlebot, Bingbot and the major AI crawlers beside a normal browser, reads robots.txt and the sitemap, follows the links the homepage points to, and names every crawler or page that was refused, redirected or served less.

First scan free with your email, then 3 credits ($3) a scan. Nothing installed, nothing changed on your site. The same scan as the homepage - about 30 read-only requests - and the same report, opened at the sections that answer this question.

How often a crawler cannot fetch the site
An answer engine is refusedANSWER_ENGINE_REFUSED5.8%Every crawler refused alikeUNIFORM_REFUSAL16%A challenge page served as 200CHALLENGE_SERVED_2000.3%A broken internal linkBROKEN_INTERNAL_LINK2.7%Share of 9,144 scans in the public dataset that carried the finding. Live; the same figures as /data.

Is your site one of them? Check it free → All 82 rates →

Measured

What the scan measures for this

Every crawler, side by side

The status, redirects and bytes Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and the other crawlers received, beside a browser. Refused, challenged and served-less are named per crawler.

See this section on a real report: Crawler permissions →

robots.txt and the sitemap

Whether robots.txt answers 200, which group applies to each crawler, and whether a sitemap is declared and can be read.

See this section on a real report: Sitemap →

Internal links

Links from the homepage that are broken or redirect, and pages it links to that the sitemap never lists.

See this section on a real report: Internal links →

Readable without JavaScript

Whether the words are in the HTML a crawler receives or appear only after scripts run.

See this section on a real report: Page weight & text →

For agents

What an agent is told about the same site

An agent does not read the report. It asks one endpoint before it reads, cites, connects or transacts, and gets a signed answer built from the same measurement.

1 · See it

The agent view

The answer an agent receives about any domain, rendered for a person: each action’s decision and the reason.

Agent view

2 · Call it

One endpoint

GET /api/v1/resolve?domain= returns the signed answer; preflight returns a decision and a receipt for one action.

QuickstartResolve spec

3 · Check it

Without trusting us

Every answer is Ed25519-signed and logged; it verifies offline with the published key.

Signing keysProof

Workflow

The workflow

  1. Run the free crawl testNo account and nothing installed. Read-only requests, about twenty seconds.
  2. Read the refusals firstA crawler that is refused or challenged gets nothing else on the list, so access is fixed before content.
  3. Then the paths a crawler followsrobots.txt, the sitemap and internal links decide which pages a crawler finds after the homepage.
  4. Re-test after the fixThe second test is the dated evidence that the crawler now gets the page.
Free

Free tools for this

Each one is a single request to the domain you name, answered as text or a download. Put the domain on the end of the URL, or use the form on the tools page.

Reading

Read next

Start from what you are checking

You cannot pay to improve a score. A licence adds the lists behind the counts and the record over time - never a better grade.

Questions about this page

QWhat is a crawl test?
A check that requests your site as crawlers do and records what each one received. It answers whether Googlebot, Bingbot and the AI crawlers can fetch the pages, and which of them are refused, redirected or served less than a browser.
QHow do I test if my site can be crawled?
Type the domain into the free crawl test. It fetches the homepage as each named crawler and as a browser, reads robots.txt and the sitemap, and follows the homepage's links, then lists every refusal and broken path in the order to fix them.
QWhy can a browser open my site when a crawler cannot?
Firewalls, bot rules and challenge pages often treat a crawler's name differently from a browser, and robots.txt can disallow one crawler but not another. A site that looks fine in your browser can still answer a crawler with a 403, a challenge page or an empty shell.
QDoes a crawl test use the real Googlebot?
No. It sends each crawler's user agent from CrawlCheck's address, which shows how your site treats a request carrying that name. A rule that verifies Google by IP can answer the real Googlebot differently, so confirm a Googlebot refusal with URL Inspection in Search Console.
QIs the crawl test free?
Yes. The test, the grade and the fix list are free with no signup.