Crawl test · free, no signup

# Crawl test: can search and AI crawlers fetch your site?

In one sentence

A crawl test requests a site the way search and AI crawlers do and records what each one was served: the status, the redirects, the robots.txt rules and the words. It shows whether a crawler can fetch the site at all, before any question of ranking or being quoted.

Type a domain. The free test fetches the homepage as Googlebot, Bingbot and the major AI crawlers beside a normal browser, reads robots.txt and the sitemap, follows the links the homepage points to, and names every crawler or page that was refused, redirected or served less.
Domain
Check my site

**First scan free with your email, then 3 credits ($3) a scan. Nothing installed, nothing changed on your site.** The same scan as the homepage - about 30 read-only requests - and the same report, opened at the sections that answer this question.
How often a crawler cannot fetch the site
[Is your site one of them? Check it free →](/) [All 82 rates →](/data)

Measured

## What the scan measures for this

### Every crawler, side by side

The status, redirects and bytes Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and the other crawlers received, beside a browser. Refused, challenged and served-less are named per crawler.

[See this section on a real report: Crawler permissions →](/demo#agents)

### robots.txt and the sitemap

Whether robots.txt answers 200, which group applies to each crawler, and whether a sitemap is declared and can be read.

[See this section on a real report: Sitemap →](/demo#sitemap)

### Internal links

Links from the homepage that are broken or redirect, and pages it links to that the sitemap never lists.

[See this section on a real report: Internal links →](/demo#links)

### Readable without JavaScript

Whether the words are in the HTML a crawler receives or appear only after scripts run.

[See this section on a real report: Page weight & text →](/demo#payload)

For agents

## What an agent is told about the same site

An agent does not read the report. It asks one endpoint before it reads, cites, connects or transacts, and gets a signed answer built from the same measurement.

1 · See it

### The agent view

The answer an agent receives about any domain, rendered for a person: each action’s decision and the reason.

[Agent view](/tools/agent-view)

2 · Call it

### One endpoint

GET /api/v1/resolve?domain= returns the signed answer; preflight returns a decision and a receipt for one action.

[Quickstart](/docs/quickstart)[Resolve spec](/spec/resolve)

3 · Check it

### Without trusting us

Every answer is Ed25519-signed and logged; it verifies offline with the published key.

[Signing keys](/keys)[Proof](/proof)

Workflow

## The workflow

- **Run the free crawl test**No account and nothing installed. Read-only requests, about twenty seconds.
- **Read the refusals first**A crawler that is refused or challenged gets nothing else on the list, so access is fixed before content.
- **Then the paths a crawler follows**robots.txt, the sitemap and internal links decide which pages a crawler finds after the homepage.
- **Re-test after the fix**The second test is the dated evidence that the crawler now gets the page.

Free

## Free tools for this

Each one is a single request to the domain you name, answered as text or a download. Put the domain on the end of the URL, or use the form on the [tools page](/tools).

[Googlebot simulatorYour page as Googlebot and Googlebot Smartphone.](/googlebot-simulator)[Test robots.txt for AI crawlersWhich rule applies to each named crawler.](/robots-txt-for-ai-crawlers)[AI searchability checkCan AI find, fetch, read and repeat the site?](/ai-searchability)[Fake Googlebot checkerIs the Googlebot in your logs real?](/tools/googlebot-check)

Reading

## Read next

- [How to check if AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site)
- [HTTP status codes: what a crawler actually gets](/guides/http-status-codes-what-a-crawler-actually-gets)
- [Crawl budget: how much of your site a crawler reads](/guides/crawl-budget-how-much-of-your-site-a-crawler-reads-in-50-fetches)

## Start from what you are checking

[llms.txt auditdoes yours resolve, parse and match the site it describes?](/llms-txt-audit)[robots.txt for AI crawlerswhat yours actually permits](/robots-txt-for-ai-crawlers)[Entity map validatordoes your entitymap.json parse, and does every claim in it resolve?](/entity-map-validator)[SEO audit toolthe technical checks, plus what AI crawlers actually receive](/seo-audit-tool)[Technical SEO auditcrawl access, sitemap, hosts, links and speed, checked in one scan](/technical-seo-audit)[Local SEO auditdoes your site say who you are, where, and does every listing agree?](/local-seo-audit-tool)[SEO audit reports for agenciesyour name on the report, the proof still attached](/white-label-seo-audit)[What is a website auditWhat is a website audit? What it checks, what it costs, and how to run one free](/what-is-a-website-audit)[What is an SEO auditWhat is an SEO audit? The checks that matter, in the order to fix them](/what-is-an-seo-audit)[Googlebot simulatorsee your page the way Googlebot gets it](/googlebot-simulator)[AI searchabilitycan ChatGPT, Claude, Perplexity and Google find, fetch, read and repeat your site?](/ai-searchability)[AEO auditcan ChatGPT, Claude, Perplexity and Google reach, read and quote your site?](/aeo-audit)[Website auditSEO, security headers, accessibility and speed, checked in one free scan](/website-audit)[Local SEO auditname, phone, address and listings, in the order to fix them](/local-seo-audit)[SEO NAP checkername, address and phone, on the site and on the directories](/seo-nap-checker)

**You cannot pay to improve a score.** A licence adds the lists behind the counts and the record over time - never a better grade.

## Questions about this page

#### Q What is a crawl test?

A check that requests your site as crawlers do and records what each one received. It answers whether Googlebot, Bingbot and the AI crawlers can fetch the pages, and which of them are refused, redirected or served less than a browser.

#### Q How do I test if my site can be crawled?

Type the domain into the free crawl test. It fetches the homepage as each named crawler and as a browser, reads robots.txt and the sitemap, and follows the homepage's links, then lists every refusal and broken path in the order to fix them.

#### Q Why can a browser open my site when a crawler cannot?

Firewalls, bot rules and challenge pages often treat a crawler's name differently from a browser, and robots.txt can disallow one crawler but not another. A site that looks fine in your browser can still answer a crawler with a 403, a challenge page or an empty shell.

#### Q Does a crawl test use the real Googlebot?

No. It sends each crawler's user agent from CrawlCheck's address, which shows how your site treats a request carrying that name. A rule that verifies Google by IP can answer the real Googlebot differently, so confirm a Googlebot refusal with URL Inspection in Search Console.

#### Q Is the crawl test free?

Yes. The test, the grade and the fix list are free with no signup.

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [bobsteelechevy.com](/registry/bobsteelechevy.com) · [dragonora-fragrance-boutique.lovable.app](/registry/dragonora-fragrance-boutique.lovable.app) · [jammiwoo.gr](/registry/jammiwoo.gr) · [patreon.com](/registry/patreon.com) · [whatsapp.com](/registry/whatsapp.com) · [agentalog.com](/registry/capabilities/agentalog.com) | **Glossary:** [hedged answer](/glossary/hedged-answer) | **Directory:** [Browserless](/directory/browserless)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
