How sites behave for agents, and how to check your own.

2026-10-032 views · 8 min

DMARC p=none to p=quarantine without losing mail: a guide

The sequence for tightening DMARC safely: inventory every sender, prove alignment from received headers, create the report mailbox first, then raise the policy in stages.

Read the guide →

2026-10-031 view · 4 min

CSP rollout: from report-only to enforced without a break

How to introduce a Content-Security-Policy without breaking the page: write it from what loads, ship it report-only with a sink that works, read the reports, then enforce.

CSP_REPORTING_BROKEN0.3%

Read the guide →

2026-10-035 min

HSTS audit: missing header, preload ineligible, removed

How to read Strict-Transport-Security from outside, what the browser preload list adds and takes away, and why CrawlCheck never tells you to preload.

HSTS_MISSING11.7%HSTS_PRELOAD_INELIGIBLE4%HSTS_PRELOAD_REMOVED0.4%

Read the guide →

2026-10-083 views · 9 min

How to block AI crawlers and agents, and what each block costs you

Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that get made as one, and the one people regret is rarely the one they intended.

AI_OPTOUT_SET4.5%ROBOTS_RULES_SHADOWED12.8%ANSWER_ENGINE_REFUSED5.8%+2

Read the guide →

2026-10-033 views · 9 min

Cloudflare AI Crawl Control vs CrawlCheck: block vs measure

One controls AI crawler access at Cloudflare's edge; the other measures, from outside, what each crawler is actually served. What each does, where each is weaker, and the routine that uses both.

ANSWER_ENGINE_REFUSED5.8%

Read the guide →

2026-10-032 views · 8 min

How to test robots.txt for GPTBot and ClaudeBot: 5 tools

Policy testers tell you whether your file allows a crawler; only a delivery check tells you whether the crawler gets the page. Five tools compared, the two-minute test, and the mistakes we see most.

ANSWER_ENGINE_REFUSED5.8%

Read the guide →

2026-10-032 views · 4 min

Best web scraping APIs compared: ScrapingBee, ZenRows and more

ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl and Apify compared on free tiers, entry plans and what a JavaScript or protected page really costs, with where CrawlCheck fits.

ANSWER_ENGINE_REFUSED5.8%

Read the guide →

2026-10-031 view · 3 min

How to check whether a GPTBot or ClaudeBot request in your logs is real

A user-agent is a claim. The address is the evidence. Here is how to adjudicate every crawler line in an access log in one paste, and how to read the three outcomes.

Read the guide →

2026-10-031 view · 10 min

Best AI web crawler on a budget: real costs compared

Five budget crawlers, their published prices converted into the pages $19 actually buys, the proxy and retry costs no pricing page shows, and the one question none of them can answer: whether ChatGPT and Claude can read your own site.

ANSWER_ENGINE_REFUSED5.8%CRAWLER_SERVED_LESS0.5%CHALLENGE_SERVED_2000.6%+3

Read the guide →

2026-10-031 view · 4 min

Open-source crawlers for LLMs: Crawl4AI vs Firecrawl vs Scrapy

Crawl4AI, self-hosted Firecrawl, Scrapy and Jina Reader compared on licence, browser rendering and output, plus the AI access question none of them answers.

CONTENT_NEEDS_JAVASCRIPT1.2%PAGE_IS_MOSTLY_CODE27.6%

Read the guide →

2026-10-085 min

Do AI crawlers and agents respect robots.txt?

The named ones largely do. The problem in practice is almost never disobedience — it is that the file says one thing and the network in front of it does another, and nothing reports the conflict.

ROBOTS_NOT_2005.9%CHALLENGE_SERVED_2000.6%ROBOTS_IS_HTML0.2%+9

Read the guide →

2026-10-089 min

Do AI crawlers and agents render JavaScript?

Assume not, and the question becomes much simpler: does your content exist in the response the server sends, before anything runs. Here is how to check in one request.

PAGE_IS_MOSTLY_CODE27.6%NO_SITEMAP_FOUND10.6%ROBOTS_NO_SITEMAP8.8%+5

Read the guide →

2026-10-0810 min

The AI crawler and agent list: what each one is for, and which can be verified

Twelve names that show up in server logs, what each is actually doing, and the number that changes how you read all of them — how much of that traffic is not who it says it is.

Read the guide →

2026-10-036 min

How to check whether AI crawlers and agents can read your site

Five requests, ten minutes, no account required. What each one catches, and why opening the page in your browser is the one test that cannot find any of it.

ROBOTS_NOT_2005.9%STALE_CACHE_SERVED17.4%ANSWER_ENGINE_REFUSED5.8%+4

Read the guide →

2026-10-034 min

Crawler delivery: refused, redirected away, or served less

How to tell whether a search or answer crawler receives the page a person receives: fetch as each crawler and as a browser control, compare, and read what each difference means.

ANSWER_ENGINE_REFUSED5.8%CRAWLER_REDIRECTED_AWAY0.3%CRAWLER_SERVED_LESS0.5%+2

Read the guide →

2026-10-035 min

How to verify Googlebot, Bingbot and AI crawlers by IP

A user-agent string is a claim anyone can type. Verification is reverse DNS with a forward confirm, or a match against the operator's published IP list. Which operators support which method, the file for each, and what to do with a crawler that publishes neither.

Read the guide →

2026-10-035 min

Screaming Frog vs Sitebulb vs CrawlCheck for AI crawler audits

Two deep technical crawlers and one AI crawler audit, compared on what each fetches, which identities it tests, robots.txt handling and price, with where CrawlCheck falls short.

ANSWER_ENGINE_REFUSED5.8%CRAWLER_SERVED_LESS0.5%ROBOTS_RULES_SHADOWED12.8%+2

Read the guide →

2026-10-034 min

How to verify GPTBot, ClaudeBot and PerplexityBot: tools compared

Where OpenAI, Anthropic, Perplexity and Google publish their crawler addresses, the tools that check traffic against them (CrawlCheck, Cloudflare, manual), and why unverifiable is its own answer.

Read the guide →

2026-10-034 min

PageSpeed Insights and Lighthouse vs an AI crawler audit

What Google's PageSpeed Insights and Lighthouse measure, what a perfect score misses about AI crawlers, and where CrawlCheck falls short on speed.

ANSWER_ENGINE_REFUSED5.8%PAGE_IS_MOSTLY_CODE27.6%MACHINE_CHAIN_HEAVY17.2%+2

Read the guide →

2026-10-034 min

Monitoring AI crawler access vs uptime and change monitors

UptimeRobot, Visualping and CrawlCheck Watch compared: what each watches, how often, and the AI crawler failures a green uptime check never shows.

ANSWER_ENGINE_REFUSED5.8%CHALLENGE_SERVED_2000.6%STALE_CACHE_SERVED17.4%+1

Read the guide →

2026-08-285 min

Is Cloudflare blocking AI crawlers on your site? What changes on 15 September 2026

A default is about to change, and the group it reaches hardest is the group that never opened the setting. How to find out which side of it you are on before the date, not after.

ANSWER_ENGINE_REFUSED5.8%AI_OPTOUT_SET4.5%CHALLENGE_SERVED_2000.6%+3

Read the guide →

2026-10-035 views · 6 min

AI visibility tools compared: what each one actually measures

Every tool in this category runs the same loop. The comparisons all sort on price and engine coverage. The difference that decides whether a tool can help you is which half of the problem it can see.

STALE_CACHE_SERVED17.4%ANSWER_ENGINE_REFUSED5.8%

Read the guide →

2026-10-081 view · 5 min

Free AI visibility checkers: what the free tier actually measures

Most of them are lead magnets, which is fine — a lead magnet can still be a good instrument. The question is whether the check is hard enough to fail, and a surprising number are not.

NO_LLMS_TXT31.7%STALE_CACHE_SERVED17.4%ANSWER_ENGINE_REFUSED5.8%+3

Read the guide →

2026-10-031 view · 5 min

SEO audit tools compared: what each one actually fetches

Site crawlers, platform audits and Lighthouse are three different instruments pointed at three different things. The gaps between them are where defects live, and one of them cannot support a finding from a single run.

STALE_CACHE_SERVED17.4%ANSWER_ENGINE_REFUSED5.8%CRAWLER_SERVED_LESS0.5%

Read the guide →

2026-10-089 min

GEO, AEO, LLM SEO and the agentic web: what the terms mean and which parts can be measured

Three names for one shift, sold as three disciplines. The distinctions are mostly marketing; the useful split is between the parts of the work that produce a checkable number and the parts that do not.

ANSWER_ENGINE_REFUSED5.8%AI_OPTOUT_SET4.5%STALE_CACHE_SERVED17.4%+2

Read the guide →

2026-10-085 min

How to measure AI and agent visibility without trusting a vendor's score

Three measurements, in order, each of which you can reproduce yourself. Two of them are free, and the one people buy first is the one that depends on the other two being clean.

ROBOTS_NOT_2005.9%STALE_CACHE_SERVED17.4%ANSWER_ENGINE_REFUSED5.8%+2

Read the guide →

2026-10-085 min

AI website audit tools compared: Semrush, Ahrefs, Screaming Frog, Sitebulb and CrawlCheck

“AI website audit” is sold as one thing and means two: a tool that uses AI to read its own findings, and a tool that audits what AI crawlers and agents actually receive from your site. Which of the five does which, what each costs, and how to pick for a small business, an e-commerce site or an agency.

Read the guide →

2026-10-035 min

Best AI visibility tools compared: prices and what they track

Profound, Peec AI, Otterly.AI, Rankscale, Knowatoa, AthenaHQ, Scrunch, Ahrefs Brand Radar, Semrush and CrawlCheck side by side: published prices, engines covered, and the access question prompt trackers cannot answer.

ANSWER_ENGINE_REFUSED5.8%CRAWLER_SERVED_LESS0.5%CHALLENGE_SERVED_2000.6%+2

Read the guide →

2026-08-285 min

How to rank in ChatGPT: the part that is actually in your control

There is no position three. There is retrieval, and there are four conditions your site has to satisfy before anything you publish is eligible to be used at all.

ANSWER_ENGINE_REFUSED5.8%STALE_CACHE_SERVED17.4%PAGE_IS_MOSTLY_CODE27.6%+2

Read the guide →

2026-10-032 views · 8 min

How to find other websites using your brand name online

A six-step routine: sweep the domains, pull registration dates from the registry, read what the other site does, publish a factual notice, tell answer engines, and keep watching.

Read the guide →

2026-08-281 view · 5 min

Entity SEO: what a machine or AI agent has to resolve before it can cite or act on you

Being read is not being identified. If a system cannot tell you apart from a similar business two towns over, it will either name the wrong one or name nobody, and both look identical from your side.

ENTITY_COLLISION3.3%ENTITY_NO_COORDINATES5.4%

Read the guide →

2026-10-035 min

JSON-LD @id and IRI errors: what breaks and how to fix it

Six identifier defects in JSON-LD that no vocabulary validator reports: blank nodes, path @ids that 404, Unicode normalisation, raw IDN hosts, encoding drift and rival roots.

Read the guide →

2026-10-035 min

NAP consistency audit: phone, address and coordinates

The checks you can run on name, address and phone: what the page declares, what directories reached by the phone number say back, and the five findings that follow.

ENTITY_NO_COORDINATES5.4%GEO_COUNTRY_MISMATCH0.2%NAP_UNDECLARED_LISTING4.4%+2

Read the guide →

2026-10-035 min

LocalBusiness schema for service-area businesses, checked

Google requires two properties. The other ten are where local records break: coordinates with the wrong sign, a name that differs from the listings, two nodes claiming one phone number. How to write the node, and the four checks that read it.

ENTITY_NO_COORDINATES5.4%GEO_COUNTRY_MISMATCH0.2%ENTITY_COLLISION3.3%+1

Read the guide →

2026-10-034 min

Rich Results Test vs Schema Markup Validator vs CrawlCheck

Three free structured-data tools, three questions: Google rich-result eligibility, schema.org validity, and whether the markup describes one coherent business machines can resolve.

ENTITY_COLLISION3.3%ENTITY_NO_COORDINATES5.4%NAP_UNDECLARED_LISTING4.4%+1

Read the guide →

2026-10-034 views · 11 min

How to audit an XML sitemap: status, content type, canonical URLs and stale entries

A sitemap can return 200, open in a browser and still be unusable. The audit has two layers, verify the document and then verify every URL it declares, and most audits stop after the first.

NO_SITEMAP_FOUND10.6%ROBOTS_NO_SITEMAP8.8%DECLARED_SITEMAP_BROKEN1.2%+6

Read the guide →

2026-09-172 views · 8 min

Does llms.txt actually do anything?

The honest answer has two halves, and the half nobody publishes is the one that decides it. Here is what is verifiable, what is not, and the test that would settle it for your own domain.

NO_LLMS_TXT31.7%MACHINE_FILE_CACHE_STALE0.5%MACHINE_FILE_CACHE_SPLIT0.1%+6

Read the guide →

2026-10-039 min

How to write an llms.txt, and how to check it is actually being served

The file takes twenty minutes. Confirming that the version a crawler receives is the version you wrote takes two more, and it is the step almost everyone skips.

MACHINE_FILE_CACHE_STALE0.5%MACHINE_FILE_CACHE_SPLIT0.1%STALE_CACHE_SERVED17.4%+6

Read the guide →

2026-10-038 min

HTTP status codes for crawlers: what a 200 can hide, when a 403 is not a block, and the one control that tells them apart

The status line is the smallest part of a response. Challenge pages ship as 200, walls answer every path the same, and a crawler can get a page with half the words the browser got. Here is how to read the second half of the answer.

CHALLENGE_SERVED_2000.6%CHALLENGE_PINNED_AT_EDGE0.4%UNIFORM_REFUSAL6.8%+2

Read the guide →

2026-10-037 min

Canonical URL audit: one page, one address, and the three ways a site publishes it twice

The canonical tag is one line and most audits stop at it. The audit that matters counts how many addresses answer 200 for the same bytes, whether the sitemap agrees, and whether the hostname is a site at all.

HOST_DUPLICATE_2005.6%HOMEPAGE_REDIRECTS_OFF_HOST2.2%DOMAIN_IS_ALIAS0.7%+1

Read the guide →

2026-10-036 min

robots.txt 404, 403, 429 and 503: what each tells a crawler

The status code on robots.txt decides how a crawler treats the whole site before it reads a single rule. A 404 means crawl everything. A 503 means crawl nothing. A 403 is where crawlers disagree.

ROBOTS_IS_CATCHALL0.7%ROBOTS_IS_HTML0.2%ROBOTS_CTYPE0.1%+2

Read the guide →

2026-10-035 min

Cached robots.txt and llms.txt: stale, split, or blocked

Why one fetch cannot tell whether a machine file is healthy, the two-fetch method that can, the four findings it produces, and the purge order that clears them.

MACHINE_FILE_CACHE_STALE0.5%MACHINE_FILE_CACHE_SPLIT0.1%ORIGIN_BLOCKED_BEHIND_CACHE0.2%+1

Read the guide →

2026-10-034 min

Staging site leak: how a staging host ends up in live pages

How a managed host's staging hostname gets into production page source, what a leaked staging copy costs, how to find it from outside and how to close it without breaking the workflow.

STAGING_LEAK0.1%HOST_DUPLICATE_2005.6%

Read the guide →

2026-10-034 min

Best llms.txt generators and validators compared

Firecrawl, Yoast, Mintlify and CrawlCheck compared on generating llms.txt, keeping it current and checking it actually reaches AI crawlers.

NO_LLMS_TXT31.7%MACHINE_FILE_CACHE_STALE0.5%

Read the guide →

2026-10-026 min

robots.txt user-agent groups: why named bots skip your rules

A crawler obeys exactly one group in robots.txt, the most specific one that names it. Add a group for GPTBot with Allow: / and every Disallow under User-agent: * stops applying to GPTBot. How to see it, and how to write the file so it cannot happen.

ROBOTS_RULES_SHADOWED12.8%ROBOTS_IS_CATCHALL0.7%

Read the guide →

2026-10-025 min

Content Signals in robots.txt: search, ai-input, ai-train

Content-Signal is a robots.txt line that states what you allow done with your content after it is fetched: search, AI input, AI training. What each value means under Cloudflare's published policy, why it does not block anything, and the one combination that costs you citations.

AI_OPTOUT_SET4.5%

Read the guide →

2026-09-167 min

Meta robots, X-Robots-Tag and robots.txt: the three permission layers and the rule placed in the wrong one

Three mechanisms share a name and answer three different questions at three different moments. Most robots failures are a directive placed in a layer that cannot read it, starting with the noindex behind a Disallow.

ROBOTS_RULES_SHADOWED12.8%ROBOTS_DISALLOW_ALL1.4%ROBOTS_IS_HTML0.2%+2

Read the guide →

2026-10-076 views · 10 min

YouTube SEO case study: a dormant channel with 81 videos rewritten, measured before and after

Terrarium Station’s channel had not uploaded in 19 months and was still growing 42% a year in search. Every title and description was rewritten in two dated batches. The baseline is published so the after cannot be moved; the result is in: search views up 19% against a flat year-earlier control, with the flat and falling rows published beside it.

Read the guide →

2026-10-032 views · 9 min

How to find broken internal links and dead anchor links

A link audit is an inventory, a fetch, a classification and a second pass over in-page anchors. The classification is where most audits go wrong, and the anchors are where most audits never look.

Read the guide →

2026-10-032 views · 9 min

How to test text color contrast on hover and keyboard focus

Static contrast checkers measure a page at rest. The failures people hit happen on hover, on keyboard focus and after scripts run. Six steps to test them, and the four causes you will find.

Read the guide →

2026-10-021 view · 5 min

Hreflang audit: return links, x-default and common errors

Hreflang tells a search engine which language or regional version of a page to show. It only counts when every version links back. The rules, the six errors that make Google ignore a set, and how to audit it from the outside.

Read the guide →

2026-10-096 min

How an AI agent should verify a website before acting on it

Before an agent reads, cites, connects to or transacts with a domain, six questions need answers it can prove: what the site claims, what it serves, who controls it, what is safe to do, what changed and what evidence backs each answer.

Read the guide →

2026-10-084 min

The agentic web for site owners: permission, delivery, identity and tools

AI systems are starting to act on websites, not just read them. What that changes for a site, the four questions an agent needs answered before it acts, and what you can check this week.

Read the guide →

2026-10-083 min

How to check an MCP server before an AI agent connects to it

An agent that connects to an MCP server reads every tool description as guidance, with no person in between. Five checks, two curl commands and the red flags we found in 25,983 registry endpoints.

Read the guide →

2026-10-083 min

Can an AI agent use your contact form? A two-minute test

When a person asks an assistant to request a quote, the task ends at your form. Five checks in an ordinary browser tell you whether an agent can finish it, and what to offer when it cannot.

Read the guide →

2026-10-034 min

How to read a CrawlCheck report: the grade, the three stages, the fix list and the evidence

A report is six views over one record. This is what each one is for, what the numbers mean, and where the bytes behind every finding live.

Read the guide →

2026-10-035 min

Text-to-HTML ratio: when a page is mostly code

Text-to-HTML ratio is not a ranking factor, but a page whose visible words are under five percent of its bytes is a page where crawlers download framework state, inline CSS and SVG before they reach a sentence. How to measure it, what the threshold means, and what to remove first.

PAGE_IS_MOSTLY_CODE27.6%

Read the guide →

2026-10-034 min

Web Bot Auth: how signed crawler and agent requests are verified

IP lists stop working when crawlers share cloud addresses. Web Bot Auth has the crawler sign each request with a key it publishes, using HTTP Message Signatures. The three headers, the key directory, how a site verifies, and what a signature does not prove.

Read the guide →

2026-10-035 min

Markdown for agents: serving text/markdown on request

An agent that sends Accept: text/markdown can be given the page's words without the scripts, styles and navigation. How content negotiation works for this, the headers that must come back, the caching trap, and what an intermediary may add on your behalf.

Read the guide →

2026-10-035 min

IndexNow: what it does, who reads it, how to check it

IndexNow lets a site tell search engines a URL changed instead of waiting to be recrawled. One submission reaches every participating engine. Which engines those are, the key file, the response codes, and what a 200 does and does not mean.

Read the guide →

2026-10-035 min

security.txt (RFC 9116): fields, expiry and location

security.txt tells someone who found a vulnerability where to report it. RFC 9116 requires two fields and one location. What goes in the file, the expiry rule most files break within a year, and how to check yours from outside.

Read the guide →

2026-10-035 min

Core Web Vitals field data for sites CrUX does not cover

PageSpeed Insights says "no data" for most small sites because the Chrome UX Report needs more real users than they have. Lab scores are not a substitute. How to collect your own field data, the thresholds to judge it against, and the sample size below which a percentile means nothing.

Read the guide →

2026-10-034 min

Agent readiness checkers: Is It Agent Ready? vs CrawlCheck

Cloudflare's Is It Agent Ready? and CrawlCheck compared: which agent standards each checks, declared versus working, the access layer underneath, and where each falls short.

ANSWER_ENGINE_REFUSED5.8%ROBOTS_RULES_SHADOWED12.8%AI_OPTOUT_SET4.5%

Read the guide →