CrawlCheck

Agent view · Guides · 2026-10-03

Web scraping APIs compared: the real cost per page

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/best-web-scraping-apis-compared: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/best-web-scraping-apis-compared>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes23,550
words3,643
sha-2562959a9ef2ac145ce7acd47ebcbcc1d0474379b03be1095f0f4865ae6bf2c39ca
repr-digestsha-256=:KVmp7yrBRc56zUfry8wdBHQ3mwO+EJXw9IZa5r8sOco=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311482;expires=1795315082;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="inBYIcEVGRK_r8YFWGMplg";tag="crawlcheck-page"
signaturesig1=:JdujOEcrdShtZVNZ0q87Eyjozd5oPQE7ep7aCSHYBMCodAR2GQ0OATI4YNNXEw2htrRZsoPmMUZ+bOopeGH5CA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 15 views

# Web scraping APIs compared: the real cost per page

ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl, Apify and Browse AI converted from credits into pages, plus Crawl4AI, Scrapy and Jina Reader for free, and the one question none of them answers about your own site.

Converted into JavaScript-rendered pages, ScrapingBee Hobby ($19) gives about 15,000, ZenRows Build ($16) about 9,000 and Firecrawl Hobby ($19) 5,000; Bright Data charges $1.50 per 1,000 successful requests and Apify bills compute. Crawl4AI and Scrapy are free but cost servers and upkeep. The bill is driven by the hardest pages, not the page count. None of these tools shows whether ChatGPT or Claude can read your own site, because each is served as itself; a free CrawlCheck scan checks that.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What does a plain, JavaScript or protected page cost?
- How many pages does an entry plan actually buy?
- How do I estimate my cost before I buy?
- What costs never appear on a pricing page?
- What about free, open-source crawlers?
- Which one should I pick?
- Why can’t a scraper tell me what ChatGPT sees on my site?
- How do I scrape the way I would want to be crawled?
- Where CrawlCheck fits, and where it does not

Web scraping APIs sell the same thing in different units: a page fetched for you, through proxies, optionally rendered in a browser. The headline plans cost about the same; what differs is how many credits a hard page costs. A page that needs JavaScript and a premium proxy costs 25 times a plain page at two of the providers below, so what roughly $19 a month buys ranges from about 1,000 to 15,000 JavaScript-rendered pages. This page converts each plan into pages, adds the free open-source options, and then covers the one question none of these tools can answer: what AI crawlers are served by your own site.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing, documentation or repository page, with its address written out under each table, read on 3 October 2026. Prices change; those pages are the source of truth. Where a provider does not publish a figure, the table says so instead of estimating.

## What does a plain, JavaScript or protected page cost? #

| API | Free | Entry plan | Plain page | JavaScript page | Behind bot protection |
| --- | --- | --- | --- | --- | --- |
| ScrapingBee | 1,000 credits, no card | Hobby $19/mo, 75,000 credits | 1 credit (rendering off) | 5 credits (the default) | Premium proxy + JS 25; stealth 75 |
| ZenRows | 5,000 credits a month, no card | Build $16/mo, 45,000 credits | 1 credit | 5 credits | Premium proxy 10; with JS 25 |
| ScraperAPI | 1,000 credits; 7-day trial with 5,000 | Hobby $49/mo, 100,000 credits | 1 credit | Not priced separately on the pricing page | +10 credits on sites behind Cloudflare, DataDome or PerimeterX |
| Bright Data Web Unlocker | 5,000 requests a month | $1.50 per 1,000 successful requests pay as you go; Scale $499/mo | Per successful request | Browser rendering included | CAPTCHA solving included |
| Firecrawl | 1,000 credits | Hobby $19/mo ($16 annual), 5,000 credits | 1 credit | 1 credit (no separate render charge listed); JSON extraction 5 | Not priced separately on the pricing page |
| Apify | $5 of usage a month, no card | Starter $19/mo of usage | Depends on the actor | Browser actors use more compute units, at $0.20 each | [Residential proxy](/glossary/residential-proxy) $8/GB |
| Browse AI | 50 credits a month | Personal $19/mo billed annually ($48 monthly), 1,000 credits | 1 credit = 10 rows or one screenshot | Same | Premium sites 2 to 10 credits per task |
| **CrawlCheck** | Unlimited scans of your own site, no account | API $99/mo, 2,000 scans | Not a scraper | Reports content that only exists after JavaScript | Reports refusals and challenges as findings instead of routing around them |

Sources: ScrapingBee (https://www.scrapingbee.com/pricing/) and its docs (https://www.scrapingbee.com/documentation/), ZenRows (https://www.zenrows.com/pricing), ScraperAPI (https://www.scraperapi.com/pricing/), Bright Data (https://brightdata.com/pricing/web-unlocker), Firecrawl (https://www.firecrawl.dev/pricing), Apify (https://apify.com/pricing), Browse AI (https://www.browse.ai/pricing), CrawlCheck [pricing](/pricing).

## How many pages does an entry plan actually buy? #

| Plan | Plain pages | JavaScript pages | JavaScript + premium proxy |
| --- | --- | --- | --- |
| ZenRows Build ($16) | 45,000 | 9,000 | 1,800 |
| ScrapingBee Hobby ($19) | 75,000 | 15,000 | 3,000 (1,000 with stealth) |
| Firecrawl Hobby ($19) | 5,000 | 5,000 (1,000 with JSON extraction) | not published |
| ScraperAPI Hobby ($49) | 100,000 | not published | about 9,000 at 11 credits |

Bright Data bills per successful request, so its unit is already a delivered page: 1,000 for $1.50 pay as you go, and a blocked attempt costs nothing. The cheapest plan on paper is not always the cheapest for your mix of pages.

## How do I estimate my cost before I buy? #

Split your pages into plain HTML, JavaScript and bot-protected, multiply each share by its credit cost and add them up. A worked example for 10,000 pages a month, 70 percent plain, 20 percent needing JavaScript and 10 percent needing JavaScript plus a premium proxy:

| Page type | Pages | Credits each (ScrapingBee and ZenRows) | Credits |
| --- | --- | --- | --- |
| Plain HTML | 7,000 | 1 | 7,000 |
| JavaScript | 2,000 | 5 | 10,000 |
| JavaScript + premium proxy | 1,000 | 25 | 25,000 |
| **Total** | 10,000 | | **42,000** |

The worked example below, drawn to scale: the hard pages drive the cost, not the page count.
That mix fits ScrapingBee’s Hobby plan (75,000 credits) and, just, ZenRows’ Build plan (45,000). The 10 percent of hard pages costs 60 percent of the credits. If the hard share doubles to 20 percent (with plain pages down to 60 percent), the same 10,000 pages need 66,000 credits.

## What costs never appear on a pricing page? #

**Residential proxy bandwidth.** Sites with bot protection refuse datacenter addresses, so crawls route through residential IPs billed by the gigabyte. Apify publishes $8 per GB. At 2 MB a page, 10,000 pages is 20 GB, or $160 a month in proxy traffic alone, eight times the plan price. Check your own pages’ transfer size in the browser’s network panel rather than trusting any average, including this one.

**Retries and failures.** A refused or timed-out request is usually retried, and on credit-metered plans a retried page can be billed twice. Read each provider’s documentation on whether failed attempts consume credits, and watch the success rate for the first week: on hard targets the failure rate decides the real cost per delivered page.

**Rendering you did not need.** Rendering is the default on some APIs and costs five times a plain fetch. Before a large job, fetch a sample of target URLs both ways and compare the text. If the plain fetch already carries what you need, switch rendering off for that site and the same credits go five times further.

## What about free, open-source crawlers? #

| Project | Licence | Runs a browser | Output | Notes |
| --- | --- | --- | --- | --- |
| Crawl4AI | Apache 2.0 | Yes, Playwright (Chromium) | LLM-ready Markdown, CSS, XPath, regex and LLM-based extraction | Setup installs Chromium |
| Firecrawl (self-hosted) | AGPL-3.0; SDKs and some UI MIT | Yes | Markdown, JSON | Docker Compose; the cloud version has extra features |
| Scrapy | BSD-3-Clause | No, not by itself | Whatever your spider yields | The high-volume HTML workhorse |
| Jina Reader | [Service](/glossary/service) code on GitHub; models CC-BY-NC 4.0, so not for commercial use without a licence | Hosted service | Markdown or JSON | Free at r.jina.ai: 20 requests a minute without a key, 500 with a free key |
| **CrawlCheck** | Not open source (hosted service) | No; measures the HTML delivered before JavaScript | Graded report; per-domain answer by API | Checks whether AI crawlers can read a site |

Sources: Crawl4AI (https://github.com/unclecode/crawl4ai), Firecrawl (https://github.com/firecrawl/firecrawl), Scrapy (https://github.com/scrapy/scrapy), Jina Reader (https://jina.ai/reader/).

Licences matter more than popularity. Apache 2.0 and BSD-3 let you build a commercial product on the code with few conditions. AGPL-3.0 requires you to publish your changes if you offer the modified software to users over a network. Free software is also not a free crawl: you pay for the server, browser memory when pages need rendering, proxies when sites refuse datacenter traffic, and the hours spent when a site changes its markup and a selector stops matching. With a developer on hand that trade is often worth it; without one, a hosted plan usually costs less in total.

## Which one should I pick? #

| Your job | Best fit | Why |
| --- | --- | --- |
| Mostly plain HTML through an API | ScrapingBee or ZenRows with rendering off | Credits go furthest at one per page |
| Lots of JavaScript-heavy pages through an API | ScrapingBee Hobby | About 15,000 rendered pages at the default five credits |
| Hard targets behind bot protection | Bright Data Web Unlocker | Per-success pricing removes the cost of failed attempts |
| Clean Markdown for an LLM or [RAG](/glossary/rag) pipeline | Firecrawl, or Crawl4AI self-hosted | One credit a page hosted; free under Apache 2.0 if you run it |
| Scheduled scrapers and ready-made actors | Apify | Usage-based with proxies and storage included |
| No-code change monitoring | Browse AI | Rows and screenshots without code |
| Millions of plain HTML pages, a developer on hand | Scrapy | No subscription; you pay in servers and upkeep |
| Whether ChatGPT, Claude and Perplexity can read *your* site | None of the above: a free CrawlCheck scan | A scraper sees what the scraper is served, not what GPTBot or ClaudeBot is served |

## Why can’t a scraper tell me what ChatGPT sees on my site? #

Because sites and their CDNs treat clients by name and network. A scraping API identifies as itself or a browser, often through residential proxies. Firecrawl identifies itself as `FirecrawlAgent` and applies the [robots.txt](/glossary/robots-txt) rules written for that [token](/glossary/token) (per Firecrawl’s documentation, https://docs.firecrawl.dev/advanced-scraping-guide), so a site that blocks GPTBot can serve Firecrawl perfectly, and the reverse. The AI companies also run several crawlers each, for training, search and user-requested fetches, and blocking one does not block the others; the [AI crawler list](/guides/ai-crawler-list-what-each-one-does) sets out which does what.

Rendering cuts the same way. A browser-based crawler such as Crawl4AI reads a JavaScript-built page in full, which is what you want when collecting data, but it is a poor guide to what most AI crawlers get, because they read the HTML the server sent. If a page reads perfectly in Crawl4AI and comes back nearly empty from Scrapy, AI crawlers are getting the Scrapy version. A quick test with Scrapy installed: `scrapy fetch --nolog https://example.com/ | wc -c` gives the size of the HTML the server sends; view the same page with JavaScript disabled to see what it says. More in [do AI crawlers render JavaScript](/guides/do-ai-crawlers-render-javascript).

None of the problems below shows up in a scraper’s output, because the scraper is not the one being refused, challenged or served less. These are the shares of sites carrying each problem across CrawlCheck’s scans, read live:

| What goes wrong for AI crawlers | Share of scanned sites |
| --- | --- |
| An [answer engine](/glossary/answer-engine)’s crawler refused while a browser was served | 5.8% |
| A named crawler served materially less text than an ordinary client | 1.1% |
| A challenge page served with status 200 instead of the content | 0.2% |
| Content that only exists after JavaScript runs | 2.1% |
| Almost nothing a machine receives is readable text | 28.4% |
| An [AI opt-out](/glossary/ai-opt-out) signal already set | 1.8% |

**Live, as you read this:** the corpus now holds 582,334 domains across 9,306 scans. The figures in this piece were measured on the date above; this line is not.

The method and every finding are on the [dataset page](/data). To check your own site, [run a free scan](/) (no account, no card), or follow [how to check whether AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site).

## How do I scrape the way I would want to be crawled? #

- **Honour robots.txt.** Scrapy’s project template turns this on with `ROBOTSTXT_OBEY = True`; check the equivalent setting in whichever tool you use rather than assuming it is on.
- **Name yourself.** Send a user-agent that identifies your crawler and links to a page explaining it, so a site owner can write a rule for you instead of blocking a whole network.
- **Limit concurrency per host.** A small site on shared hosting can be slowed by a few parallel browser sessions.
- **Collect only what the job needs** and refresh unchanged pages on a schedule rather than refetching everything.

Sites increasingly treat unknown crawlers as hostile, so a crawler that ignores these rules tends to end up behind challenges and 403s, which costs you the data you were after.

## Where CrawlCheck fits, and where it does not #

CrawlCheck is not a scraper and not a substitute for anything above. It does not extract data from other people’s sites, export datasets, run arbitrary jobs or self-host. It answers the reverse question for your own site: when GPTBot, ClaudeBot and PerplexityBot fetch it, are they allowed, served and able to quote it? It sends each crawler’s published user-agent but cannot send from the AI companies’ own IP addresses, so a block keyed only to their IP ranges shows as a difference between identities rather than being guessed at. The first scan is free; monitoring is $29 a month for up to three domains and the API starts at $99 a month.
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.
Entries marked Verified paid a one-time listing fee. The fee buys a link, a badge and a re-test. It does not change any result. [Get listed](/get-listed)

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-web-scraping-apis-compared%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-web-scraping-apis-compared%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Web%20scraping%20APIs%20compared%3A%20the%20real%20cost%20per%20page)[Email](mailto:?subject=Web%20scraping%20APIs%20compared%3A%20the%20real%20cost%20per%20page&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-web-scraping-apis-compared)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is the best web scraping API?

It depends on the pages. For plain HTML, ScrapingBee or ZenRows with rendering switched off make credits go furthest. For hard targets behind bot protection, Bright Data Web Unlocker charges only for successful requests. For Markdown into LLM pipelines, Firecrawl. For scheduled scrapers with proxies, Apify. Convert each plan into pages for your own mix before comparing prices.

### Why do scraping API credits not equal pages?

Most credit-based APIs charge a multiple of a plain fetch for a page rendered in a browser, and a larger multiple again when a premium or stealth proxy is needed. A plan's credit allowance therefore buys far fewer JavaScript or bot-protected pages than plain ones, and the hardest pages usually drive most of the bill.

### Is Crawl4AI free?

Yes. Crawl4AI is open source under the Apache 2.0 licence and outputs LLM-ready Markdown. It runs real browsers through Playwright, so you pay for the server, its memory, any proxies and the time to maintain it rather than a subscription.

### Is Firecrawl open source?

Yes. The repository is mainly AGPL-3.0, with SDKs and some UI components under MIT, and it can be self-hosted with Docker Compose. The cloud version has additional features. The AGPL requires you to publish your changes if you offer a modified version to users over a network.

### Does a web scraper see what GPTBot or ClaudeBot see?

No. Each client is served according to its own name and network, and sites and CDNs often treat crawlers differently. Firecrawl identifies itself as FirecrawlAgent, for example, so a site that blocks GPTBot can still serve it, and the reverse. A scraper that renders JavaScript also reads more than most AI crawlers, which read the HTML the server sent.

### Is CrawlCheck a web scraping API?

No. CrawlCheck checks whether AI crawlers can reach, read and quote your own site, comparing what each crawler identity is served against a browser. It does not extract data from other sites. Its API returns scan results rather than scraped pages.

## Related findings

[2026-08-28 Do AI crawlers render JavaScript? Check what they see Assume AI crawlers do not run JavaScript. A one-command test shows whether your content is in the raw HTML, and why text inside a](/guides/do-ai-crawlers-render-javascript)[2026-10-18 How to verify Amazonbot and Amzn-SearchBot by IP address Amazon publishes Amazonbot and Amzn-SearchBot addresses on web pages. How to check a request against them, and how many claims we](/guides/how-to-verify-amazonbot-and-amzn-searchbot)[2026-10-18 How to check if your website supports HTTP/3 A site advertises HTTP/3 in its Alt-Svc header and its DNS HTTPS record. How to check both, what google.com and others show, and w](/guides/http-3-how-to-check-whether-your-site-offers-it)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [AI policy](/registry/aipolicy) · [rosstree.net](/registry/rosstree.net) · [Trade: Fencing](/registry/trade/fencing) · [exa.ai](/registry/capabilities/exa.ai) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Website audit](/website-audit) · [Conduct: ChatGPT-User](/agents/conduct/chatgpt-user) | **Datasets:** [Technologies: Hubspot](/datasets/technologies/hubspot) · [Technologies: Powr](/datasets/technologies/powr) | **Guides:** [HTTP 3 how to check whether your site offers it](/guides/http-3-how-to-check-whether-your-site-offers-it)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/best-web-scraping-apis-compared.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.