CrawlCheck

Guides · 2026-10-03 · By · 15 views

Web scraping APIs compared: the real cost per page

ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl, Apify and Browse AI converted from credits into pages, plus Crawl4AI, Scrapy and Jina Reader for free, and the one question none of them answers about your own site.

Converted into JavaScript-rendered pages, ScrapingBee Hobby ($19) gives about 15,000, ZenRows Build ($16) about 9,000 and Firecrawl Hobby ($19) 5,000; Bright Data charges $1.50 per 1,000 successful requests and Apify bills compute. Crawl4AI and Scrapy are free but cost servers and upkeep. The bill is driven by the hardest pages, not the page count. None of these tools shows whether ChatGPT or Claude can read your own site, because each is served as itself; a free CrawlCheck scan checks that.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Web scraping APIs sell the same thing in different units: a page fetched for you, through proxies, optionally rendered in a browser. The headline plans cost about the same; what differs is how many credits a hard page costs. A page that needs JavaScript and a premium proxy costs 25 times a plain page at two of the providers below, so what roughly $19 a month buys ranges from about 1,000 to 15,000 JavaScript-rendered pages. This page converts each plan into pages, adds the free open-source options, and then covers the one question none of these tools can answer: what AI crawlers are served by your own site.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing, documentation or repository page, with its address written out under each table, read on 3 October 2026. Prices change; those pages are the source of truth. Where a provider does not publish a figure, the table says so instead of estimating.

What does a plain, JavaScript or protected page cost? #

APIFreeEntry planPlain pageJavaScript pageBehind bot protection
ScrapingBee1,000 credits, no cardHobby $19/mo, 75,000 credits1 credit (rendering off)5 credits (the default)Premium proxy + JS 25; stealth 75
ZenRows5,000 credits a month, no cardBuild $16/mo, 45,000 credits1 credit5 creditsPremium proxy 10; with JS 25
ScraperAPI1,000 credits; 7-day trial with 5,000Hobby $49/mo, 100,000 credits1 creditNot priced separately on the pricing page+10 credits on sites behind Cloudflare, DataDome or PerimeterX
Bright Data Web Unlocker5,000 requests a month$1.50 per 1,000 successful requests pay as you go; Scale $499/moPer successful requestBrowser rendering includedCAPTCHA solving included
Firecrawl1,000 creditsHobby $19/mo ($16 annual), 5,000 credits1 credit1 credit (no separate render charge listed); JSON extraction 5Not priced separately on the pricing page
Apify$5 of usage a month, no cardStarter $19/mo of usageDepends on the actorBrowser actors use more compute units, at $0.20 eachResidential proxy $8/GB
Browse AI50 credits a monthPersonal $19/mo billed annually ($48 monthly), 1,000 credits1 credit = 10 rows or one screenshotSamePremium sites 2 to 10 credits per task
CrawlCheckUnlimited scans of your own site, no accountAPI $99/mo, 2,000 scansNot a scraperReports content that only exists after JavaScriptReports refusals and challenges as findings instead of routing around them

Sources: ScrapingBee (https://www.scrapingbee.com/pricing/) and its docs (https://www.scrapingbee.com/documentation/), ZenRows (https://www.zenrows.com/pricing), ScraperAPI (https://www.scraperapi.com/pricing/), Bright Data (https://brightdata.com/pricing/web-unlocker), Firecrawl (https://www.firecrawl.dev/pricing), Apify (https://apify.com/pricing), Browse AI (https://www.browse.ai/pricing), CrawlCheck pricing.

How many pages does an entry plan actually buy? #

PlanPlain pagesJavaScript pagesJavaScript + premium proxy
ZenRows Build ($16)45,0009,0001,800
ScrapingBee Hobby ($19)75,00015,0003,000 (1,000 with stealth)
Firecrawl Hobby ($19)5,0005,000 (1,000 with JSON extraction)not published
ScraperAPI Hobby ($49)100,000not publishedabout 9,000 at 11 credits

Bright Data bills per successful request, so its unit is already a delivered page: 1,000 for $1.50 pay as you go, and a blocked attempt costs nothing. The cheapest plan on paper is not always the cheapest for your mix of pages.

How do I estimate my cost before I buy? #

Split your pages into plain HTML, JavaScript and bot-protected, multiply each share by its credit cost and add them up. A worked example for 10,000 pages a month, 70 percent plain, 20 percent needing JavaScript and 10 percent needing JavaScript plus a premium proxy:

Page typePagesCredits each (ScrapingBee and ZenRows)Credits
Plain HTML7,00017,000
JavaScript2,000510,000
JavaScript + premium proxy1,0002525,000
Total10,00042,000
10,000 pages a month on ScrapingBee or ZenRows credit pricingPagesCredits7,000 plain2,000 JS1,000 JS + premium proxy7,00010,00025,000 for the JS + premium proxy pages42,000 credits in totalThe hardest pages are the smallest slice of the job and the largest slice of the bill.
The worked example below, drawn to scale: the hard pages drive the cost, not the page count.

That mix fits ScrapingBee’s Hobby plan (75,000 credits) and, just, ZenRows’ Build plan (45,000). The 10 percent of hard pages costs 60 percent of the credits. If the hard share doubles to 20 percent (with plain pages down to 60 percent), the same 10,000 pages need 66,000 credits.

What costs never appear on a pricing page? #

Residential proxy bandwidth. Sites with bot protection refuse datacenter addresses, so crawls route through residential IPs billed by the gigabyte. Apify publishes $8 per GB. At 2 MB a page, 10,000 pages is 20 GB, or $160 a month in proxy traffic alone, eight times the plan price. Check your own pages’ transfer size in the browser’s network panel rather than trusting any average, including this one.

Retries and failures. A refused or timed-out request is usually retried, and on credit-metered plans a retried page can be billed twice. Read each provider’s documentation on whether failed attempts consume credits, and watch the success rate for the first week: on hard targets the failure rate decides the real cost per delivered page.

Rendering you did not need. Rendering is the default on some APIs and costs five times a plain fetch. Before a large job, fetch a sample of target URLs both ways and compare the text. If the plain fetch already carries what you need, switch rendering off for that site and the same credits go five times further.

What about free, open-source crawlers? #

ProjectLicenceRuns a browserOutputNotes
Crawl4AIApache 2.0Yes, Playwright (Chromium)LLM-ready Markdown, CSS, XPath, regex and LLM-based extractionSetup installs Chromium
Firecrawl (self-hosted)AGPL-3.0; SDKs and some UI MITYesMarkdown, JSONDocker Compose; the cloud version has extra features
ScrapyBSD-3-ClauseNo, not by itselfWhatever your spider yieldsThe high-volume HTML workhorse
Jina ReaderService code on GitHub; models CC-BY-NC 4.0, so not for commercial use without a licenceHosted serviceMarkdown or JSONFree at r.jina.ai: 20 requests a minute without a key, 500 with a free key
CrawlCheckNot open source (hosted service)No; measures the HTML delivered before JavaScriptGraded report; per-domain answer by APIChecks whether AI crawlers can read a site

Sources: Crawl4AI (https://github.com/unclecode/crawl4ai), Firecrawl (https://github.com/firecrawl/firecrawl), Scrapy (https://github.com/scrapy/scrapy), Jina Reader (https://jina.ai/reader/).

Licences matter more than popularity. Apache 2.0 and BSD-3 let you build a commercial product on the code with few conditions. AGPL-3.0 requires you to publish your changes if you offer the modified software to users over a network. Free software is also not a free crawl: you pay for the server, browser memory when pages need rendering, proxies when sites refuse datacenter traffic, and the hours spent when a site changes its markup and a selector stops matching. With a developer on hand that trade is often worth it; without one, a hosted plan usually costs less in total.

Which one should I pick? #

Your jobBest fitWhy
Mostly plain HTML through an APIScrapingBee or ZenRows with rendering offCredits go furthest at one per page
Lots of JavaScript-heavy pages through an APIScrapingBee HobbyAbout 15,000 rendered pages at the default five credits
Hard targets behind bot protectionBright Data Web UnlockerPer-success pricing removes the cost of failed attempts
Clean Markdown for an LLM or RAG pipelineFirecrawl, or Crawl4AI self-hostedOne credit a page hosted; free under Apache 2.0 if you run it
Scheduled scrapers and ready-made actorsApifyUsage-based with proxies and storage included
No-code change monitoringBrowse AIRows and screenshots without code
Millions of plain HTML pages, a developer on handScrapyNo subscription; you pay in servers and upkeep
Whether ChatGPT, Claude and Perplexity can read your siteNone of the above: a free CrawlCheck scanA scraper sees what the scraper is served, not what GPTBot or ClaudeBot is served

Why can’t a scraper tell me what ChatGPT sees on my site? #

Because sites and their CDNs treat clients by name and network. A scraping API identifies as itself or a browser, often through residential proxies. Firecrawl identifies itself as FirecrawlAgent and applies the robots.txt rules written for that token (per Firecrawl’s documentation, https://docs.firecrawl.dev/advanced-scraping-guide), so a site that blocks GPTBot can serve Firecrawl perfectly, and the reverse. The AI companies also run several crawlers each, for training, search and user-requested fetches, and blocking one does not block the others; the AI crawler list sets out which does what.

Rendering cuts the same way. A browser-based crawler such as Crawl4AI reads a JavaScript-built page in full, which is what you want when collecting data, but it is a poor guide to what most AI crawlers get, because they read the HTML the server sent. If a page reads perfectly in Crawl4AI and comes back nearly empty from Scrapy, AI crawlers are getting the Scrapy version. A quick test with Scrapy installed: scrapy fetch --nolog https://example.com/ | wc -c gives the size of the HTML the server sends; view the same page with JavaScript disabled to see what it says. More in do AI crawlers render JavaScript.

None of the problems below shows up in a scraper’s output, because the scraper is not the one being refused, challenged or served less. These are the shares of sites carrying each problem across CrawlCheck’s scans, read live:

What goes wrong for AI crawlersShare of scanned sites
An answer engine’s crawler refused while a browser was served5.8%
A named crawler served materially less text than an ordinary client1.1%
A challenge page served with status 200 instead of the content0.2%
Content that only exists after JavaScript runs2.1%
Almost nothing a machine receives is readable text28.4%
An AI opt-out signal already set1.8%

Live, as you read this: the corpus now holds 582,334 domains across 9,304 scans. The figures in this piece were measured on the date above; this line is not.

The method and every finding are on the dataset page. To check your own site, run a free scan (no account, no card), or follow how to check whether AI crawlers can read your site.

How do I scrape the way I would want to be crawled? #

  • Honour robots.txt. Scrapy’s project template turns this on with ROBOTSTXT_OBEY = True; check the equivalent setting in whichever tool you use rather than assuming it is on.
  • Name yourself. Send a user-agent that identifies your crawler and links to a page explaining it, so a site owner can write a rule for you instead of blocking a whole network.
  • Limit concurrency per host. A small site on shared hosting can be slowed by a few parallel browser sessions.
  • Collect only what the job needs and refresh unchanged pages on a schedule rather than refetching everything.

Sites increasingly treat unknown crawlers as hostile, so a crawler that ignores these rules tends to end up behind challenges and 403s, which costs you the data you were after.

Where CrawlCheck fits, and where it does not #

CrawlCheck is not a scraper and not a substitute for anything above. It does not extract data from other people’s sites, export datasets, run arbitrary jobs or self-host. It answers the reverse question for your own site: when GPTBot, ClaudeBot and PerplexityBot fetch it, are they allowed, served and able to quote it? It sends each crawler’s published user-agent but cannot send from the AI companies’ own IP addresses, so a block keyed only to their IP ranges shows as a difference between identities rather than being guessed at. The first scan is free; monitoring is $29 a month for up to three domains and the API starts at $99 a month.

Share of scanned sites with each finding in this articlePage is mostly code28.4%Answer-engine crawler refused5.8%Content needs JavaScript to appear2.1%AI training opt-out set1.8%Crawler served less than a browser1.1%Bot challenge served as a 200 page0.2%Share of scanned sites with eachfinding in this articlePage is mostly code28.4%Answer-engine crawler refused5.8%Content needs JavaScript to appear2.1%AI training opt-out set1.8%Crawler served less than a browser1.1%Bot challenge served as a 200 page0.2%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Entries marked Verified paid a one-time listing fee. The fee buys a link, a badge and a re-test. It does not change any result. Get listed

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What is the best web scraping API?

It depends on the pages. For plain HTML, ScrapingBee or ZenRows with rendering switched off make credits go furthest. For hard targets behind bot protection, Bright Data Web Unlocker charges only for successful requests. For Markdown into LLM pipelines, Firecrawl. For scheduled scrapers with proxies, Apify. Convert each plan into pages for your own mix before comparing prices.

Why do scraping API credits not equal pages?

Most credit-based APIs charge a multiple of a plain fetch for a page rendered in a browser, and a larger multiple again when a premium or stealth proxy is needed. A plan's credit allowance therefore buys far fewer JavaScript or bot-protected pages than plain ones, and the hardest pages usually drive most of the bill.

Is Crawl4AI free?

Yes. Crawl4AI is open source under the Apache 2.0 licence and outputs LLM-ready Markdown. It runs real browsers through Playwright, so you pay for the server, its memory, any proxies and the time to maintain it rather than a subscription.

Is Firecrawl open source?

Yes. The repository is mainly AGPL-3.0, with SDKs and some UI components under MIT, and it can be self-hosted with Docker Compose. The cloud version has additional features. The AGPL requires you to publish your changes if you offer a modified version to users over a network.

Does a web scraper see what GPTBot or ClaudeBot see?

No. Each client is served according to its own name and network, and sites and CDNs often treat crawlers differently. Firecrawl identifies itself as FirecrawlAgent, for example, so a site that blocks GPTBot can still serve it, and the reverse. A scraper that renders JavaScript also reads more than most AI crawlers, which read the HTML the server sent.

Is CrawlCheck a web scraping API?

No. CrawlCheck checks whether AI crawlers can reach, read and quote your own site, comparing what each crawler identity is served against a browser. It does not extract data from other sites. Its API returns scan results rather than scraped pages.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works