Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 15 views
Web scraping APIs compared: the real cost per page
ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl, Apify and Browse AI converted from credits into pages, plus Crawl4AI, Scrapy and Jina Reader for free, and the one question none of them answers about your own site.
Converted into JavaScript-rendered pages, ScrapingBee Hobby ($19) gives about 15,000, ZenRows Build ($16) about 9,000 and Firecrawl Hobby ($19) 5,000; Bright Data charges $1.50 per 1,000 successful requests and Apify bills compute. Crawl4AI and Scrapy are free but cost servers and upkeep. The bill is driven by the hardest pages, not the page count. None of these tools shows whether ChatGPT or Claude can read your own site, because each is served as itself; a free CrawlCheck scan checks that.
Web scraping APIs sell the same thing in different units: a page fetched for you, through proxies, optionally rendered in a browser. The headline plans cost about the same; what differs is how many credits a hard page costs. A page that needs JavaScript and a premium proxy costs 25 times a plain page at two of the providers below, so what roughly $19 a month buys ranges from about 1,000 to 15,000 JavaScript-rendered pages. This page converts each plan into pages, adds the free open-source options, and then covers the one question none of these tools can answer: what AI crawlers are served by your own site.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing, documentation or repository page, with its address written out under each table, read on 3 October 2026. Prices change; those pages are the source of truth. Where a provider does not publish a figure, the table says so instead of estimating.
What does a plain, JavaScript or protected page cost? #
| API | Free | Entry plan | Plain page | JavaScript page | Behind bot protection |
|---|---|---|---|---|---|
| ScrapingBee | 1,000 credits, no card | Hobby $19/mo, 75,000 credits | 1 credit (rendering off) | 5 credits (the default) | Premium proxy + JS 25; stealth 75 |
| ZenRows | 5,000 credits a month, no card | Build $16/mo, 45,000 credits | 1 credit | 5 credits | Premium proxy 10; with JS 25 |
| ScraperAPI | 1,000 credits; 7-day trial with 5,000 | Hobby $49/mo, 100,000 credits | 1 credit | Not priced separately on the pricing page | +10 credits on sites behind Cloudflare, DataDome or PerimeterX |
| Bright Data Web Unlocker | 5,000 requests a month | $1.50 per 1,000 successful requests pay as you go; Scale $499/mo | Per successful request | Browser rendering included | CAPTCHA solving included |
| Firecrawl | 1,000 credits | Hobby $19/mo ($16 annual), 5,000 credits | 1 credit | 1 credit (no separate render charge listed); JSON extraction 5 | Not priced separately on the pricing page |
| Apify | $5 of usage a month, no card | Starter $19/mo of usage | Depends on the actor | Browser actors use more compute units, at $0.20 each | Residential proxy $8/GB |
| Browse AI | 50 credits a month | Personal $19/mo billed annually ($48 monthly), 1,000 credits | 1 credit = 10 rows or one screenshot | Same | Premium sites 2 to 10 credits per task |
| CrawlCheck | Unlimited scans of your own site, no account | API $99/mo, 2,000 scans | Not a scraper | Reports content that only exists after JavaScript | Reports refusals and challenges as findings instead of routing around them |
Sources: ScrapingBee (https://www.scrapingbee.com/pricing/) and its docs (https://www.scrapingbee.com/documentation/), ZenRows (https://www.zenrows.com/pricing), ScraperAPI (https://www.scraperapi.com/pricing/), Bright Data (https://brightdata.com/pricing/web-unlocker), Firecrawl (https://www.firecrawl.dev/pricing), Apify (https://apify.com/pricing), Browse AI (https://www.browse.ai/pricing), CrawlCheck pricing.
How many pages does an entry plan actually buy? #
| Plan | Plain pages | JavaScript pages | JavaScript + premium proxy |
|---|---|---|---|
| ZenRows Build ($16) | 45,000 | 9,000 | 1,800 |
| ScrapingBee Hobby ($19) | 75,000 | 15,000 | 3,000 (1,000 with stealth) |
| Firecrawl Hobby ($19) | 5,000 | 5,000 (1,000 with JSON extraction) | not published |
| ScraperAPI Hobby ($49) | 100,000 | not published | about 9,000 at 11 credits |
Bright Data bills per successful request, so its unit is already a delivered page: 1,000 for $1.50 pay as you go, and a blocked attempt costs nothing. The cheapest plan on paper is not always the cheapest for your mix of pages.
How do I estimate my cost before I buy? #
Split your pages into plain HTML, JavaScript and bot-protected, multiply each share by its credit cost and add them up. A worked example for 10,000 pages a month, 70 percent plain, 20 percent needing JavaScript and 10 percent needing JavaScript plus a premium proxy:
| Page type | Pages | Credits each (ScrapingBee and ZenRows) | Credits |
|---|---|---|---|
| Plain HTML | 7,000 | 1 | 7,000 |
| JavaScript | 2,000 | 5 | 10,000 |
| JavaScript + premium proxy | 1,000 | 25 | 25,000 |
| Total | 10,000 | 42,000 |
That mix fits ScrapingBee’s Hobby plan (75,000 credits) and, just, ZenRows’ Build plan (45,000). The 10 percent of hard pages costs 60 percent of the credits. If the hard share doubles to 20 percent (with plain pages down to 60 percent), the same 10,000 pages need 66,000 credits.
What costs never appear on a pricing page? #
Residential proxy bandwidth. Sites with bot protection refuse datacenter addresses, so crawls route through residential IPs billed by the gigabyte. Apify publishes $8 per GB. At 2 MB a page, 10,000 pages is 20 GB, or $160 a month in proxy traffic alone, eight times the plan price. Check your own pages’ transfer size in the browser’s network panel rather than trusting any average, including this one.
Retries and failures. A refused or timed-out request is usually retried, and on credit-metered plans a retried page can be billed twice. Read each provider’s documentation on whether failed attempts consume credits, and watch the success rate for the first week: on hard targets the failure rate decides the real cost per delivered page.
Rendering you did not need. Rendering is the default on some APIs and costs five times a plain fetch. Before a large job, fetch a sample of target URLs both ways and compare the text. If the plain fetch already carries what you need, switch rendering off for that site and the same credits go five times further.
What about free, open-source crawlers? #
| Project | Licence | Runs a browser | Output | Notes |
|---|---|---|---|---|
| Crawl4AI | Apache 2.0 | Yes, Playwright (Chromium) | LLM-ready Markdown, CSS, XPath, regex and LLM-based extraction | Setup installs Chromium |
| Firecrawl (self-hosted) | AGPL-3.0; SDKs and some UI MIT | Yes | Markdown, JSON | Docker Compose; the cloud version has extra features |
| Scrapy | BSD-3-Clause | No, not by itself | Whatever your spider yields | The high-volume HTML workhorse |
| Jina Reader | Service code on GitHub; models CC-BY-NC 4.0, so not for commercial use without a licence | Hosted service | Markdown or JSON | Free at r.jina.ai: 20 requests a minute without a key, 500 with a free key |
| CrawlCheck | Not open source (hosted service) | No; measures the HTML delivered before JavaScript | Graded report; per-domain answer by API | Checks whether AI crawlers can read a site |
Sources: Crawl4AI (https://github.com/unclecode/crawl4ai), Firecrawl (https://github.com/firecrawl/firecrawl), Scrapy (https://github.com/scrapy/scrapy), Jina Reader (https://jina.ai/reader/).
Licences matter more than popularity. Apache 2.0 and BSD-3 let you build a commercial product on the code with few conditions. AGPL-3.0 requires you to publish your changes if you offer the modified software to users over a network. Free software is also not a free crawl: you pay for the server, browser memory when pages need rendering, proxies when sites refuse datacenter traffic, and the hours spent when a site changes its markup and a selector stops matching. With a developer on hand that trade is often worth it; without one, a hosted plan usually costs less in total.
Which one should I pick? #
| Your job | Best fit | Why |
|---|---|---|
| Mostly plain HTML through an API | ScrapingBee or ZenRows with rendering off | Credits go furthest at one per page |
| Lots of JavaScript-heavy pages through an API | ScrapingBee Hobby | About 15,000 rendered pages at the default five credits |
| Hard targets behind bot protection | Bright Data Web Unlocker | Per-success pricing removes the cost of failed attempts |
| Clean Markdown for an LLM or RAG pipeline | Firecrawl, or Crawl4AI self-hosted | One credit a page hosted; free under Apache 2.0 if you run it |
| Scheduled scrapers and ready-made actors | Apify | Usage-based with proxies and storage included |
| No-code change monitoring | Browse AI | Rows and screenshots without code |
| Millions of plain HTML pages, a developer on hand | Scrapy | No subscription; you pay in servers and upkeep |
| Whether ChatGPT, Claude and Perplexity can read your site | None of the above: a free CrawlCheck scan | A scraper sees what the scraper is served, not what GPTBot or ClaudeBot is served |
Why can’t a scraper tell me what ChatGPT sees on my site? #
Because sites and their CDNs treat clients by name and network. A scraping API identifies as itself or a browser, often through residential proxies. Firecrawl identifies itself as FirecrawlAgent and applies the robots.txt rules written for that token (per Firecrawl’s documentation, https://docs.firecrawl.dev/advanced-scraping-guide), so a site that blocks GPTBot can serve Firecrawl perfectly, and the reverse. The AI companies also run several crawlers each, for training, search and user-requested fetches, and blocking one does not block the others; the AI crawler list sets out which does what.
Rendering cuts the same way. A browser-based crawler such as Crawl4AI reads a JavaScript-built page in full, which is what you want when collecting data, but it is a poor guide to what most AI crawlers get, because they read the HTML the server sent. If a page reads perfectly in Crawl4AI and comes back nearly empty from Scrapy, AI crawlers are getting the Scrapy version. A quick test with Scrapy installed: scrapy fetch --nolog https://example.com/ | wc -c gives the size of the HTML the server sends; view the same page with JavaScript disabled to see what it says. More in do AI crawlers render JavaScript.
None of the problems below shows up in a scraper’s output, because the scraper is not the one being refused, challenged or served less. These are the shares of sites carrying each problem across CrawlCheck’s scans, read live:
| What goes wrong for AI crawlers | Share of scanned sites |
|---|---|
| An answer engine’s crawler refused while a browser was served | 5.8% |
| A named crawler served materially less text than an ordinary client | 1.1% |
| A challenge page served with status 200 instead of the content | 0.2% |
| Content that only exists after JavaScript runs | 2.1% |
| Almost nothing a machine receives is readable text | 28.4% |
| An AI opt-out signal already set | 1.8% |
Live, as you read this: the corpus now holds 582,334 domains across 9,295 scans. The figures in this piece were measured on the date above; this line is not.
The method and every finding are on the dataset page. To check your own site, run a free scan (no account, no card), or follow how to check whether AI crawlers can read your site.
How do I scrape the way I would want to be crawled? #
- Honour robots.txt. Scrapy’s project template turns this on with
ROBOTSTXT_OBEY = True; check the equivalent setting in whichever tool you use rather than assuming it is on. - Name yourself. Send a user-agent that identifies your crawler and links to a page explaining it, so a site owner can write a rule for you instead of blocking a whole network.
- Limit concurrency per host. A small site on shared hosting can be slowed by a few parallel browser sessions.
- Collect only what the job needs and refresh unchanged pages on a schedule rather than refetching everything.
Sites increasingly treat unknown crawlers as hostile, so a crawler that ignores these rules tends to end up behind challenges and 403s, which costs you the data you were after.
Where CrawlCheck fits, and where it does not #
CrawlCheck is not a scraper and not a substitute for anything above. It does not extract data from other people’s sites, export datasets, run arbitrary jobs or self-host. It answers the reverse question for your own site: when GPTBot, ClaudeBot and PerplexityBot fetch it, are they allowed, served and able to quote it? It sends each crawler’s published user-agent but cannot send from the AI companies’ own IP addresses, so a block keyed only to their IP ranges shows as a difference between identities rather than being guessed at. The first scan is free; monitoring is $29 a month for up to three domains and the API starts at $99 a month.
Entries marked Verified paid a one-time listing fee. The fee buys a link, a badge and a re-test. It does not change any result. Get listed
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is the best web scraping API?
It depends on the pages. For plain HTML, ScrapingBee or ZenRows with rendering switched off make credits go furthest. For hard targets behind bot protection, Bright Data Web Unlocker charges only for successful requests. For Markdown into LLM pipelines, Firecrawl. For scheduled scrapers with proxies, Apify. Convert each plan into pages for your own mix before comparing prices.
Why do scraping API credits not equal pages?
Most credit-based APIs charge a multiple of a plain fetch for a page rendered in a browser, and a larger multiple again when a premium or stealth proxy is needed. A plan's credit allowance therefore buys far fewer JavaScript or bot-protected pages than plain ones, and the hardest pages usually drive most of the bill.
Is Crawl4AI free?
Yes. Crawl4AI is open source under the Apache 2.0 licence and outputs LLM-ready Markdown. It runs real browsers through Playwright, so you pay for the server, its memory, any proxies and the time to maintain it rather than a subscription.
Is Firecrawl open source?
Yes. The repository is mainly AGPL-3.0, with SDKs and some UI components under MIT, and it can be self-hosted with Docker Compose. The cloud version has additional features. The AGPL requires you to publish your changes if you offer a modified version to users over a network.
Does a web scraper see what GPTBot or ClaudeBot see?
No. Each client is served according to its own name and network, and sites and CDNs often treat crawlers differently. Firecrawl identifies itself as FirecrawlAgent, for example, so a site that blocks GPTBot can still serve it, and the reverse. A scraper that renders JavaScript also reads more than most AI crawlers, which read the HTML the server sent.
Is CrawlCheck a web scraping API?
No. CrawlCheck checks whether AI crawlers can reach, read and quote your own site, comparing what each crawler identity is served against a browser. It does not extract data from other sites. Its API returns scan results rather than scraped pages.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.