Web Scraping Budget Calculator: How Much Does Scraping Really Cost?
Web scraping costs range from $0 to $5,000+ per month — and most budget estimates are wrong because they ignore three of the four real cost drivers. Compute and proxy fees show up on invoices. Developer time and maintenance almost never do.
This guide breaks down every cost component, runs the numbers at four realistic scales (1K, 10K, 100K, and 1M pages/month), and gives you a side-by-side comparison of Apify, Firecrawl, and self-hosted infrastructure on Liquid Web so you can build an honest budget before you start.
The four cost components of web scraping
Every scraping operation has four cost drivers. Most people price only the first one.
| Cost component | What it covers | Often missed? |
|---|---|---|
| Compute | CPU/RAM for running scrapers | No |
| Proxies | Residential, datacenter, or ISP IPs | Partially |
| Storage | Dataset persistence, KV stores, object storage | Usually |
| Development + maintenance | Initial build, bug fixes, site changes, retries | Almost always |
The ratio between these components shifts dramatically with scale. At 1K pages/month, developer time dominates. At 1M pages/month, proxies dominate. Getting the scale wrong by one order of magnitude can mean underestimating your budget by 5×.
Cost component 1: Compute
Compute is the most visible cost — it's what cloud dashboards show. But the unit varies by approach.
Managed platforms (Apify)
Apify bills in compute units (CU), where 1 CU = 1 GB RAM running for 1 hour.
- A lightweight HTML scraper using 128 MB RAM typically costs ~$0.0003–$0.001 per page
- A JavaScript-rendered page using 256–512 MB RAM costs ~$0.001–$0.005 per page
- A full browser session (Playwright) using 1 GB RAM costs ~$0.003–$0.01 per page
Apify's per-CU rates by plan:
| Plan | Monthly fee | CU rate | Best for |
|---|---|---|---|
| Free | $0 | $0.30/CU | Testing, <500 pages/mo ($5/mo platform cap) |
| Starter | $29/mo | $0.30/CU | Light automation |
| Scale | $199/mo | $0.25/CU | Regular scraping |
| Business | $999/mo | $0.20/CU | High-volume pipelines |
The monthly fee includes prepaid credits that count toward compute, proxies, and storage — it's not a separate subscription on top of usage.
Self-hosted (VPS on Liquid Web)
Liquid Web managed VPS plans start at around $30–$80/month for entry-level servers (2 vCPU, 4–8 GB RAM). A mid-tier setup capable of running 10+ concurrent Playwright sessions would need at least $80–$150/month in compute.
Self-hosted compute looks cheaper per-unit, but requires you to handle scaling, uptime, job queuing, and failure retries yourself — all of which cost developer time.
API-based services (Firecrawl)
Firecrawl charges per-page crawled:
| Plan | Monthly fee | Included pages | Overage |
|---|---|---|---|
| Free | $0 | 500/mo | – |
| Starter | $16/mo | 3,000/mo | ~$3.50/1K |
| Standard | $83/mo | 100,000/mo | ~$1.65/1K |
| Growth | $333/mo | 500,000/mo | ~$1.00/1K |
Firecrawl bills purely for crawled pages — you pay nothing for setup, proxies, or infrastructure management. That simplicity comes with a higher per-page rate than self-hosted at scale.
Cost component 2: Proxies
Proxies are where budgets blow up at scale. A residential proxy at $8/GB sounds cheap until you realize that scraping 100K JavaScript-rendered pages can consume 50–200 GB of bandwidth.
Proxy types and approximate rates (2026)
| Proxy type | Typical rate | Best for |
|---|---|---|
| Datacenter | $0.60–$1.30/GB | Non-protected sites, high volume |
| ISP (static residential) | $0.90–$2.50/GB | E-commerce, moderate protection |
| Residential | $3.00–$8.40/GB | Anti-bot protected, geo-targeting |
| Mobile (4G/5G) | $7.00–$20/GB | Hardest targets, social platforms |
Bright Data is the enterprise leader here. Residential proxies drop from $8.40/GB pay-as-you-go to $3.00/GB+ on committed monthly plans. Their Web Unlocker API — which handles JavaScript rendering, CAPTCHA solving, and retry logic — runs around $3/1,000 successful responses, effectively eliminating separate proxy + compute costs for difficult targets.
IPRoyal offers a non-expiring residential pool at around $7/GB. Bandwidth purchased never expires, making it economical for intermittent, high-volume bursts (quarterly audits, one-off research projects) where you'd otherwise waste a Bright Data monthly commitment.
Apify's built-in proxies for Scale plan and above include datacenter IPs, with residential proxies billed at $7–$8/GB within the platform. For most Apify-hosted scrapers, you never need to provision a separate proxy service — it's integrated.
Bandwidth estimation by page type
How much data does each page consume?
| Page type | Avg bandwidth | Notes |
|---|---|---|
| Simple HTML (news, blogs) | 50–150 KB/page | Low JS, no media |
| E-commerce product page | 200–500 KB/page | Product images, JSON-LD |
| JS-heavy SPA (LinkedIn, Airbnb) | 500 KB–2 MB/page | Full browser session |
| Social media profile | 1–5 MB/page | Images, video previews |
For 100K simple HTML pages: ~5–15 GB bandwidth → $15–$126/mo in residential proxies
For 100K JS-heavy pages: ~50–200 GB bandwidth → $150–$1,680/mo in residential proxies
This variance is why proxy costs are the hardest to predict — and why managed platforms that bundle anti-bot handling can be cheaper than raw proxy + compute at difficult targets.
Cost component 3: Storage
Storage costs are small at low scale and grow predictably with data volume.
Apify storage rates
- Dataset storage: $1.00/1,000 GB-hours (Scale: $0.90)
- Key-value store: $1.00/1,000 GB-hours
- Dataset reads: $0.0004/1,000 reads
- Dataset writes: $0.005/1,000 writes
For a typical 100K-page scrape that stores 1 KB of structured data per page (100 MB total), monthly storage costs are under $0.10. Even at 1M pages/month with 5 KB average output (5 GB), Apify storage costs are under $2/month.
Storage is rarely a meaningful line item on managed platforms. On self-hosted setups, you'll pay for the storage tier of your PostgreSQL, MongoDB, or S3-equivalent — typically $5–$20/month for moderate volumes.
Where storage costs actually appear
Large-scale scraping with heavy outputs (images, raw HTML archives, video metadata with thumbnails) can balloon storage costs. A pipeline archiving 10 GB of raw HTML daily (~300 GB/month) on cloud object storage costs roughly $6–$10/month on AWS S3 or equivalent. Still a minor line item compared to proxies and compute.
Cost component 4: Development time and maintenance
This is the cost that almost never appears in comparison articles — and the one that matters most at small to medium scale.
Initial development cost
| Scraper type | Estimated dev hours | At $75/hr |
|---|---|---|
| Simple HTML scraper (1 target) | 2–5 hrs | $150–$375 |
| JS-rendered single target | 5–15 hrs | $375–$1,125 |
| Multi-site enterprise pipeline | 40–200+ hrs | $3,000–$15,000+ |
| Using pre-built Actor (Apify Store) | 0–2 hrs config | $0–$150 |
Ongoing maintenance cost
Scrapers break. Target sites redesign, add CAPTCHAs, or change HTML structure. A realistic maintenance budget:
- Basic scraper (1–2 targets): 2–5 hrs/month → $150–$375/month
- Mid-scale pipeline (5–10 targets): 10–20 hrs/month → $750–$1,500/month
- Enterprise pipeline (50+ targets): Full-time engineer
Managed platforms reduce (but don't eliminate) maintenance. When Apify Actors fail due to site changes, either the community updates the Actor or you file an issue. Self-hosted scrapers are 100% your maintenance burden.
The budget calculator: total cost at 4 scales
Let's run the numbers for three approaches at four realistic monthly scales. Assumptions:
- Pages are moderately JS-rendered (LinkedIn, e-commerce difficulty)
- Using residential proxies (~500 KB/page average bandwidth)
- Developer cost: $75/hour
- Maintenance: 2 hrs/month for managed, 5 hrs/month for self-hosted
- One-time setup costs are amortized over 12 months
1,000 pages/month
| Approach | Compute | Proxies | Storage | Dev/maintenance | Total/month |
|---|---|---|---|---|---|
| Apify Free | $0 (included) | $4 (~0.5 GB @ $8/GB) | <$0.01 | $150 setup amortized | ~$154 |
| Firecrawl Free | $0 (500 included) | Included | Included | $50 setup amortized | ~$50 |
| Self-hosted (Liquid Web VPS) | $30/mo VPS | $4 (~0.5 GB) | $0 | $200 setup + $150/mo maint | ~$384 |
At 1K pages/month, self-hosting is the most expensive option because fixed VPS cost and developer time dominate. Firecrawl's free tier wins on pure cost. Apify's free tier works but caps you at $5/month in total platform usage (compute + proxies).
10,000 pages/month
| Approach | Compute | Proxies | Storage | Dev/maintenance | Total/month |
|---|---|---|---|---|---|
| Apify Starter | $29 (included) | $28 (~3.5 GB @ $8/GB) | <$0.05 | $150/mo maintenance | ~$207 |
| Firecrawl Starter | $16/mo | Included | Included | $75/mo maintenance | ~$91 |
| Self-hosted (Liquid Web) | $50/mo VPS | $28 (~3.5 GB @ $8/GB) | $2/mo | $375/mo maintenance | ~$455 |
At 10K pages/month, Firecrawl is cheapest if your targets are Firecrawl-compatible. Apify is competitive when you factor in the managed infrastructure. Self-hosting is still more expensive after maintenance hours.
100,000 pages/month
| Approach | Compute | Proxies | Storage | Dev/maintenance | Total/month |
|---|---|---|---|---|---|
| Apify Scale | $199 (included credits) | $175 (~25 GB @ $7/GB) | $0.50 | $375/mo maintenance | ~$750 |
| Firecrawl Standard | $83/mo | Included | Included | $225/mo maintenance | ~$308 |
| Self-hosted (Liquid Web) | $120/mo VPS | $200 (~25 GB residential) | $5/mo | $750/mo maintenance | ~$1,075 |
At 100K pages/month, Firecrawl remains cheapest if your scraping targets are compatible. Apify offers the best combination of managed infrastructure + proxy quality. Self-hosting is now significantly more expensive than Apify when you price maintenance honestly.
1,000,000 pages/month
| Approach | Compute | Proxies | Storage | Dev/maintenance | Total/month |
|---|---|---|---|---|---|
| Apify Business | $999 (included credits) | $1,400 (~200 GB @ $7/GB) | $3/mo | $750/mo maintenance | ~$3,152 |
| Firecrawl Growth | $333/mo | Included | Included | $750/mo maintenance | ~$1,083 |
| Self-hosted (Liquid Web) | $400/mo cluster | $1,600 (~200 GB residential) | $20/mo | $3,000/mo maintenance | ~$5,020 |
At 1M pages/month, Firecrawl wins on headline cost. However, Firecrawl's rate limits, target compatibility, and lack of custom Actor logic mean most teams at this scale use Apify or a hybrid approach. Self-hosting requires a full-time DevOps engineer, making it the most expensive real option.
Managed vs self-hosted: honest trade-offs
The budget tables above tell one story. Here's the full picture.
| Factor | Apify managed | Firecrawl API | Self-hosted (Liquid Web) |
|---|---|---|---|
| Setup time | Hours | Minutes | Days to weeks |
| Maintenance burden | Low (platform handles infra) | Very low | High (your team) |
| Anti-bot handling | Built-in (proxy rotation, retries) | Built-in | DIY |
| Custom scraping logic | Full Crawlee/Playwright support | Limited to crawl+extract | Unlimited |
| Pre-built scrapers | 19,000+ Actors in Apify Store | None | None |
| Proxy quality | Residential + datacenter included | Managed by Firecrawl | Must provision separately |
| Scale ceiling | 256 GB RAM concurrent (Business) | API rate limits apply | Unlimited (with spend) |
| Vendor lock-in | Medium (Actors are portable code) | Medium (API dependency) | None |
| Data privacy | Apify cloud stores your data | Firecrawl cloud | Your infrastructure |
| Best for | Teams that need custom + managed | Simple crawl-and-extract jobs | Full control, data sovereignty |
How to reduce your scraping costs
1. Start with HTML scraping when possible
JavaScript rendering costs 3–10× more than plain HTTP. If a target site loads core data without JS (check with curl or disable JS in browser devtools), use a lightweight HTML scraper.
2. Use datacenter proxies for non-protected targets
Residential proxies cost $7–$8/GB. Datacenter proxies cost $0.60–$1.30/GB — that's an 8–10× reduction. Use residential only when the target actively blocks datacenter ASNs.
3. Cache aggressively
Every page you scrape twice is a cost you pay twice. Build in deduplication at the URL level. For recurring jobs, only re-scrape pages that changed (use If-Modified-Since headers or content hashing).
4. Use non-expiring bandwidth for burst workloads
If you run a large quarterly scraping project rather than daily pipelines, IPRoyal's non-expiring residential bandwidth means you pay once per GB and use it when you need it — no monthly subscription waste.
5. Use Apify Store Actors before building custom scrapers
For common targets (Google Maps, Instagram, LinkedIn, Amazon), someone has already built and maintained an Actor on the Apify Store. A pre-built Google Maps Actor eliminates 10–40 hours of development time. That's $750–$3,000 in developer cost saved before you scrape your first record.
6. Match your plan to actual usage
Apify's prepaid credits don't roll over — unused credits expire at the end of each billing cycle (confirmed in Apify's FAQ). If you're consistently using 60% of your Scale plan credits, you're overpaying by 40%. Monitor your usage dashboard and downgrade if you're under-utilizing.
Hidden costs checklist
Before finalizing your scraping budget, check for these frequently missed costs:
- CAPTCHA solving: Some targets require CAPTCHA bypass services. Expect $1–$3 per 1,000 CAPTCHAs solved (~$1–$3/month at low volume, $50–$300/month at scale).
- IP bans and retries: Failed requests that require retries count toward compute and proxy usage. A 20% retry rate inflates your effective proxy cost by 20%.
- Data transformation: Cleaning, deduplicating, and normalizing raw scraped data adds 1–3 hours/month of data engineering time.
- Monitoring and alerting: Production scrapers need uptime monitoring. Allow 1–2 hours/month for alert triage.
- Legal review: For commercial data collection, a one-time legal review of your use case and target sites is worth budgeting. Roughly $500–$2,000 for an initial opinion.
- Storage egress: Exporting large datasets from cloud storage incurs egress fees. AWS S3 charges ~$0.09/GB for data transferred out to the internet.
Which approach should you choose?
Use Apify if you:
- Need custom scraping logic beyond simple crawl-and-extract
- Want access to 19,000+ pre-built scrapers for common targets
- Are scraping anti-bot protected sites and want proxies + retry logic bundled
- Have a team that can write Crawlee or Playwright code
- Want reliable scheduling, storage, and monitoring without DevOps overhead
Use Firecrawl if you:
- Need clean markdown or structured data from publicly accessible sites
- Are building LLM pipelines and want to skip scraper maintenance entirely
- Are under 100K pages/month and want the simplest possible setup
- Your targets are content sites, documentation, or non-protected e-commerce
Use self-hosted on Liquid Web if you:
- Have strict data sovereignty requirements (data cannot leave your infrastructure)
- Are running workloads at very high scale where the economics justify the DevOps cost
- Need to combine scraping with other server workloads on the same infrastructure
- Have an existing engineering team experienced in Scrapy or Playwright infrastructure
Use Bright Data proxies if you:
- Are on any platform but need enterprise-grade proxy quality
- Are targeting sites with aggressive bot protection (Datadome, Akamai, Cloudflare)
- Want the broadest IP pool and highest success rate at scale
Use IPRoyal proxies if you:
- Have bursty, intermittent scraping needs (monthly or quarterly)
- Don't want to lose unused bandwidth at month-end
- Are in the $7/GB residential price range and want non-expiring credits
FAQ
Web scraping costs $0–$5,000+ per month depending on scale and approach. At 1,000 pages/month, a managed API like Firecrawl costs under $50. At 1,000,000 pages/month, expect $1,000–$5,000+ when including compute, proxies, storage, and developer maintenance time. The biggest variable is proxy cost — residential proxies at $7–$8/GB can dominate your budget at scale.
In most cases, yes — once you factor in developer maintenance time. Apify's managed infrastructure eliminates the cost of configuring proxy rotation, handling retries, managing server uptime, and maintaining custom scraping code. At 10K–100K pages/month, self-hosted infrastructure on a VPS typically costs 2–3× more than Apify when maintenance hours are included. At 1M+ pages/month with a large engineering team, self-hosting can become competitive.
The most underestimated hidden costs are: (1) developer maintenance time when target sites change — typically 2–10 hours/month per active scraper; (2) retry bandwidth from failed requests, which inflates proxy costs 10–30%; (3) CAPTCHA solving services for protected targets; (4) data egress fees when exporting large datasets from cloud storage; and (5) legal review costs for commercial data collection projects.
Bandwidth varies by page type: simple HTML pages use 50–150 KB each; e-commerce pages use 200–500 KB; JavaScript-heavy SPAs use 500 KB–2 MB per page. For 100K pages of typical e-commerce content, expect 20–50 GB of bandwidth, which costs $140–$420/month in residential proxies ($7/GB). Using datacenter proxies ($1/GB) for non-protected targets cuts this cost to $20–$50.
At low scale (<10K pages/month), Firecrawl's free or Starter tier is cheapest. At medium scale (10K–500K pages/month), Apify often wins when total cost of ownership includes maintenance. At very high scale (1M+ pages/month), Firecrawl Growth at $333/month can undercut Apify on headline cost, but lacks support for custom scraping targets. Self-hosting is rarely cheapest once developer maintenance is priced honestly.