Scale scrapers with async concurrency, job queues, and proxy pools. Grow Apify workloads using per-domain rate limits so millions of pages do not melt IPs.
Scaling scrapers means async concurrency, job queues, and proxy pools that grow without melting IPs. These guides cover per-domain rate limits and the patterns that take crawls from thousands to millions of pages.
Throughput comes from controlled concurrency and respectful pacing, not just more machines. Apify scales runs with these controls built in. Below you will find tutorials for scaling crawls safely.
Claude API applications that scrape data at scale hit rate limits and blocks. Bright Data proxies provide unblockable infrastructure: residential, datacenter, and ISP options that integrate with Python requests or httpx. If you are just getting started, you can try Claude free for a week and grab an Anthropic API key, then try Bright Data for AI pipelines.
A linear Python script with requests and a for loop over 500 URLs is not a production system. In real deployments, markup changes, socket timeouts, and bad proxy exits eventually break naive runs.
To move from a side project to production, your pipeline needs fault tolerance, state, and observability.
This guide covers four practical building blocks for running high-volume extraction reliably.
Scale horizontally by adding more concurrent actor runs rather than increasing single-run concurrency beyond safe limits. Use Apify's auto-scale feature to spawn actors based on queue depth. Distribute URL sets across multiple actors. Monitor block rates as you scale—more concurrency often means more blocks unless proxy rotation scales proportionally.
Proxy pool exhaustion (not enough IPs relative to request volume), target site rate limiting (per-IP or per-session throttling), storage I/O (writing millions of records quickly), memory limits (browser pools leak over time), and network bandwidth. Profile each bottleneck independently before throwing more compute at a scaling problem.
For browser scrapers: 5-15 concurrent pages per 1GB RAM is typical. For HTTP scrapers: hundreds per core. Rate limits on target sites often constrain below infrastructure capacity—1-5 requests per second per IP on sensitive sites. Total throughput is target-site-constrained, not infrastructure-constrained, for most production operations.
Run at 10% of target concurrency for an hour and measure block rates, success rates, and cost per record. Gradually increase to 25%, then 50%, monitoring at each step. If block rates exceed 5%, pause and investigate before continuing. This prevents burning your proxy pool or getting your scraping IP ranges banned site-wide.
