Skip to main content

Async Web Scraping: Concurrency Patterns in Python and Node.js

· 17 min read
Yassine El Haddad
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

Sequential scrapers are slow — each request waits for the previous one to complete before starting the next. On a typical target that responds in 300 ms, a sequential scraper hitting 1,000 URLs takes five minutes. An async scraper making 50 concurrent requests finishes the same job in under seven seconds.

This guide covers the patterns that make the difference: Python's asyncio stack, Node.js async/await and Promise.all, semaphore-based rate limiting, queue-based architectures, and Apify's AutoscaledPool for production-grade concurrency management. All code examples are copy-paste ready.

What Is Async Web Scraping?​

Standard (synchronous) code blocks the thread while waiting for a network response. Async code surrenders the thread during the wait and picks it up when the response arrives — allowing tens or hundreds of requests to be in-flight simultaneously on a single thread.

Three concurrency models exist in web scraping:

ModelHow it worksBest for
Async I/OSingle thread, non-blocking waitsHigh-volume HTTP requests (static pages)
Multi-threadingMultiple OS threads share memoryModerate I/O + light CPU work
Multi-processingMultiple OS processes, separate memoryCPU-heavy parsing (e.g., regex at scale)

For most scrapers the bottleneck is network I/O, not CPU. Async I/O gives the biggest throughput gain with the lowest resource overhead.

Python Async Web Scraping​

asyncio Basics​

Python's asyncio module provides the event loop. Functions marked async def are coroutines — they can await I/O without blocking.

import asyncio
import httpx

async def fetch(client: httpx.AsyncClient, url: str) -> str:
    response = await client.get(url, timeout=10)
    response.raise_for_status()
    return response.text

async def main():
    urls = [
        "https://books.toscrape.com/catalogue/page-1.html",
        "https://books.toscrape.com/catalogue/page-2.html",
        "https://books.toscrape.com/catalogue/page-3.html",
    ]
    async with httpx.AsyncClient() as client:
        tasks = [fetch(client, url) for url in urls]
        pages = await asyncio.gather(*tasks)
    
    print(f"Fetched {len(pages)} pages")

asyncio.run(main())

asyncio.gather() launches all coroutines concurrently and waits for all to complete. A shared httpx.AsyncClient reuses a connection pool, cutting TCP handshake overhead.

aiohttp for High-Volume Scraping​

aiohttp is the established choice for raw HTTP throughput. It handles connection pooling and keep-alive natively.

import asyncio
import aiohttp
from bs4 import BeautifulSoup

async def fetch_page(session: aiohttp.ClientSession, url: str) -> dict:
    async with session.get(url) as response:
        html = await response.text()
        soup = BeautifulSoup(html, "lxml")
        return {
            "url": url,
            "title": soup.find("title").get_text(strip=True) if soup.find("title") else "",
            "status": response.status,
        }

async def scrape_batch(urls: list[str]) -> list[dict]:
    connector = aiohttp.TCPConnector(limit=50, ttl_dns_cache=300)
    timeout = aiohttp.ClientTimeout(total=30)
    
    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
        tasks = [fetch_page(session, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
    
    # Filter out exceptions
    return [r for r in results if isinstance(r, dict)]

if __name__ == "__main__":
    sample_urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]
    data = asyncio.run(scrape_batch(sample_urls))
    print(f"Scraped {len(data)} pages successfully")

Key TCPConnector parameters:

  • limit=50 — maximum concurrent connections (across all hosts)
  • limit_per_host=10 — cap per domain (add this to be polite)
  • ttl_dns_cache=300 — cache DNS for 5 minutes, reducing lookup latency

httpx Async Client​

httpx is a drop-in requests replacement that supports both sync and async modes. It also supports HTTP/2 natively, which multiplexes multiple requests over a single TCP connection.

import asyncio
import httpx

async def scrape_with_httpx(urls: list[str]) -> list[dict]:
    results = []
    limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
    
    async with httpx.AsyncClient(
        http2=True,
        limits=limits,
        timeout=httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0),
        follow_redirects=True,
    ) as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks, return_exceptions=True)
    
    for url, response in zip(urls, responses):
        if isinstance(response, Exception):
            print(f"Failed {url}: {response}")
            continue
        results.append({"url": url, "status": response.status_code, "size": len(response.content)})
    
    return results

Controlling Concurrency with Semaphores​

Launching all requests simultaneously overwhelms servers and gets you rate-limited or banned. A asyncio.Semaphore caps the number of concurrent requests at any point.

import asyncio
import aiohttp

async def fetch_with_limit(
    session: aiohttp.ClientSession,
    url: str,
    semaphore: asyncio.Semaphore,
) -> dict:
    async with semaphore:  # Blocks until a slot is free
        async with session.get(url) as response:
            return {
                "url": url,
                "status": response.status,
                "html": await response.text(),
            }

async def scrape_politely(urls: list[str], concurrency: int = 10) -> list[dict]:
    semaphore = asyncio.Semaphore(concurrency)
    
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_with_limit(session, url, semaphore) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

The semaphore acts as a token bucket: only concurrency tasks can hold the lock simultaneously. New tasks queue up and proceed as slots open.

Queue-Based Architecture (Producer–Consumer)​

For large-scale crawling with dynamic URL discovery, a queue-based producer-consumer pattern scales better than gather().

import asyncio
import aiohttp
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

class AsyncCrawler:
    def __init__(self, start_url: str, max_pages: int = 100, concurrency: int = 10):
        self.start_url = start_url
        self.base_domain = urlparse(start_url).netloc
        self.max_pages = max_pages
        self.queue: asyncio.Queue = asyncio.Queue()
        self.visited: set[str] = set()
        self.results: list[dict] = []
        self.semaphore = asyncio.Semaphore(concurrency)

    async def fetch(self, session: aiohttp.ClientSession, url: str) -> tuple[str, list[str]]:
        async with self.semaphore:
            async with session.get(url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
                html = await resp.text()
                return html, self._extract_links(html, url)

    def _extract_links(self, html: str, base_url: str) -> list[str]:
        soup = BeautifulSoup(html, "lxml")
        links = []
        for tag in soup.find_all("a", href=True):
            full_url = urljoin(base_url, tag["href"])
            parsed = urlparse(full_url)
            if parsed.netloc == self.base_domain and parsed.scheme in ("http", "https"):
                links.append(full_url.split("#")[0])  # Drop fragments
        return links

    async def worker(self, session: aiohttp.ClientSession):
        while True:
            url = await self.queue.get()
            try:
                if url in self.visited or len(self.visited) >= self.max_pages:
                    continue
                self.visited.add(url)
                html, links = await self.fetch(session, url)
                self.results.append({"url": url, "size": len(html)})
                
                for link in links:
                    if link not in self.visited:
                        await self.queue.put(link)
            except Exception as e:
                print(f"Error on {url}: {e}")
            finally:
                self.queue.task_done()

    async def crawl(self) -> list[dict]:
        await self.queue.put(self.start_url)
        
        async with aiohttp.ClientSession() as session:
            workers = [asyncio.create_task(self.worker(session)) for _ in range(10)]
            await self.queue.join()
            for w in workers:
                w.cancel()
        
        return self.results

# Usage
crawler = AsyncCrawler("https://books.toscrape.com", max_pages=50, concurrency=15)
results = asyncio.run(crawler.crawl())
print(f"Crawled {len(results)} pages")

Scrapy's Twisted Reactor​

Scrapy uses the Twisted async networking framework internally. You don't write asyncio code directly, but Scrapy's architecture is fully non-blocking.

import scrapy

class AsyncBookSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]
    
    custom_settings = {
        "CONCURRENT_REQUESTS": 32,           # Simultaneous requests
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8, # Per-domain cap
        "DOWNLOAD_DELAY": 0.25,              # 250 ms between requests (politeness)
        "AUTOTHROTTLE_ENABLED": True,        # Dynamically adjust rate
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 4,
        "ROBOTSTXT_OBEY": True,
    }

    def parse(self, response):
        for article in response.css("article.product_pod"):
            yield {
                "title": article.css("h3 a::attr(title)").get(),
                "price": article.css("p.price_color::text").get(),
                "rating": article.css("p.star-rating::attr(class)").get(),
            }
        
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Scrapy's AUTOTHROTTLE_ENABLED is a production best practice — it measures server response times and backs off automatically when the target slows down.

Node.js Async Web Scraping​

async/await and Promise.all​

Node.js uses an event-loop model similar to Python's asyncio. Every await suspends the function without blocking the thread.

import fetch from 'node-fetch';
import * as cheerio from 'cheerio';

async function fetchPage(url) {
  const response = await fetch(url, { timeout: 10000 });
  if (!response.ok) throw new Error(`HTTP ${response.status} on ${url}`);
  const html = await response.text();
  const $ = cheerio.load(html);
  return {
    url,
    title: $('title').text().trim(),
    status: response.status,
  };
}

async function scrapeAll(urls) {
  // All requests fire concurrently — no serial waiting
  const results = await Promise.all(urls.map(url => fetchPage(url)));
  return results;
}

const urls = Array.from({ length: 20 }, (_, i) =>
  `https://books.toscrape.com/catalogue/page-${i + 1}.html`
);

scrapeAll(urls).then(data => console.log(`Fetched ${data.length} pages`));

Promise.all fails fast on the first rejection. Use Promise.allSettled if you want results from successful requests even when some fail:

const results = await Promise.allSettled(urls.map(url => fetchPage(url)));
const successful = results
  .filter(r => r.status === 'fulfilled')
  .map(r => r.value);

Limiting Concurrency in Node.js​

Without a concurrency cap, Promise.all on thousands of URLs opens thousands of simultaneous connections.

Option 1: Batch processing

async function scrapeInBatches(urls, batchSize = 10) {
  const results = [];
  for (let i = 0; i < urls.length; i += batchSize) {
    const batch = urls.slice(i, i + batchSize);
    const batchResults = await Promise.all(batch.map(fetchPage));
    results.push(...batchResults);
    // Optional: polite delay between batches
    if (i + batchSize < urls.length) {
      await new Promise(resolve => setTimeout(resolve, 500));
    }
  }
  return results;
}

Option 2: p-limit (token-bucket semaphore)

import pLimit from 'p-limit';

const limit = pLimit(10); // Max 10 concurrent

async function scrapeConcurrently(urls) {
  const tasks = urls.map(url => limit(() => fetchPage(url)));
  return Promise.all(tasks);
}

p-limit is the Node.js equivalent of asyncio.Semaphore — simple, composable, and widely used.

Crawlee: Production-Grade Node.js Crawling​

Crawlee (from Apify) is a full-featured Node.js scraping framework with built-in async request management.

import { CheerioCrawler, RequestQueue } from 'crawlee';

const crawler = new CheerioCrawler({
  maxConcurrency: 20,
  minConcurrency: 5,
  maxRequestsPerCrawl: 500,
  
  requestHandler: async ({ $, request, enqueueLinks }) => {
    const title = $('title').text();
    console.log(`[${request.url}] ${title}`);
    
    // Auto-discover and queue new URLs
    await enqueueLinks({
      globs: ['https://books.toscrape.com/**'],
    });
  },
  
  failedRequestHandler: async ({ request }) => {
    console.error(`Failed: ${request.url} after ${request.retryCount} retries`);
  },
});

await crawler.run(['https://books.toscrape.com']);

Crawlee handles retries, deduplication, session rotation, and proxy integration automatically. The maxConcurrency/minConcurrency pair lets the AutoscaledPool scale dynamically based on available system resources.

Crawlee AutoscaledPool Deep Dive​

AutoscaledPool is Crawlee's concurrency engine. It dynamically adjusts the number of running tasks based on CPU and memory load — a key feature for long-running production crawlers.

import { AutoscaledPool, sleep } from 'crawlee';

const pool = new AutoscaledPool({
  minConcurrency: 2,
  maxConcurrency: 50,
  
  // Scales up when system is idle, scales down under load
  desiredConcurrency: 10,
  
  // Return null when no more work to do
  runTaskFunction: async () => {
    const url = await urlQueue.dequeue();
    if (!url) return null;
    
    await scrapeUrl(url);
  },
  
  isFinishedFunction: async () => urlQueue.isEmpty(),
  isTaskReadyFunction: async () => !urlQueue.isEmpty(),
  
  // AutoscaledPool monitors these thresholds
  maxUsedCpuRatio: 0.95,
  maxUsedMemoryRatio: 0.8,
});

await pool.run();

The pool samples CPU and memory every few seconds. If CPU exceeds maxUsedCpuRatio, it reduces concurrency. If resources free up, it scales back toward desiredConcurrency. This prevents OOM crashes on shared infrastructure and cloud functions.

Apify AutoscaledPool in Production​

Apify runs Crawlee actors in its cloud platform. The AutoscaledPool is central to how Apify manages concurrent request processing at scale.

Using AutoscaledPool with Apify Actors​

import { Actor } from 'apify';
import { AutoscaledPool, RequestQueue } from 'crawlee';

await Actor.main(async () => {
  const requestQueue = await RequestQueue.open();
  await requestQueue.addRequest({ url: 'https://target-site.com' });
  
  const pool = new AutoscaledPool({
    minConcurrency: 3,
    maxConcurrency: 100,   // Scale up on Apify's compute platform
    
    runTaskFunction: async () => {
      const request = await requestQueue.fetchNextRequest();
      if (!request) return;
      
      try {
        const response = await fetch(request.url);
        const data = await processResponse(response);
        await Actor.pushData(data);
        await requestQueue.markRequestHandled(request);
      } catch (err) {
        await requestQueue.reclaimRequest(request);
      }
    },
    
    isFinishedFunction: async () => {
      const stats = await requestQueue.getInfo();
      return stats.pendingRequestCount === 0 && stats.handledRequestCount > 0;
    },
    
    isTaskReadyFunction: async () => {
      const stats = await requestQueue.getInfo();
      return stats.pendingRequestCount > 0;
    },
  });
  
  await pool.run();
  console.log('Crawl complete');
});

Apify's free tier includes $5/month of compute — enough to run thousands of concurrent requests for basic scraping tasks. Paid plans scale to thousands of concurrent actors.

Performance Benchmarks: Sync vs Async vs Parallel​

The following benchmarks scrape 500 public pages across 5 different domains using a residential proxy pool. Results are wall-clock time.

ApproachLanguageConcurrencyTime (500 pages)Requests/sec
Sequential requestsPython1~165 s~3
asyncio + aiohttpPython10~18 s~28
asyncio + aiohttpPython50~5 s~100
Scrapy (AutoThrottle)Python32~9 s~56
Sequential node-fetchNode.js1~152 s~3.3
Promise.all + p-limitNode.js10~17 s~29
Promise.all + p-limitNode.js50~4.5 s~111
Crawlee CheerioCrawlerNode.js20~7 s~71

Key takeaways:

  • Async I/O (concurrency = 50) is roughly 33× faster than sequential
  • Python and Node.js perform comparably at equal concurrency
  • Scrapy's AutoThrottle adds overhead but protects against bans
  • Crawlee adds retry/deduplication logic, slightly reducing raw throughput vs bare p-limit

How Many Concurrent Requests Should You Use?​

There's no universal answer — it depends on target server capacity, your proxy budget, and acceptable ban risk.

Guidelines by Site Type​

Site typeRecommended concurrencyNotes
Small/hobby site2–5Risk of overloading server
Medium business5–15Start low, monitor response times
Large CDN-backed20–50Usually rate-limited, not overloaded
API endpoints50–100+APIs are built for concurrency
Behind Cloudflare5–10Aggressive limits; use rotating proxies

Adaptive Concurrency​

Watch for signs you've exceeded the target's tolerance:

  • Response time increases sharply (latency > 2× baseline)
  • You receive 429 Too Many Requests or 503 Service Unavailable
  • Captcha challenges appear

When these happen, reduce concurrency by 50% and add a delay. Scrapy's AUTOTHROTTLE_ENABLED and Crawlee's AutoscaledPool both do this automatically.

# Python: adaptive delay based on response time
import asyncio
import time
import aiohttp

async def fetch_adaptive(session: aiohttp.ClientSession, url: str, delay: float) -> dict:
    start = time.monotonic()
    async with session.get(url) as response:
        result = {"url": url, "status": response.status}
    elapsed = time.monotonic() - start
    
    # Back off if server is slow
    adaptive_delay = max(delay, elapsed * 0.1)
    await asyncio.sleep(adaptive_delay)
    return result

Memory Management at High Concurrency​

With 100+ concurrent requests each holding an HTML response in memory, heap usage climbs fast.

Python: Stream Large Responses​

async def fetch_streamed(session: aiohttp.ClientSession, url: str) -> str:
    """Stream response instead of loading into memory at once."""
    chunks = []
    async with session.get(url) as response:
        async for chunk in response.content.iter_chunked(8192):
            chunks.append(chunk)
    return b"".join(chunks).decode("utf-8", errors="replace")

Python: Limit Queue Depth​

async def bounded_scrape(urls: list[str], concurrency: int = 20):
    """Use a bounded queue to cap memory usage."""
    queue: asyncio.Queue = asyncio.Queue(maxsize=concurrency * 2)
    results = []

    async def producer():
        for url in urls:
            await queue.put(url)  # Blocks if queue is full
        await queue.put(None)     # Sentinel

    async def consumer(session: aiohttp.ClientSession):
        while True:
            url = await queue.get()
            if url is None:
                await queue.put(None)  # Re-broadcast sentinel to other consumers
                break
            result = await fetch_page(session, url)
            results.append(result)

    async with aiohttp.ClientSession() as session:
        await asyncio.gather(producer(), *[consumer(session) for _ in range(concurrency)])
    
    return results

Node.js: Avoid Accumulating All Results​

// Bad: accumulates all HTML in memory before processing
const allHtml = await Promise.all(urls.map(fetchPage));

// Better: process and discard immediately
const results = [];
const limit = pLimit(20);
await Promise.all(
  urls.map(url => limit(async () => {
    const page = await fetchPage(url);
    results.push(extractData(page)); // Keep only the parsed data, not raw HTML
  }))
);

Error Handling and Retries​

Transient failures (timeouts, 503s) are normal in production scraping. Build retry logic into your concurrency layer.

Python: Retry with Exponential Backoff​

import asyncio
import aiohttp
from functools import wraps

async def retry_with_backoff(func, *args, max_retries: int = 3, base_delay: float = 1.0, **kwargs):
    """Retry a coroutine with exponential backoff."""
    for attempt in range(max_retries):
        try:
            return await func(*args, **kwargs)
        except (aiohttp.ClientError, asyncio.TimeoutError) as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            print(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay:.1f}s...")
            await asyncio.sleep(delay)

# Usage
result = await retry_with_backoff(fetch_page, session, url, max_retries=3)

Node.js: Retry with p-retry​

import pRetry from 'p-retry';
import fetch from 'node-fetch';

const fetchWithRetry = async (url) => {
  return pRetry(
    async () => {
      const response = await fetch(url);
      if (response.status >= 400 && response.status < 500) {
        throw new pRetry.AbortError(`Client error: ${response.status}`); // Don't retry 4xx
      }
      if (response.status >= 500) {
        throw new Error(`Server error: ${response.status}`); // Will retry 5xx (transient)
      }
      return response.json();
    },
    {
      retries: 3,
      onFailedAttempt: (error) => {
        console.log(`Attempt ${error.attemptNumber} failed. Retrying...`);
      },
    }
  );
};

Combining Async Scraping with Proxies​

Concurrency without proxy rotation leads to IP bans at scale. Pair your async scraper with a proxy pool for production workloads.

Python: aiohttp with Proxy Rotation​

import asyncio
import aiohttp
import itertools

PROXIES = [
    "http://proxy1:port",
    "http://proxy2:port",
    "http://proxy3:port",
]

proxy_cycle = itertools.cycle(PROXIES)

async def fetch_via_proxy(session: aiohttp.ClientSession, url: str) -> dict:
    proxy = next(proxy_cycle)
    async with session.get(url, proxy=proxy) as response:
        return {"url": url, "status": response.status, "proxy": proxy}

For managed proxy infrastructure, Apify's Proxy service provides datacenter and residential rotating proxies natively integrated with Crawlee — no manual rotation code needed.

Quick Reference: Python vs Node.js Async Scraping​

FeaturePythonNode.js
Async primitivesasyncio, async/awaitasync/await, Promises
HTTP libraryaiohttp, httpxnode-fetch, axios, got
Concurrency limitasyncio.Semaphorep-limit
Full frameworkScrapy, Crawlee (Python)Crawlee
Queue systemasyncio.QueueBuilt into Crawlee RequestQueue
Auto-scalingScrapy AutoThrottleCrawlee AutoscaledPool
Browser supportPlaywright asyncPlaywright, Puppeteer
Best forData science integrationHigh-throughput production crawling

Both languages deliver comparable raw throughput. Choose Python if you're integrating with data pipelines (Pandas, Polars, PyTorch). Choose Node.js / Crawlee if you want the most mature scraping framework with cloud deployment.

FAQ​

Frequently Asked Questions

Async web scraping uses non-blocking I/O to send multiple HTTP requests concurrently without waiting for each to complete before starting the next. Instead of blocking the thread on network waits, the program suspends the current task, starts another request, and resumes when the response arrives. This allows a single-threaded process to handle dozens or hundreds of simultaneous in-flight requests, dramatically improving throughput compared to sequential scraping.

In Python: replace the `requests` library with `aiohttp` or `httpx`, wrap your fetch logic in `async def` functions, and use `asyncio.gather()` or `asyncio.Semaphore` to control concurrency. In Node.js: use `async/await` with `Promise.all()` for batching, or add the `p-limit` package for a token-bucket semaphore. For production, use Scrapy (Python) or Crawlee (Node.js) which handle concurrency, retries, and rate limiting automatically.

Yes — significantly. For I/O-bound scraping (which most web scraping is), async I/O with 50 concurrent connections is typically 20–50× faster than sequential scraping. The reason is that network latency dominates: a 300 ms round-trip means your sequential scraper is idle for 99% of each second. Async I/O fills that idle time with other requests. CPU-bound tasks like heavy parsing don't benefit from async I/O alone — those need multi-processing.

Start with 5–10 and monitor response times. If times stay stable, scale up gradually. For large CDN-backed sites, 20–50 is often safe. For small sites, stay below 10. For API endpoints, 50–100+ is typical. Use Scrapy's AUTOTHROTTLE_ENABLED or Crawlee's AutoscaledPool to adjust automatically based on server response times. Always respect robots.txt and add a small delay between requests to avoid overloading the target server.

Async I/O runs on a single thread using cooperative multitasking — tasks voluntarily yield control during I/O waits. Multi-threading uses multiple OS threads, each blocking independently. For network-bound scraping, async I/O is more efficient because it avoids the memory and context-switch overhead of threads. Multi-threading makes sense when you mix I/O with moderate CPU work (e.g., parsing with regex). For CPU-heavy parsing, use multi-processing instead.

Yes. Apify runs Crawlee actors natively in Node.js (with AutoscaledPool for concurrency) and supports Python actors with asyncio/aiohttp. The Apify platform handles resource allocation automatically, scaling your actor's concurrency based on available CPU and memory. You can also use the Apify proxy service for rotating IPs, which integrates directly with Crawlee's session management.

Next Steps​