Python Extraction Architectures: httpx vs Playwright vs Crawlee

Python is a common choice when your stack already lives there—PyTorch training loops, Polars pipelines, or internal services. Keeping extraction in Python avoids extra RPC glue between languages.

This guide walks from simple static fetches (httpx + BeautifulSoup) to browser automation and Crawlee for heavier jobs.

Contains affiliate links. Disclosure.

Layer 1: Async HTTP + parsing (httpx)

Start by moving off blocking requests, which waits on every I/O and limits concurrency. Pair httpx with BeautifulSoup for fast pulls of server-rendered HTML or stable JSON endpoints.

import httpx
from bs4 import BeautifulSoup

async def extract_telemetry(targets: list[str]) -> list[dict]:
    extracted_data = []
    
    # Persistent connection pool via context manager
    async with httpx.AsyncClient(http2=True) as client:
        for url in targets:
            response = await client.get(url)
            response.raise_for_status()
            
            # lxml backend is faster than pure-Python parsers for large HTML
            soup = BeautifulSoup(response.text, "lxml")
            
            for row in soup.select("table.datagrid tr"):
                cells = row.select("td")
                if cells:
                    extracted_data.append({
                        "id": cells[0].get_text(strip=True),
                        "value": cells[1].get_text(strip=True)
                    })
                    
    return extracted_data

Good fit: Large URL lists where the data is in HTML or JSON without a heavy client-side app.
Watch out (TLS fingerprinting): Many WAFs recognize default library TLS stacks. Stock httpx against a strict target (e.g. Cloudflare Turnstile) often returns 403. You may need impersonation libraries (curl_cffi) or a real browser.

Layer 2: Headless browsers (Playwright)

When content only appears after JavaScript runs (React, client routing), text-only HTTP is not enough. Use Playwright and drive a real page lifecycle.

import asyncio
from playwright.async_api import async_playwright

async def orchestrate_hydration(engine_uri: str) -> dict:
    async with async_playwright() as execution_engine:
        browser = await execution_engine.chromium.launch(headless=True)
        context = await browser.new_context()
        page = await context.new_page()
        
        await page.goto(engine_uri, wait_until="networkidle")
        
        metrics = await page.evaluate(
            "() => document.querySelector('#kpi-node').innerText"
        )
        
        await browser.close()
        return {"kpi": metrics}

Good fit: Anti-bot flows, forms, and SPAs where you need the DOM the user sees.
Watch out (orphan browsers): If an exception skips await browser.close(), Chromium can stay running. Run that often enough without cleanup and you will exhaust RAM. Prefer try/finally, context managers, or a framework that owns browser lifetime.

Layer 3: Production wrappers (Crawlee Python)

For queues, retries, storage, and concurrency limits, many teams wrap Playwright in Crawlee.

The Python release mirrors the JS version: request queues, datasets, and key-value stores, with less hand-rolled asyncio wiring.

import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

async def master_deployment():
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=2000,
        max_concurrency=8, 
        request_handler_timeout=45.0
    )

    @crawler.router.default_handler
    async def request_processor(context: PlaywrightCrawlingContext) -> None:
        await context.page.wait_for_selector(".product-matrix")
        
        skus = await context.page.locator(".product-id").all_inner_texts()
        
        await context.push_data({"uri": context.request.url, "skus": skus})
        
        await context.enqueue_links(selector="a.btn-next")

    await crawler.run(["https://engine-origin.com/products"])

if __name__ == '__main__':
    asyncio.run(master_deployment())

Good fit: Crawls at serious scale with predictable limits and durable queues.
Watch out (blocking the event loop): Crawler handlers run on the same asyncio loop as Playwright. Heavy synchronous pandas work or CPU-heavy crypto inside request_processor can stall the loop and cause timeouts. Offload that work with asyncio.to_thread() or separate workers.

Scaling and proxies

Even good code fails when IPs burn. Many teams ship the same patterns to the Apify Serverless Cluster so proxy rotation and fleet sizing are handled outside the notebook.

Frequently Asked Questions

Scrapy is still used, but it predates modern asyncio-first stacks. Its Twisted core does not fit asyncio as cleanly as today’s Playwright-first tools, and bolting Playwright into Scrapy middleware is often more brittle than Crawlee’s first-class browser support.

Less often for new builds. Selenium’s classic WebDriver model tends to add latency versus Playwright’s CDP-style control. For greenfield automation, Playwright is the more common default.

You can for small, synchronous scripts. Inside an async crawler, `requests` blocks the thread on each call and destroys concurrency—`httpx` async mode is the better match.