Complete Guide to Web Scraping with Python in 2026: Tools, Code, and Best Practices

Contains affiliate links. Disclosure.

Python remains the dominant language for web scraping in 2026. Whether you need static HTML parsing, JavaScript-rendered pages, or production-grade crawlers, the Python ecosystem delivers: requests, BeautifulSoup, httpx, Playwright, Scrapy, and Crawlee for Python. This guide covers the full stack—libraries, comparison tables, code examples, and data storage—so you can choose and build with confidence. Try Apify for managed Python Actors or run Crawlee Python locally.

Python Scraping Stack 2026: Overview

Library	JS Rendering	Speed	Async Support	Best For
requests + BeautifulSoup	❌	Fast	❌	Static HTML, quick prototypes
httpx + BeautifulSoup	❌	Fast	✅	High-concurrency static scraping
Playwright for Python	✅	Slower	✅	SPAs, anti-bot sites
Scrapy	Via add-ons	Very fast	Via Twisted	Large-scale crawling, pipelines
Crawlee for Python	✅ (Playwright) / ❌ (BeautifulSoup)	Medium–Fast	✅	Apify-ready, typed crawlers

Static sites (documentation, blogs, simple product pages) work with requests and BeautifulSoup. JavaScript-heavy pages, SPAs, and anti-bot targets require Playwright or a browser-based crawler. For production pipelines and Apify deployment, Crawlee for Python or Scrapy are the go-to choices.

requests + BeautifulSoup: Quick Static Scraping

For simple HTML pages, nothing beats the classic combo:

import requests
from bs4 import BeautifulSoup

def scrape_links(url: str) -> list[str]:
    resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (compatible; Bot/1.0)"})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    return [a["href"] for a in soup.select("a[href]") if a.get("href", "").startswith("http")]

links = scrape_links("https://example.com")

BeautifulSoup parses HTML with CSS selectors (soup.select) or find methods. Extract text with .get_text(), attributes with ["href"]. For production, add retries, proxies, and rate limiting—see Apify error handling.

httpx + Async: High-Concurrency Static Scraping

When you need to hit hundreds of URLs concurrently, httpx's async support scales:

import asyncio
import httpx
from bs4 import BeautifulSoup

async def fetch_and_parse(client: httpx.AsyncClient, url: str) -> dict | None:
    try:
        resp = await client.get(url, timeout=15.0)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "html.parser")
        return {"url": url, "title": soup.select_one("title").get_text(strip=True) if soup.select_one("title") else None}
    except Exception as e:
        return {"url": url, "error": str(e)}

async def scrape_many(urls: list[str], max_concurrent: int = 20) -> list[dict]:
    async with httpx.AsyncClient(headers={"User-Agent": "Mozilla/5.0 (compatible; Bot/1.0)"}) as client:
        sem = asyncio.Semaphore(max_concurrent)
        async def bounded_fetch(url): 
            async with sem: 
                return await fetch_and_parse(client, url)
        return await asyncio.gather(*[bounded_fetch(u) for u in urls])

Use a semaphore to cap concurrency and avoid overwhelming the target. For TLS fingerprinting-sensitive sites, httpx alone may be detected; consider Playwright or tls-client.

Playwright for Python: Dynamic Pages and Browser Automation

Playwright drives a real browser. Use it when the target requires JavaScript execution, cookies, or human-like behavior:

from playwright.sync_api import sync_playwright

def scrape_spa(url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        page.wait_for_selector(".product-listing", timeout=15_000)
        products = page.locator(".product-card").evaluate_all(
            """nodes => nodes.map(n => ({
                title: n.querySelector(".title")?.textContent?.trim(),
                price: n.querySelector(".price")?.textContent?.trim()
            }))"""
        )
        browser.close()
        return {"url": url, "products": products}

Playwright auto-waits for elements and network idle. For async, use async_playwright. Anti-bot sites may still block headless Chrome; pair with Bright Data Scraping Browser or residential proxies.

Scrapy: Spider Architecture, Pipelines, Middlewares

Scrapy is a full-featured framework: spiders, pipelines, middlewares, and settings. Ideal for large-scale crawling with structured output.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for item in response.css(".product-card"):
            yield {
                "title": item.css(".title::text").get(),
                "price": item.css(".price::text").get(),
            }
        next_page = response.css("a.next-page::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Configure pipelines for deduplication, validation, and DB writes. Use middlewares for proxy rotation and User-Agent rotation. Scrapy's default is static HTML; add scrapy-playwright for JS rendering.

Crawlee for Python: Apify's Typed Crawler

Crawlee for Python provides PlaywrightCrawler and BeautifulSoupCrawler with a request queue, automatic retries, and optional Apify storage:

from crawlee.playwright_crawler import PlaywrightCrawler

async def main():
    crawler = PlaywrightCrawler()
    
    @crawler.router.default_handler
    async def handler(context):
        page = context.page
        await page.wait_for_selector(".content", timeout=10_000)
        items = await page.locator(".item").evaluate_all(
            """nodes => nodes.map(n => ({ text: n.textContent.trim() }))"""
        )
        for item in items:
            await context.push_data(item)
    
    await crawler.run(
        urls=["https://example.com/page-1", "https://example.com/page-2"],
        max_requests_per_crawl=100,
    )

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Crawlee handles concurrency, retries, and request queue semantics. Deploy to Apify by wrapping this logic in an Actor—see building an Apify Actor with TypeScript for the Actor pattern (TypeScript example applies conceptually to Python).

Data Storage: pandas, SQLite, PostgreSQL

pandas DataFrame — quick analysis and export:

import pandas as pd

df = pd.DataFrame(scraped_items)
df.to_csv("output.csv", index=False)
df.to_parquet("output.parquet", index=False)

SQLite — lightweight, file-based:

import sqlite3

conn = sqlite3.connect("scraped.db")
df.to_sql("products", conn, if_exists="append", index=False)

PostgreSQL — production pipelines, with psycopg3:

import psycopg
from psycopg.rows import dict_row

with psycopg.connect("postgresql://user:pass@localhost/db", row_factory=dict_row) as conn:
    with conn.cursor() as cur:
        for item in scraped_items:
            cur.execute(
                """INSERT INTO products (url, title, price) VALUES (%s, %s, %s)
                   ON CONFLICT (url) DO UPDATE SET title = EXCLUDED.title, price = EXCLUDED.price""",
                (item["url"], item["title"], item["price"])
            )

Design your schema for idempotent writes (e.g., ON CONFLICT) and downstream pipelines. See web scraping architecture patterns for staging and ETL integration.

Code Example: Crawlee Python Reading 100 URLs Concurrently

from crawlee.beautiful_soup_crawler import BeautifulSoupCrawler

async def main():
    crawler = BeautifulSoupCrawler(max_concurrency=10)
    
    @crawler.router.default_handler
    async def extract(context):
        soup = context.soup
        for link in soup.select("a[href^='http']"):
            await context.push_data({"url": context.request.url, "link": link.get("href")})
    
    urls = [f"https://example.com/page-{i}" for i in range(100)]
    await crawler.run(urls=urls, max_requests_per_crawl=100)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Crawlee manages the queue and concurrency. For Playwright-based crawling, swap BeautifulSoupCrawler for PlaywrightCrawler and use context.page instead of context.soup.

When to Choose Python Over JavaScript

Python dominates data science and ML pipelines. If your scraped data feeds into pandas, scikit-learn, or a Jupyter workflow, Python keeps everything in one stack. Scrapy's mature ecosystem (middlewares, pipelines, extensive documentation) also favors Python for large-scale projects. JavaScript/TypeScript excels when you want to deploy directly to Apify (Crawlee is Node-first), share code with frontend tooling, or leverage the npm ecosystem. For most scrapers, either language works; choose based on team preference and downstream tooling.

Production Checklist

Before scaling your Python scraper:

• Proxies — Add rotation for targets with rate limits. Configure via environment or Apify proxy.
• Retries — Exponential backoff on 429, 503. Crawlee and Scrapy have built-in retry logic.
• Schema versioning — Tag output with schema version for pipeline compatibility.
• Logging — Structured logs (JSON) with request ID, URL, status. Helps with error handling and observability.
• Resource limits — Cap memory and CPU when running in containers. Playwright browsers are memory-heavy.

Common Pitfalls and Fixes

Selectors break after site redesign — Prefer semantic selectors (IDs, data attributes) over brittle class names. Use relative XPath or CSS that targets structure. Version your selectors and add tests.

Memory leaks with Playwright — Always close browsers and pages. Use async with or try/finally. For long runs, consider launching a fresh browser every N requests to clear accumulated memory.

Blocking the event loop — In async code, avoid requests.get (blocking). Use httpx.AsyncClient or aiohttp. For CPU-bound parsing, offload to run_in_executor if it blocks.

Quick Reference: Install Commands

# Static HTML
pip install requests beautifulsoup4

# Async static
pip install httpx beautifulsoup4

# Browser automation
pip install playwright && playwright install chromium

# Full framework
pip install crawlee[playwright]  # or crawlee[beautifulsoup]

# Scrapy
pip install scrapy

For Apify deployment, add apify-sdk and structure your script to call Actor.init(), Actor.getInput(), and Actor.pushData(). Crawlee for Python integrates natively; Scrapy requires wrapping in an Actor entrypoint.

Match library to target

Static HTML → requests/BeautifulSoup or Crawlee BeautifulSoupCrawler. JavaScript-rendered → Playwright or Crawlee PlaywrightCrawler. Large scale + pipelines → Scrapy. Apify deployment → Crawlee Python.

Try Apify | Bright Data Proxies

Frequently Asked Questions

Use BeautifulSoup (with requests or httpx) for static HTML. Use Playwright when the page renders content via JavaScript, requires cookies, or has anti-bot protection.

Scrapy excels at static HTML at scale with pipelines and middlewares. For JavaScript-heavy targets, use Scrapy with scrapy-playwright or consider Crawlee Python with PlaywrightCrawler.

Use httpx.AsyncClient with asyncio.gather and a semaphore to limit concurrency. Or use Crawlee for Python, which manages the request queue and concurrency for you.

Yes. Apify supports Python Actors. Use Crawlee for Python for an Apify-ready structure, or build a custom Actor with your preferred stack (Scrapy, Playwright) and add the Apify SDK for storage and input.

For quick analysis: pandas DataFrame → CSV or Parquet. For pipelines: PostgreSQL with upsert (ON CONFLICT). For lightweight: SQLite. Design schemas for idempotent writes.