Ship scrapers in Python: requests, BeautifulSoup, Scrapy, and Playwright patterns for reliable extraction pipelines you can run locally or on Apify Actors.
Python is the most popular language for web scraping thanks to libraries like Requests, BeautifulSoup, Scrapy, and Playwright. These guides cover the full toolkit: fetching pages, parsing HTML, handling dynamic JavaScript sites, and structuring output into JSON or CSV your analysis can use. Code examples run locally or deploy straight to Apify Actors.
Which library fits depends on the target. Requests plus BeautifulSoup handles static pages, Scrapy scales to large crawls with built-in pipelines, and Playwright drives a real browser for JavaScript-heavy sites. Below you will find step-by-step tutorials, library comparisons, and patterns for adding proxies, retries, and scheduling so your Python scrapers stay reliable in production.
An AI research agent automates the full loop: given a research question, it searches the web via Apify, fetches and reads pages, extracts key findings, and synthesizes a structured report. This guide walks you through building one with LangGraph and the Apify Python client.
Scrapy is the mature Python web crawling framework. Apify is a cloud platform (with Crawlee as its open-source framework) that handles infrastructure, scaling, and storage on top of Node.js.
They're not direct competitors — Scrapy is a code framework, Apify is a full platform — but teams frequently choose between them. This comparison covers where each excels.
Describe your scraping target to Claude and get Python Apify Actor code. Deploy with apify push. Claude does not run the scraper for you: it generates code you review, test, and deploy. New to Claude? You can try Claude free for a week to draft and iterate on Actor code before you commit. For pre-built targets, check the Apify Store first. Claude shines when you need a custom target: an internal portal, a niche marketplace, or a site with a unique layout. The Store covers popular sites (Indeed, Amazon, LinkedIn); Claude fills the gaps. Expect 15–30 minutes from prompt to first successful run for a simple target.
Fine-tuning adapts a pre-trained model to your domain — support tickets, legal docs, code style, or brand voice. QLoRA makes it practical on a single GPU: quantize the base model to 4-bit, train small LoRA adapters, then merge and deploy. This guide covers the full pipeline: dataset format, libraries, training script, saving and loading, evaluation, and cost estimates. For hardware, Liquid Web offers A100 and H100 GPU servers by the hour.
IPRoyal provides residential rotating proxies with a 32M+ IP pool, starting at approximately $7/GB for small purchases and scaling to ~$1.75/GB at 500 GB — with bandwidth that never expires. This guide covers the exact configuration for Python, Node.js, Playwright, and Crawlee.
Freshness note: Endpoint and format verified March 2026. Check IPRoyal dashboard for current credentials format.
LangGraph is LangChain's graph-based framework for building stateful, multi-step AI agents. Unlike simple chains, LangGraph lets you define nodes (functions), edges (transitions), conditional branching, loops, and human-in-the-loop checkpoints. It's the go-to choice for production agents that need persistence, interrupts, and complex control flow. Use Apify for web data in LangGraph.
Retrieval Augmented Generation (RAG) lets your chatbot answer from your own documents instead of relying solely on the model's training data. This guide shows you how to build a fully local RAG pipeline: documents → embeddings → ChromaDB vector store → retrieval → Ollama LLM → answers. No cloud APIs. No per-token billing. Everything runs on your machine or a Liquid Web GPU VPS for heavier workloads.
Playwright is the dominant headless browser for web scraping in 2026 — faster than Selenium, more reliable than Puppeteer, and with native support for Chromium, Firefox, and WebKit. This tutorial takes you from install to a production-ready scraper in under an hour.
Freshness note: Examples updated for Playwright v1.58.x (current as of March 2026). Check the official changelog for the latest release notes.
Python remains the dominant language for web scraping in 2026. Whether you need static HTML parsing, JavaScript-rendered pages, or production-grade crawlers, the Python ecosystem delivers: requests, BeautifulSoup, httpx, Playwright, Scrapy, and Crawlee for Python. This guide covers the full stack—libraries, comparison tables, code examples, and data storage—so you can choose and build with confidence. Try Apify for managed Python Actors or run Crawlee Python locally.
The Firecrawl Python SDK (firecrawl-py) provides a simple interface: scrape(), crawl(), and map_url(). Install with pip install firecrawl-py, initialize with your API key, and start scraping. The SDK returns snake_case by default and supports async via FirecrawlAsync.
Requests and BeautifulSoup cover static sites; Playwright-Python and Selenium handle JavaScript rendering; Scrapy provides a full framework for large crawls. Crawlee Python is the newest option, offering Apify Cloud integration with the ergonomics of Playwright. Choose based on whether you need browser rendering and how much framework structure you want.
Use asyncio with aiohttp for plain HTTP requests, or Playwright's async API for browser automation. Define an async def handler, gather tasks with asyncio.gather or a semaphore-limited pool, and await each network call. Apify's Python SDK handles the async event loop setup so your actor focuses on extraction logic.
Use a virtual environment (venv or conda), pin dependencies in requirements.txt, and lock with pip-tools. Docker containers ensure reproducibility in production. Apify actor templates include a pre-built Dockerfile for Python actors. Keep heavy dependencies like OpenCV optional behind try/except imports.
Crawlee Python is a port of the JavaScript Crawlee library to Python, providing HTTP and Playwright crawlers, request queue management, persistent storage, and Apify Cloud deployment. It abstracts retry logic, session rotation, and dataset writing so you write handlers rather than infrastructure code. Released in 2024 and growing rapidly.
