Getting Started with Crawlee for Python: The Complete 2026 Guide

Contains affiliate links. Disclosure.

Quick Answer

Crawlee for Python is an open-source web scraping framework. Install with pip, create a crawler (HTTP or Playwright), define a request handler, and run it. It is the Python equivalent of Crawlee for Node.js. Pair it with an Apify account and the Apify CLI to push the same code to the cloud as an Actor.

Crawlee for Python is Apify’s library for production-grade scraping in Python: retries, sessions, proxy-friendly request flow, and first-class BeautifulSoup-based HTTP crawling plus Playwright when the DOM needs a real browser. Because Crawlee is open-source and built by Apify, the same project runs on your laptop today and on Apify’s serverless cloud tomorrow with no rewrite.

Which crawler should I use?

Crawlee for Python ships three crawler classes. Pick by how the target page delivers its content.

Crawler	Parsing engine	When to use
BeautifulSoupCrawler	BeautifulSoup (context.soup)	Static HTML, forgiving parsing, beginner-friendly
ParselCrawler	Parsel (context.selector, CSS/XPath)	Static HTML, faster, Scrapy-style selectors
PlaywrightCrawler	Headless browser (context.page)	JavaScript-rendered pages, infinite scroll, logins

Start with BeautifulSoupCrawler or ParselCrawler for HTTP-only sites. Reach for PlaywrightCrawler only when the data is not in the raw HTML, since a browser is slower and heavier.

Prerequisites

• Python 3.10+.
• An Apify account (sign up free).
• Node.js and npm (for apify-cli).

Step 1: Install the Apify CLI

npm install -g apify-cli

Step 2: Create your project

From an empty folder:

apify create my-first-scraper -t python-crawlee-beautifulsoup

Or add Crawlee to an existing project:

pip install "crawlee[beautifulsoup]"

For Playwright crawling you will also use:

pip install "crawlee[playwright]"
# Playwright browsers (once per environment)
playwright install

Step 3: HTTP crawler with BeautifulSoup (main.py)

The template centers on BeautifulSoupCrawler, @crawler.router.default_handler, context.soup, and await context.push_data(...) (Dataset rows).

Replace src/main.py with a minimal Hacker News headline scraper:

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext


async def main() -> None:
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=10)

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        title = context.soup.title.string if context.soup.title else ""
        await context.push_data(
            {
                "url": context.request.url,
                "title": title,
            }
        )

    await crawler.run(["https://news.ycombinator.com"])


if __name__ == "__main__":
    asyncio.run(main())

The typed BeautifulSoupCrawlingContext gives you autocomplete on context.soup, context.request, context.push_data, and context.enqueue_links. max_requests_per_crawl caps the run so a runaway crawl never balloons your bill.

Prefer Scrapy-style selectors? Use ParselCrawler

If you already think in CSS or XPath, ParselCrawler parses with Parsel (the same selector engine Scrapy uses) and is a touch faster than BeautifulSoup on large pages. Only the import and the selector calls change:

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=10)

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        title = context.selector.css("title::text").get()
        await context.push_data({"url": context.request.url, "title": title})
        await context.enqueue_links()

    await crawler.run(["https://news.ycombinator.com"])


if __name__ == "__main__":
    asyncio.run(main())

Step 4: Playwright crawler (JavaScript-heavy pages)

Use PlaywrightCrawler when content is rendered in the browser. This pattern matches the official Playwright crawler example:

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
)


async def main() -> None:
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=10,
        headless=True,
        browser_type="chromium",
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        posts = await context.page.query_selector_all(".athing")
        rows = []

        for post in posts:
            title_el = await post.query_selector(".title a")
            rank_el = await post.query_selector(".rank")
            title = await title_el.inner_text() if title_el else None
            rank = await rank_el.inner_text() if rank_el else None
            href = await title_el.get_attribute("href") if title_el else None
            rows.append({"title": title, "rank": rank, "href": href})

        await context.push_data(rows)
        await context.enqueue_links(selector=".morelink")

    await crawler.run(["https://news.ycombinator.com/"])


if __name__ == "__main__":
    asyncio.run(main())

Working with RequestQueue and Dataset

enqueue_links and push_data are convenience wrappers over two storages Crawlee manages for you: the RequestQueue (the crawl frontier of URLs still to visit) and the Dataset (your output rows). For most crawls the defaults are all you need, but you can open them explicitly to seed URLs up front or read results back at the end.

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.storages import Dataset, RequestQueue


async def main() -> None:
    # Seed the frontier before the crawl starts
    queue = await RequestQueue.open()
    await queue.add_requests(
        [
            "https://news.ycombinator.com/news?p=1",
            "https://news.ycombinator.com/news?p=2",
        ]
    )

    crawler = BeautifulSoupCrawler(request_manager=queue)

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        for row in context.soup.select(".titleline a"):
            await context.push_data(
                {"title": row.get_text(), "href": row.get("href")}
            )

    await crawler.run()

    # Read the collected rows back, or export them to a file
    dataset = await Dataset.open()
    await dataset.export_to(key="results", content_type="csv")


if __name__ == "__main__":
    asyncio.run(main())

Locally, the RequestQueue and Dataset live under ./storage. When the same code runs on Apify, Crawlee transparently swaps in cloud-backed storages, so you never touch the persistence layer.

Step 5: Run locally

cd my-first-scraper
apify login
apify run

Results land under storage/datasets/default as JSON fragments.

Step 6: Deploy to Apify as an Actor

Crawlee runs anywhere Python runs, but to get scheduling, a managed proxy pool, cloud datasets, and an API endpoint, wrap your crawler in the Apify Actor context manager. The single line async with Actor: initializes the run, switches Crawlee to cloud storages, and handles a clean exit:

import asyncio

from apify import Actor
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext


async def main() -> None:
    async with Actor:
        crawler = BeautifulSoupCrawler(max_requests_per_crawl=50)

        @crawler.router.default_handler
        async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
            title = context.soup.title.string if context.soup.title else ""
            await context.push_data({"url": context.request.url, "title": title})

        await crawler.run(["https://news.ycombinator.com"])


if __name__ == "__main__":
    asyncio.run(main())

The Apify CLI templates already include this wrapper, so a project created with apify create is push-ready. Then deploy:

apify push

Start the new Actor from the Apify Console and open Storage to download the dataset. To swap your local IP for Apify’s rotating pool, pass a proxy configuration into the crawler. See rotating proxies for the pattern, and the Actor programming model for how input, storage, and lifecycle fit together.

Next steps

• Scraping dynamic websites: deeper Playwright and anti-blocking notes.
• Processing scraped data: clean and export at scale.
• Build an Apify Actor: publish to the Store.
• JavaScript Crawlee tutorial: the Node.js parallel.

More reference: Crawlee for Python docs and Apify Python SDK.

Apify Python web scraping: how Crawlee compares

Crawlee Python vs Scrapy

Topic	Crawlee Python	Scrapy
Concurrency	asyncio-first	Twisted-based
Browser	PlaywrightCrawler built in	Via plugins
Deploy to managed cloud	apify push	Self-hosted / third-party
Proxies & sessions	Integrated patterns	Bring-your-own

Choose Crawlee when you want one codebase from laptop to Apify cloud, strong Playwright support, and modern typing. Choose Scrapy when you depend on a large plugin ecosystem and existing Scrapy pipelines.

Crawlee vs “BeautifulSoup + requests”

BeautifulSoup only parses HTML. Crawlee adds the crawl frontier, retries, storage hooks, and optional Playwright, so the same patterns work for a 10-URL script and a 10M-page job.

Crawlee for Python vs Crawlee for Node.js

The two libraries share the same concepts (crawlers, routers, RequestQueue, Dataset, sessions) and both deploy to Apify the same way. Node.js Crawlee is older and has the wider feature set today (more crawler types, longer-tested anti-blocking). Python Crawlee is the natural fit if your data, ML, or pandas pipeline already lives in Python. The JavaScript Crawlee tutorial covers the Node.js side, and either path ends at building an Apify Actor.

Verdict

Crawlee for Python is a strong default for new Python scrapers that may land on Apify. It is younger than Scrapy; if your org is all-in on Scrapy, keep Scrapy unless Apify’s platform benefits justify a migration.

Frequently Asked Questions

It is an open-source Python framework for web scraping and crawling. Install with pip, pick an HTTP (BeautifulSoup) or Playwright crawler, implement a request handler, and run locally or on Apify. It mirrors Crawlee for Node.js.

Use pip: pip install "crawlee[beautifulsoup]" for HTML parsing over HTTP, pip install "crawlee[playwright]" plus playwright install for browser crawling, or pip install "crawlee[all]" for extras. The Apify CLI template python-crawlee-beautifulsoup pins compatible versions.

Yes for teams that want Crawlee’s features plus hosting: scheduling, datasets, proxies, and integrations. You can develop with Crawlee locally and push the same project as an Actor.

Both can run in production. Scrapy has more history and extensions; Crawlee Python offers asyncio-native design, built-in Playwright crawling, and tight integration with Apify. Pick based on ecosystem lock-in vs platform workflow.

Python 3.10 or newer.

Wrap the crawler in the Apify Actor context manager (async with Actor:), run apify run locally, then apify push to create or update an Actor on Apify. Runs store output in platform datasets you can export or pull via the API.

Use BeautifulSoupCrawler or ParselCrawler for static HTML over HTTP; ParselCrawler uses Parsel CSS/XPath selectors and is slightly faster, BeautifulSoup is more forgiving. Use PlaywrightCrawler only for JavaScript-rendered pages, since a real browser is slower and heavier.

RequestQueue is the crawl frontier of URLs still to visit (fed by enqueue_links or add_requests). Dataset is your output store (fed by push_data). Locally they live under ./storage; on Apify they become cloud-backed automatically with no code change.

They share the same concepts and both deploy to Apify identically. Node.js Crawlee is older with a wider feature set; Python Crawlee suits teams whose data or ML stack is already in Python.