Crawlee: guides & tutorials

Crawlee is an open-source Node.js framework from Apify that bundles everything a production scraper needs: request deduplication, auto-retry, proxy rotation, session management, persistent storage, and Playwright/Puppeteer/HTTP crawlers under one API.

Where raw Playwright requires wiring all those pieces manually, Crawlee provides them out of the box — letting you focus on extraction logic.

Freshness note: Examples verified against Crawlee 3.x (March 2026). Install crawlee@latest to get the current release.

JavaScript and Node.js power some of the most capable web scrapers in 2026. The ecosystem spans Axios and node-fetch for HTTP, Cheerio for HTML parsing, Playwright and Puppeteer for browser automation, and Crawlee as the full framework that powers Apify Actors. This guide covers the JS scraping stack, comparison tables, TypeScript patterns, data output options, and a complete Crawlee TypeScript Actor example. Try Apify to run Crawlee Actors in the cloud.

Build an Apify Actor from scratch with TypeScript, Crawlee, and the Apify SDK. Scaffold with apify create, choose Cheerio or Playwright, define input, run locally, then push to the cloud. Start building with a free Apify account.

Crawlee is Apify’s open-source crawling library. It handles queues, retries, anti-bot fingerprinting, and storage. This guide covers deploying Crawlee on a Liquid Web VPS for production scraping.

Data engineering in 2026 is sharply divided by two distinct extraction paradigms: utilizing a managed API for rapid data normalization, or deploying an orchestration framework for deterministic, high-volume control.

The two dominant solutions representing these philosophies are Firecrawl (an API-first pipeline optimized for LLM ingestion) and Crawlee (the industry-standard open-source scraping framework maintained by Apify).

This guide provides a strict architectural comparison to determine which tool fits your extraction parameters.

Python is a common choice when your stack already lives there—PyTorch training loops, Polars pipelines, or internal services. Keeping extraction in Python avoids extra RPC glue between languages.

This guide walks from simple static fetches (httpx + BeautifulSoup) to browser automation and Crawlee for heavier jobs.

Node.js possesses outsized advantages for data extraction pipelines: its single-threaded, non-blocking asynchronous event loop naturally aligns with high-concurrency network I/O, and its DOM-manipulation syntax mirrors native browser behavior.

This guide provides a formal architectural breakdown of the three primary abstraction layers available to JavaScript data engineers in 2026.

A linear Python script with requests and a for loop over 500 URLs is not a production system. In real deployments, markup changes, socket timeouts, and bad proxy exits eventually break naive runs.

To move from a side project to production, your pipeline needs fault tolerance, state, and observability.

This guide covers four practical building blocks for running high-volume extraction reliably.

“Free” web scraping usually means free software or a free tier—not free infrastructure. Bandwidth, headless browsers, CAPTCHAs, and IP reputation still cost money somewhere. This guide lists practical free and freemium options, what each is good for, where free breaks, and how to choose before you pay for proxies or platforms.

These three tools are frequently compared but rarely doing the same job. BeautifulSoup is not a crawler — it's an HTML parser. Scrapy is a Python crawling framework. Crawlee is a Node.js (and Python) crawling library with first-class browser support.

Picking the wrong one means building a codebase with the wrong tool for your actual target. This guide makes the differences concrete.