Crawlee Node.js Tutorial: Production Web Scraping Without the Boilerplate (2026)

Contains affiliate links. Disclosure.

Crawlee is an open-source Node.js framework from Apify that bundles everything a production scraper needs: request deduplication, auto-retry, proxy rotation, session management, persistent storage, and Playwright/Puppeteer/HTTP crawlers under one API.

Where raw Playwright requires wiring all those pieces manually, Crawlee provides them out of the box — letting you focus on extraction logic.

Freshness note: Examples verified against Crawlee 3.x (March 2026). Install crawlee@latest to get the current release.

Install Crawlee

npm init -y
npm install crawlee
# For PlaywrightCrawler
npx playwright install chromium

---

CheerioCrawler (Fast HTML Scraping)

Use this for static HTML sites. Crawlee fetches pages with HTTP and parses with Cheerio — no browser needed.

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  async requestHandler({ $, request, enqueueLinks }) {
    const title = $('h1').text();
    const price = $('.price').text();

    await crawler.pushData({ url: request.url, title, price });

    // Follow pagination links automatically
    await enqueueLinks({ selector: 'a.next-page' });
  },
  maxRequestsPerCrawl: 100,
});

await crawler.run(['https://books.toscrape.com/']);

Crawlee automatically stores results in ./storage/datasets/default/. You get one JSON file per item.

---

PlaywrightCrawler (JavaScript-Rendered Pages)

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  async requestHandler({ page, request, enqueueLinks }) {
    await page.waitForSelector('.product-grid');

    const products = await page.$$eval('.product-card', (cards) =>
      cards.map((c) => ({
        name: c.querySelector('.name').innerText,
        price: c.querySelector('.price').innerText,
      }))
    );

    await crawler.pushData(products);

    await enqueueLinks({ selector: 'a.pagination-next' });
  },
  headless: true,
  maxConcurrency: 5,
});

await crawler.run(['https://example-spa.com/shop']);

---

Proxy Rotation

Pass proxies through ProxyConfiguration:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [
    'http://user:pass@gate.iproyal.com:7777',
    'http://user:pass@proxy2.example.com:8888',
  ],
});

const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  async requestHandler({ page }) {
    // Each request uses a different proxy from the pool
    const data = await page.$eval('.result', (el) => el.innerText);
    await crawler.pushData({ data });
  },
});

For residential rotating proxies with geo-targeting, IPRoyal is a cost-effective option (prices vary by volume; non-expiring bandwidth) and integrates directly with Crawlee's ProxyConfiguration.

---

Session Management

Sessions let you persist cookies across requests to stay logged in or avoid per-session detection:

import { PlaywrightCrawler, SessionPool } from 'crawlee';

const crawler = new PlaywrightCrawler({
  useSessionPool: true,
  persistCookiesPerSession: true,

  async requestHandler({ page, session }) {
    // Login once per session
    if (!session.userData.loggedIn) {
      await page.goto('https://example.com/login');
      await page.fill('#email', 'user@example.com');
      await page.fill('#password', 'secret');
      await page.click('[type=submit]');
      session.userData.loggedIn = true;
    }

    await page.goto('https://example.com/protected-data');
    const data = await page.$eval('.data', (el) => el.innerText);
    await crawler.pushData({ data });
  },
});

---

Request Queue and State

Crawlee's RequestQueue persists to disk, so scrapes survive crashes and can be resumed:

import { CheerioCrawler, RequestQueue } from 'crawlee';

const queue = await RequestQueue.open();
await queue.addRequest({ url: 'https://news.ycombinator.com/' });

const crawler = new CheerioCrawler({
  requestQueue: queue,
  async requestHandler({ $, enqueueLinks }) {
    const stories = [];
    $('.athing').each((_, el) => {
      stories.push({ title: $(el).find('.titleline a').text() });
    });
    await crawler.pushData(stories);

    // Only follow story comment pages, not external links
    await enqueueLinks({
      selector: '.subtext a[href^="item"]',
      baseUrl: 'https://news.ycombinator.com',
    });
  },
  maxRequestsPerCrawl: 50,
});

await crawler.run();

---

Export Data

Crawlee stores data in ./storage/datasets/default/. To export to JSON:

# Data is already JSON — just copy the directory or use the Apify CLI
npx apify push   # Deploys and lets you download via Dataset API

Or read programmatically:

import { Dataset } from 'crawlee';
const dataset = await Dataset.open();
const { items } = await dataset.getData();
console.log(items);

---

Deploy to Apify

Crawlee and Apify share the same Actor runtime. Deploying is three commands:

npm install -g apify-cli
apify login
apify push

Your scraper runs in the cloud with:

• Automatic proxy rotation (Apify Proxy)
• Persistent RequestQueue and Dataset storage
• Scheduled runs via Apify Scheduler
• Webhook triggers and API access

Apify free tier includes $5/month in platform credits — enough to run several full crawls per month.

Pricing verified June 2026. Plans change frequently — check the vendor’s official pricing page for current rates.

---

CheerioCrawler vs PlaywrightCrawler: When to Use Which

Scenario	Use
Static HTML	CheerioCrawler (10× faster, minimal cost)
JavaScript SPA	PlaywrightCrawler
API-backed page	Direct fetch + JSON parse
Logged-in scraping	PlaywrightCrawler + useSessionPool
Scale > 100 concurrent	Apify + CheerioCrawler
Cloudflare-protected site	PlaywrightCrawler + residential proxy

---

FAQ

Frequently Asked Questions

Crawlee is an open-source web scraping framework built on top of Puppeteer and Playwright. Puppeteer (and Playwright) are browser automation libraries — they give you control over a browser. Crawlee adds the scraping-specific layer on top: request queues, automatic retries, session management, proxy rotation, dataset storage, and CheerioCrawler (which uses Cheerio for fast HTML parsing without launching a browser at all).

The simplest approach is to deploy your Crawlee scraper as an Apify Actor and use Apify's built-in scheduler (cron syntax). Alternatively, deploy to any VPS (Liquid Web, Hetzner, DigitalOcean) and use system cron or a process manager like PM2. Crawlee itself has no built-in scheduler — scheduling is handled at the infrastructure layer.

Yes. Crawlee has both JavaScript (Node.js) and Python editions. The Python Crawlee library (pip install crawlee) provides BeautifulSoupCrawler, PlaywrightCrawler, and HttpCrawler. The Node.js edition is more mature and has more features. For new projects, Node.js Crawlee is recommended unless your team is primarily Python.

Yes. Crawlee is 100% free and open-source (MIT license). There are no usage fees. The only cost is your infrastructure (VPS, proxy bandwidth). You can optionally deploy to Apify cloud, which has a free tier with $5/month in platform credits.