Self-Host Crawlee on Liquid Web: Production Deployment

Contains affiliate links. Disclosure.

Crawlee is Apify’s open-source crawling library. It handles queues, retries, anti-bot fingerprinting, and storage. This guide covers deploying Crawlee on a Liquid Web VPS for production scraping.

For managed Crawlee without servers, run Actors on Apify. Self-hosting is for teams that want full control and lower per-run cost at scale.

Prerequisites

• Liquid Web VPS with 4 GB+ RAM (8 GB for heavier crawls)
• Ubuntu 22.04 or 24.04
• Node.js 18+

See Liquid Web VPS setup for provisioning.

Step 1: Install Crawlee and Playwright

mkdir -p ~/crawler && cd ~/crawler
npm init -y
npm install crawlee playwright
npx playwright install chromium
npx playwright install-deps chromium

Or use the Crawlee CLI:

npx crawlee create my-crawler
cd my-crawler
npm install

Step 2: Minimal PlaywrightCrawler

// crawler.js
const { PlaywrightCrawler } = require('crawlee');

const crawler = new PlaywrightCrawler({
  maxRequestsPerCrawl: 10,
  maxConcurrency: 2,
  requestHandler: async ({ page, request }) => {
    const title = await page.title();
    console.log(`${request.url} -> ${title}`);
  }
});

await crawler.run(['https://example.com']);

Run:

node crawler.js

Step 3: Storage Configuration

Crawlee stores state in ./storage by default. For production:

export CRAWLEE_STORAGE_DIR=/var/lib/crawlee
mkdir -p /var/lib/crawlee
chown $USER:$USER /var/lib/crawlee

Or use a custom storage backend (e.g. S3) via Crawlee’s storage API.

Step 4: Deployment Options

Option A: PM2

npm install -g pm2
pm2 start crawler.js --name crawler
pm2 startup
pm2 save

Option B: Cron

0 2 * * * cd /home/user/crawler && node crawler.js >> /var/log/crawler.log 2>&1

Option C: Docker

Use the same Playwright Dockerfile as in Docker web scraping. Add Crawlee to package.json.

Step 5: Resource Limits

Crawl type	RAM	Concurrent pages
CheerioCrawler	1–2 GB	10–20
PlaywrightCrawler	4 GB	2–4
PlaywrightCrawler (heavy)	8 GB	4–8

Set maxConcurrency to avoid OOM:

const crawler = new PlaywrightCrawler({
  maxConcurrency: 2,
  // ...
});

When to Use Apify Cloud Instead

• Anti-bot targets — Apify manages proxies and Fire-engine
• Scheduling — Built-in cron and triggers
• Scaling — No server capacity planning
• Low volume — Free tier covers many use cases

Apify runs Crawlee-based Actors in the cloud. Self-host when you have steady volume and want to minimize cost per run.

Next step

Start with CheerioCrawler for static pages (faster, lighter). Use PlaywrightCrawler only when you need JavaScript rendering.

Frequently Asked Questions

Yes. Crawlee is open-source. Install with npm, add Playwright or Puppeteer, run on a VPS. Apify Cloud is the managed alternative.

4 GB RAM for PlaywrightCrawler. 8 GB for heavy crawls. CheerioCrawler runs on 1–2 GB.

Use PM2 or systemd for process management. Set CRAWLEE_STORAGE_DIR. Limit maxConcurrency. Monitor memory and logs.