Self-Hosting Web Scrapers: Complete Infrastructure Guide for 2026

Contains affiliate links. Disclosure.

Self-hosting web scrapers gives you full control: custom stack, sensitive data stays on your infra, and at high volume you can optimize cost. This guide covers when self-hosting makes sense vs using Apify, infrastructure components (VPS, proxies, queue, storage, monitoring), Docker Compose setup with Crawlee, scaling strategies, and a cost comparison. Run on a Liquid Web VPS for reliable hosting with SSD storage and managed support.

When to Self-Host vs Use Apify

Factor	Self-Host	Apify
Data volume	High (millions of pages) — cost optimizes	Lower–medium — pay-per-run scales simply
Sensitive data	Keep on your infra	Evaluate data residency; Apify is cloud
Custom stack	Full control (Crawlee, Scrapy, etc.)	Constrained to Actor model
Maintenance	You own infra, monitoring, updates	Apify handles runtime and scaling
Time to production	Longer setup	Faster: pick Actor, run

Self-host when: You have high, predictable volume; need data sovereignty; or have a mature DevOps team. Use Apify when: You want speed, no infra management, or access to the Actor marketplace.

Infrastructure Components

Component	Role	Options
VPS / dedicated server	Compute for crawlers	Liquid Web, DigitalOcean, Hetzner
Proxy pool	Anti-bot bypass, IP rotation	Bright Data, IPRoyal, Oxylabs
Request queue	Persist URLs, retries	Redis, BullMQ, Crawlee RequestQueue (in-memory or Redis)
Storage	Store extracted data	PostgreSQL, MongoDB, S3, MinIO
Monitoring	Metrics, alerts	Prometheus, Grafana, PagerDuty, Slack

VPS Selection for Playwright Scrapers

Playwright and Crawlee with browser automation are memory-heavy. Minimum specs:

Spec	Minimum	Recommended
RAM	8 GB	16 GB for 2–3 concurrent browsers
vCPU	4	8 for higher concurrency
Storage	50 GB SSD	100+ GB for datasets
OS	Ubuntu 22.04 / 24.04	LTS with security updates

Liquid Web VPS offers 8 GB RAM plans with SSD. See Liquid Web VPS Scraping Setup for step-by-step provisioning.

Docker Compose Stack: Redis + Crawlee + PostgreSQL

# docker-compose.yml
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    volumes: [redis_data:/data]

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: scraped
      POSTGRES_USER: scraper
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes: [pg_data:/var/lib/postgresql/data]

  workers:
    build: ./workers
    environment:
      REDIS_URL: redis://redis:6379
      DATABASE_URL: postgresql://scraper:${DB_PASSWORD}@postgres:5432/scraped
      PROXY_URL: ${BRIGHT_DATA_PROXY}
    depends_on: [redis, postgres]
    deploy:
      replicas: 2
    restart: unless-stopped

volumes:
  redis_data: {}
  pg_data: {}

Workers run Crawlee with Playwright. Redis backs the RequestQueue for persistence. PostgreSQL stores extracted items. See Docker Compose Production Guide for secrets, health checks, and limits.

Proxy Integration

For anti-bot targets, use residential proxies. Bright Data and IPRoyal provide HTTP/SOCKS5 gateways. Configure Crawlee:

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [process.env.PROXY_URL],
});
const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  // ...
});

Or use Apify Proxy as a custom proxy in your self-hosted Crawlee—gives you Apify's proxy pool without running on Apify platform.

Monitoring and Alerting

Metric	Tool	Alert when
Crawl rate	Prometheus + Grafana	Throughput drops > 50%
Error rate	Crawler logs, Prometheus	5xx or block rate > 10%
Queue depth	Redis INFO	Queue grows unbounded
Worker health	Docker / systemd	Worker exits or OOM

Export Crawlee metrics (requests/min, failed count) to Prometheus. Alert to PagerDuty or Slack when thresholds are breached.

Scaling Strategy

Approach	When	How
Vertical	Single bottleneck	Upgrade VPS (more RAM, CPU)
Horizontal	Need more throughput	Add worker replicas; share Redis queue
Sharding	Very large queues	Partition URLs by domain or hash; multiple queues

Start with 1–2 workers. Scale horizontally when queue depth grows and CPU is underutilized.

Cost Comparison: Self-Hosted vs Apify

Rough estimates per million pages crawled (with JS rendering):

Cost	Self-Hosted (8 GB VPS + proxies)	Apify (pay-per-run)
Compute	~$30–50/mo (VPS)	Included in platform
Proxies	$50–150/mo (residential)	Apify Proxy or custom
Total	~$80–200/mo fixed	~$0.25–1.00 per 1K pages → $250–1000/mo at 1M

At low volume (< 100K pages/mo), Apify is often cheaper. At high volume (> 1M/mo), self-hosting can save 50–70% if you optimize.

Crawlee Worker Example

A minimal Crawlee worker for the Docker stack:

// workers/src/main.ts
import { PlaywrightCrawler } from 'crawlee';
import { Client } from 'pg';

const crawler = new PlaywrightCrawler({
  maxConcurrency: 2,
  requestHandler: async ({ page, request }) => {
    const title = await page.title();
    const price = await page.$eval('.price', el => el.textContent).catch(() => null);
    const pg = new Client(process.env.DATABASE_URL);
    await pg.connect();
    await pg.query(
      'INSERT INTO products (url, title, price) VALUES ($1, $2, $3)',
      [request.url, title, price]
    );
    await pg.end();
  },
});

const urls = process.env.START_URLS?.split(',') || [];
await crawler.run(urls);

Add BullMQ or similar to pull URLs from Redis. Run with docker compose up workers after building the image.

Cron and Scheduling

For recurring runs, use cron or a task scheduler: 0 2 * * * cd /app/workers && node dist/main.js runs daily at 2 AM. Or use a job queue (BullMQ, Celery) to enqueue crawl jobs on a schedule. For hybrid control, trigger self-hosted workers via webhooks from Apify Schedules.

Data Pipeline and Backup

Extract data flows: Crawlee → PostgreSQL or Redis → downstream ETL. Use connection pooling and batch inserts. For disaster recovery, run daily pg_dump, Redis RDB snapshots, and replicate extracted data to S3. See Docker Compose Production Guide for container redundancy.

Security Considerations

Self-hosted scrapers handle outbound traffic and stored data. Harden the stack: (1) run workers as non-root; (2) store proxy and DB credentials in secrets (Docker secrets, env files not in git); (3) use TLS for DB connections; (4) restrict firewall to required ports only; (5) keep OS and dependencies updated. See VPS Security Hardening Checklist for server-level hardening.

Worker Configuration Best Practices

• maxConcurrency: Start at 2–3 for Playwright; increase only if CPU/RAM allow.
• navigationTimeoutSecs: 30–60 for slow targets; avoid infinite waits.
• maxRequestsPerCrawl: Cap runs to control cost; enqueue more via scheduler.
• requestHandler: Keep logic focused; offload heavy processing to a queue.

Failure Modes and Mitigation

Failure	Mitigation
Worker crash	Health checks, auto-restart, redundant replicas
IP ban	Residential proxies, session rotation, rate limiting
Target site redesign	Flexible selectors, smoke tests, AI extraction fallback
Storage full	Retention policies, S3 lifecycle, archive old data
Redis / DB down	Replication, backups, failover

Apply patterns from Apify Error Handling: retries, session pools, failed-request handling.

Start small

Begin with a single VPS, one worker, and a small proxy budget. Validate throughput and cost. Scale when you hit limits. Liquid Web VPS · Bright Data proxies

Frequently Asked Questions

Minimum 8 GB RAM, 4 vCPU. Each headless browser uses ~300–500 MB. 16 GB allows 2–3 concurrent browsers comfortably. SSD storage recommended.

Yes. Configure Apify Proxy as a custom proxy URL in Crawlee. You pay for Apify Proxy usage while running workers on your own infra.

Use residential proxies (Bright Data, IPRoyal). Rotate sessions. Add delays between requests. Respect robots.txt and rate limits.

Often yes at > 1M pages/month. Fixed VPS + proxy costs can undercut pay-per-run. Below ~100K pages/month, Apify is usually simpler and competitive.

Track crawl rate, error rate, queue depth, worker health. Use Prometheus + Grafana. Alert to Slack/PagerDuty when throughput drops or errors spike.

Use Redis or another external store. Crawlee supports RequestQueue with Redis. Configure storage in crawler options so queue survives worker restarts.