Web Scraping Architecture Patterns: From Prototype to Production (2026)
Web scraping architectures evolve from a single script to queue-based, distributed, and managed platforms. Each level trades simplicity for scalability, reliability, and maintainability. This guide describes four architecture levels, when to move between them, data pipeline integration, anti-detection at the architecture level, observability, and a Level 2 code example with Crawlee, Redis, and PostgreSQL. For managed infrastructure that handles queues and scaling for you, Apify provides a Level 4 option out of the box.
Level 1: Single-Script (Prototype)
The simplest architecture: one script, direct HTTP requests, in-memory processing, output to CSV or JSON.
Characteristics
- Execution: Single process, sequential or light concurrency
- State: In-memory; crash loses everything
- Output: File (CSV, JSON) or stdout
- Retry: Ad hoc or none
- Proxies: Optional, often hardcoded or env vars
When to Use
Proof of concept, one-off extraction, learning. Targets with no anti-bot, low volume, no SLA.
Limitations
No durability, no horizontal scale, no shared state. Breaks on targets with rate limits or anti-bot. Not suitable for production.
Level 2: Queue-Based (Production)
Add a request queue (Redis, BullMQ, SQS), retry logic, structured output to a database, and error reporting.
Characteristics
- Queue: Redis + BullMQ (or similar) for request jobs
- Retry: Exponential backoff, max retries per URL
- Output: PostgreSQL, MySQL, or cloud DB
- Error handling: Dead-letter queue, alerts on failure rate
- Proxies: Rotation via proxy pool service
When to Use
Recurring scrapes, moderate volume (thousands to hundreds of thousands of URLs), need for reliability. Team has DevOps capacity to run Redis and DB.
Data Flow
[URL list] → [Request Queue] → [Worker] → [Parse] → [DB]
↑ ↓
└──── [Retry on failure] ←──────────────┘
See Apify error handling for retry and fallback patterns. Crawlee (used by Apify) has built-in request queue semantics; the pattern applies to custom deployments.
Level 3: Distributed
Multiple worker nodes, shared queue, centralized proxy pool, monitoring.
Characteristics
- Workers: N processes/machines consuming from shared queue
- Queue: Centralized (Redis Cluster, SQS)
- Proxy pool: Dedicated proxy service (e.g., Bright Data, IPRoyal) with rotation
- Coordination: Queue is the coordination mechanism; no shared memory
- Monitoring: Distributed tracing, queue depth, success rate
When to Use
High volume (millions of URLs), need for throughput and redundancy. Team can operate multi-node infrastructure.
Trade-offs
More moving parts: queue, workers, proxy service, DB. Requires monitoring and operational runbooks. See Apify proxy configuration guide for proxy integration patterns.
Level 4: Managed Platform
Fully managed: Apify, Bright Data, or similar. You define Actor logic or configuration; the platform handles queues, workers, proxies, storage, and scaling.
Characteristics
- Abstraction: Focus on extraction logic; platform handles infrastructure
- Scaling: Automatic; pay per compute/bandwidth
- Proxies: Built-in or add-on; configured per run
- Storage: Datasets, key-value stores; export to S3, webhooks
When to Use
Teams that want to focus on data, not infra. Variable or unpredictable volume. Need for pre-built Actors (Apify Store) or datasets (Bright Data). Apify is the primary example; Bright Data offers Scraping Browser and datasets.
Decision Matrix: When to Move Between Levels
| Factor | Level 1 | Level 2 | Level 3 | Level 4 |
|---|---|---|---|---|
| Scale | under 1K URLs | 1K–500K | 500K+ | Any |
| Budget | Minimal | Moderate (Redis, DB) | Higher (workers, proxies) | Pay-per-use |
| Team | 1 dev | 1–2 devs | 2+ devs + DevOps | Minimal |
| SLA | None | Moderate | High | Contractual |
| Anti-bot | Light/none | Moderate | Heavy | Managed |
Move to Level 2 when you need reliability and recurring runs. Move to Level 3 when single-node throughput is insufficient. Move to Level 4 when infra burden outweighs build cost. Many teams skip Level 3 and go straight to Level 4 (Apify).
Data Pipeline Integration
Scrapers feed downstream systems. Typical flow:
[Scraper] → [Staging DB] → [ETL/Transform] → [Analytics/Warehouse]
- Staging: Raw scraped output, schema versioning, append or upsert by key
- ETL: Deduplication, validation, enrichment, normalization
- Analytics: BI tools, ML pipelines, reporting
Design the scraper output schema for the pipeline. Use IDs (URL, product ID) for idempotent writes. Log run metadata (start, end, record count) for lineage. See building an Apify Actor with TypeScript for schema and storage patterns.
Anti-Detection at the Architecture Level
Architecture choices affect detectability:
- Request timing: Randomize delays between requests; avoid burst patterns. Level 2+ allows per-request delay configuration.
- Proxy rotation: Centralized pool with rotation (per request or per session). Level 3 and 4 provide this; Level 1 often does not.
- User-Agent management: Rotate UAs to match proxy geo; ensure TLS fingerprint correlates. Architecture should support config-driven UA lists.
- Concurrency: Too high concurrency from few IPs triggers rate limits. Distribute load across proxy pool; limit concurrency per proxy.
Level 2+ architectures make these configurations explicit. Managed platforms (Level 4) bundle proxy rotation and fingerprint management.
Observability: Tracing, Alerts, Freshness
- Distributed tracing: Trace IDs across queue, worker, and DB. Correlate slow requests with specific URLs or proxies.
- Alerting on crawl coverage: Alert when success rate drops below threshold (e.g., 90%) or error rate spikes.
- Dataset freshness: Track last successful run per dataset. Alert when freshness exceeds SLA (e.g., "daily" dataset not updated in 28 hours).
Instrument workers with metrics (requests/sec, success rate, queue depth). Export to Prometheus, DataDog, or similar. Apify error handling covers retry and alerting patterns.
Code Example: Level 2 with Crawlee + Redis + PostgreSQL
Crawlee provides a request queue, automatic retries, and proxy support. Below is a simplified structure for a Level 2 deployment:
import { CheerioCrawler, Configuration } from 'crawlee';
import { Pool } from 'pg';
const config = Configuration.getGlobalConfig();
config.set('requestHandlerTimeoutSecs', 60);
const crawler = new CheerioCrawler({
requestHandler: async ({ request, $ }) => {
const items = [];
$('.product').each((_, el) => {
items.push({
url: request.url,
title: $(el).find('.title').text().trim(),
price: $(el).find('.price').text().trim(),
});
});
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
for (const item of items) {
await pool.query(
`INSERT INTO products (url, title, price) VALUES ($1, $2, $3)
ON CONFLICT (url) DO UPDATE SET title = $2, price = $3`,
[item.url, item.title, item.price]
);
}
},
maxRequestsPerCrawl: 10000,
maxConcurrency: 5,
requestHandlerTimeoutSecs: 60,
failedRequestHandler: async ({ request }) => {
console.error(`Failed: ${request.url}`);
// Alert, dead-letter, etc.
},
});
await crawler.run(['https://example.com/products']);
For production: add Redis storage for the request queue (Crawlee supports it), configure proxy rotation, add structured logging and metrics. Deploy as an Apify Actor for managed execution, or run on your own infra with Crawlee's Redis storage.
Migration Path: Level 1 → 2 → 4
A common path: start with a Level 1 script to validate the target and schema. When you need reliability, add a queue (Redis + BullMQ or Crawlee's built-in queue) and DB output—Level 2. When infra burden grows, migrate to Apify: wrap your Crawlee logic in an Actor, deploy to Apify, and use their storage and scheduling. Your extraction logic carries over; only the execution environment changes. See building an Apify Actor with TypeScript for the Actor structure. Apify memory and CPU optimization helps tune resource usage once deployed.
Prototype? Single script. Recurring, reliable? Queue-based (Level 2) or Apify. High volume? Distributed or Apify. Want zero infra? Apify. Try Apify →
When you need recurring runs, retries, and durable output. Level 1 loses state on crash; Level 2's queue and DB persist. Move when reliability matters.
Crawlee bundles queue, retries, and proxy support. Use it when building custom Level 2/3. For fully managed, use Apify—Crawlee powers Apify Actors internally.
Add tracing (trace IDs across queue/worker/DB), metrics (requests/sec, success rate, queue depth), and freshness checks (last successful run). Alert on coverage drops and freshness SLA breaches.
Level 3: you run workers, queue, proxies. Level 4: Apify runs everything; you provide Actor logic. Use Level 4 when infra burden outweighs cost of managed platform. Apify scales automatically.
Architecture must support: configurable request timing, proxy rotation, UA rotation, and concurrency limits per IP. Level 2+ allows this; Level 4 platforms bundle it.