Production scrapers: retries, structured logs, alerts, and data validation. Harden Apify Actors and pipelines so failures are visible and recoverable.
Production scrapers need retries, structured logs, alerts, and data validation so failures are visible and recoverable. These guides cover hardening Apify Actors and pipelines for real workloads.
The difference between a demo and a production scraper is how it handles failure and drift. Apify provides logging, monitoring, and retries to lean on. Below you will find patterns for production-grade scraping.
Development docker-compose.yml works for local runs — production needs secrets management, health checks, resource limits, named volumes, and zero-downtime deploys. This guide shows what sets production Docker Compose apart and provides an annotated template you can adapt. Run it on a Liquid Web VPS for a stable, performant host with SSD storage and managed monitoring.
Web scraping architectures evolve from a single script to queue-based, distributed, and managed platforms. Each level trades simplicity for scalability, reliability, and maintainability. This guide describes four architecture levels, when to move between them, data pipeline integration, anti-detection at the architecture level, observability, and a Level 2 code example with Crawlee, Redis, and PostgreSQL. For managed infrastructure that handles queues and scaling for you, Apify provides a Level 4 option out of the box.
Deploy Meta's LLaMA 3 as a high-throughput, OpenAI-compatible inference API on a dedicated GPU server — with quantization, load testing, and Prometheus monitoring included.
If you've run LLaMA 3 locally with Ollama or llama.cpp, you've hit the ceiling: single-user latency is fine, but concurrency crumbles past three or four simultaneous requests. vLLM solves that. It uses PagedAttention and continuous batching to serve dozens of concurrent requests on the same GPU that would otherwise stall at one.
This guide walks through deploying LLaMA 3 (8B and 70B) on a dedicated GPU server using vLLM — covering Docker setup, model quantization, the OpenAI-compatible API, load testing with Locust, and Prometheus monitoring. All steps are tested on an NVIDIA A100 80 GB, which you can rent from Liquid Web.
Error handling and retries for transient failures, structured logging for observability, schema validation on outputs, monitoring with alerts on failure rates and data freshness, version-controlled deployment, documentation of inputs and outputs, and a runbook for common failure scenarios. Production scrapers are maintained software, not one-off scripts.
Track run success rate, records extracted per run, null rates per field, cost per run, and data freshness against SLAs. Use Apify webhooks to push metrics to your monitoring stack. Alert on anomalies—sudden volume drops indicate site changes, cost spikes indicate runaway concurrency, freshness breaches require immediate investigation.
Implement retry logic with exponential backoff for transient errors. Use dead-letter queues for failed URLs. Save checkpoints so partial runs can resume without re-scraping. Alert on-call when success rates drop below threshold. Keep a snapshot of last known-good HTML for each target to compare against when diagnosing structure changes.
Unit test parsing functions against fixture HTML files. Integration test full actor runs against a small curated URL set. Schema validate outputs in CI. Run canary tests against production targets daily and alert on failures. Regression tests that run against archived HTML snapshots catch parser bugs without hitting live sites repeatedly.
