Skip to main content
use-apify.com

Infrastructure: guides & tutorials

Scraping infrastructure: queues, storage, proxies, and metrics. Build Apify-friendly pipelines that split crawl, parse, and export for cleaner scaling.

3 articles

View all tags

Scraping infrastructure ties together queues, storage, proxies, and metrics into a pipeline that scales. These guides cover building Apify-friendly setups that split crawl, parse, and export.

Separating concerns makes each layer easier to scale and observe as volume grows. Apify provides much of this infrastructure out of the box. Below you will find patterns for designing scraping infrastructure.

Related topics

Self-hosted AI13 min read

The $20/Month AI Operations Stack: Self-Host Ollama + n8n + Coolify on a VPS

· 13 min read
Yassine El Haddad
Yassine El Haddad
Software Developer & Automation Specialist

Cloud API costs scale linearly with usage. Self-hosting Ollama + n8n + Coolify on a $20/month VPS removes per-token fees for local Ollama inference — you still pay for the VPS, bandwidth, and your time; throughput is limited by CPU/RAM. When workflows call Apify, email, or other SaaS APIs, data transits those providers — not only your server.

This guide covers the complete setup from bare VPS to production-ready AI pipeline, with official documentation links for every step.

TL;DR:

ComponentRoleOfficial docs
CoolifyInfrastructure manager (replaces Heroku/Vercel for self-hosting)coolify.io/docs
OllamaLocal LLM inference (Llama 3.1, Mistral, DeepSeek)ollama.com
n8nWorkflow orchestration (visual automation)docs.n8n.io
QdrantVector database for AI memory/RAG (Retrieval-Augmented Generation)qdrant.tech/documentation
PostgreSQLStructured data storagepostgresql.org

Prerequisites:

  • A VPS with 4+ CPU cores, 8+ GB RAM, 80+ GB SSD ($16–$25/month range). Minimum 8 GB RAM required for running Llama 3.1 8B comfortably alongside n8n.
  • A domain name pointed to the VPS IP
  • Basic terminal/SSH knowledge
  • 30–60 minutes of setup time
Guide7 min read

Self-Hosting Web Scrapers: Complete Infrastructure Guide for 2026

· 7 min read
Yassine El Haddad
Yassine El Haddad
Software Developer & Automation Specialist

Self-hosting web scrapers gives you full control: custom stack, sensitive data stays on your infra, and at high volume you can optimize cost. This guide covers when self-hosting makes sense vs using Apify, infrastructure components (VPS, proxies, queue, storage, monitoring), Docker Compose setup with Crawlee, scaling strategies, and a cost comparison. Run on a Liquid Web VPS for reliable hosting with SSD storage and managed support.

Guides on this site

Frequently asked questions

Frequently Asked Questions

Request queues (Redis or SQS), compute workers (Apify Cloud or Docker), proxy rotation (Apify Proxy or Bright Data), object storage (S3/GCS), and a relational or time series database for results. Monitoring, alerting, and secret management complete the stack. Apify Cloud bundles most of this; self-hosted setups assemble each component separately.

Use redundant proxy pools so a single provider outage does not stop production. Checkpoint crawls so failed runs can resume. Alert on data freshness breaches. Run scraping actors in multiple regions for geo-diversity. Treat external web sources as unreliable external dependencies with circuit breakers and fallback strategies.

When a single worker cannot complete the crawl within your freshness SLA. Horizontal scaling means more concurrent workers with the same code, not more threads per worker. Add workers incrementally and monitor block rates—scaling too fast can trigger site-wide IP blocks that scale down the entire operation.

Use HTTP crawlers instead of browsers where JavaScript is not needed—often 10x cheaper. Schedule heavy crawls during off-peak hours for potential spot pricing discounts. Deduplicate URLs before adding to the queue to reduce wasted compute. Apify run cost analysis tools help identify high-cost actors worth optimizing.