Apify Architecture: Scaling Serverless Execution Containers (2026)
A single Playwright script on your laptop is easy to reason about. Crawling millions of URLs with hundreds of workers, rotating proxies, and durable storage is a different job: you need orchestration, quotas, and clear failure behavior.
Apify is a platform for running containerized scrapers (“Actors”) in the cloud, with built-in storage, proxies, and scheduling. This post walks through the main pieces and a few limits to plan around.
1. Serverless execution (Actors)
An Actor is a containerized app (Node, Python, or other supported stacks) that runs on Apify’s infrastructure.
Typical flow:
- You write extraction logic (often with Crawlee).
- You build from a base image such as
apify/actor-node-playwright, which bundles Chromium and tooling. - Input is validated JSON.
- Apify starts the container with a chosen memory size (Compute Units / CU) and tears it down when the run ends.
OOM kills: Actors are ephemeral. A memory leak can push a worker past its limit (for example 4096 MB) and the process dies. Use SDK hooks such as Actor.on_abort to persist partial state to a Key-Value Store before shutdown when you need graceful recovery.
2. Storage: datasets, key-value stores, request queues
Heavy crawls rarely belong in a single RDBMS row stream. Apify splits storage by use case:
Datasets (append-only results)
Tabular or document-shaped output. You append rows (await Actor.pushData(row)). They are append-oriented, not a random-access SQL table—good for crawl output, not for transactional edits mid-run.
Key-value stores (KVS)
Arbitrary blobs: PDFs, screenshots, checkpoints. Long runs can write cursor or token state here so a restarted worker can resume without redoing everything.
Request queues
Distributed URL queues with deduplication and retries—Crawlee’s default backbone for parallel crawling across many workers.
3. Proxies
Many targets block datacenter IPs. Apify exposes proxy configuration through a gateway instead of hand-wiring lists in code.
- Datacenter: Often enough for internal or low-friction APIs.
- Residential: Exits through consumer ISP ranges when you need that signal.
Concurrency vs. subnets: Even with residential pools, blasting one domain at extreme concurrency without session discipline can reuse the same subnets and burn reputation. Tune concurrency and session pools to match the site’s tolerance.
4. Schedules, webhooks, and Tasks
- Schedules: Cron-style triggers in the console replace many home-grown cron + VM setups.
- Webhooks: Notify external systems when a run finishes (warehouse ingest, Zapier, Lambda, etc.).
- Tasks: Saved input presets for the same Actor—useful when one scraper serves multiple brands or regions without duplicating code.
When Apify is a good fit
If you are past “a weekly cron on one URL list,” running your own fleet of VMs, proxy billing, and retry logic gets old fast. Apify concentrates state, proxies, and scaling so you spend more time on selectors and data quality—and less on plumbing.
Not really as a generic low-latency proxy. Cold-starting a Playwright Actor takes seconds. For sub-500 ms single-shot HTTP you usually want a dedicated proxy product or a long-lived warm worker (which costs more to keep running).
No—they are optimized for high-throughput append from parallel workers, not multi-row transactions. For strict relational invariants, land data in PostgreSQL (or similar) in a downstream step.
Yes. `apify push` builds a Docker image from your project; private Actors stay private. Apify does not publish your source.