Apify Memory and CPU Optimization: Cut Actor Costs by 50%
Apify bills by Compute Units: Memory (GB) × runtime (hours). Oversizing Actors wastes money; undersizing causes OOM crashes. This guide covers memory sizing, Cheerio vs Playwright trade-offs, Playwright resource blocking, concurrency tuning, and dataset batching so you cut costs by 30–50% without losing throughput. Optimize your Actors in the Apify Console.
How Apify billing works
Compute Units (CUs) = Memory (GB) × Runtime (hours). Example: 1 GB for 2 hours = 2 CUs. Apify rounds up. A 2 GB Actor that runs 15 minutes still bills for 2 GB × 0.25 h = 0.5 CUs.
Free plan: $5/month in credits (~$0.25 per CU). Paid plans: usage-based. Every MB and minute counts.
Choosing the right memory
| Use Case | Recommended Memory | Rationale |
|---|---|---|
| CheerioCrawler | 256 MB | Lightweight, no browser. Rarely needs more. |
| PlaywrightCrawler (simple) | 512 MB – 1 GB | Single page, minimal JS. Works for most SERP/product scrapes. |
| PlaywrightCrawler (heavy) | 1–2 GB | Multiple tabs, complex DOM, PDF generation. |
| Puppeteer with PDF | 2 GB+ | Chromium + PDF rendering is memory-hungry. |
Start low. Check Run → Details for peak memory. If you never hit 50% of allocation, downgrade. See the Actor building guide for scaffolding.
CheerioCrawler vs PlaywrightCrawler: CPU and memory
| Attribute | CheerioCrawler | PlaywrightCrawler | Best For |
|---|---|---|---|
| Memory | ~50–150 MB | 300 MB – 2 GB | Cheerio: static HTML |
| CPU | Low (no JS engine) | High (full Chromium) | Cheerio: bulk scraping |
| Use case | Static HTML, APIs | JS-rendered, SPAs, screenshots | Playwright: dynamic content |
| Concurrency | 20–50 typical | 5–15 typical | Cheerio: higher throughput |
| Cost per 1K pages | ~$0.05–0.10 | ~$0.20–0.50 | Cheerio: 3–5× cheaper |
Winner: Cheerio when HTML is static or pre-fetched. Playwright only when you need rendering, clicks, or screenshots. Use Apify's Website Content Crawler or similar for many sites that load content via API — it often uses Cheerio under the hood.
Disabling images, CSS, and fonts in Playwright
Loading images and styles increases memory and network. Block them when you only need text or structure.
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
launchContext: {
launchOptions: {
args: ['--disable-extensions', '--no-sandbox'],
},
},
preNavigationHooks: [
async ({ page }) => {
await page.route('**/*', (route) => {
const type = route.request().resourceType();
if (['image', 'stylesheet', 'font', 'media'].includes(type)) {
route.abort();
} else {
route.continue();
}
});
},
],
async requestHandler({ page, request }) {
// ... extraction logic
},
});
Saves 30–50% memory on image-heavy pages. Test that your selectors still work — some sites inject content via CSS. You can also disable JavaScript entirely for static pages — add await page.setJavaScriptEnabled(false) in a preNavigationHook — but only if the page doesn't need JS to render content.
Request concurrency tuning
maxConcurrency caps parallel requests. minConcurrency controls Crawlee's AutoscaledPool floor. Higher concurrency = faster but more memory and CPU.
| Scenario | maxConcurrency | minConcurrency | Notes |
|---|---|---|---|
| Cheerio, cheap targets | 20–30 | 5 | Throughput-first |
| Playwright, simple pages | 10–15 | 3 | Balanced |
| Playwright, heavy pages | 5–8 | 2 | Avoid OOM |
| Rate-limited sites | 2–5 | 1 | Reduce bans |
const crawler = new PlaywrightCrawler({
maxConcurrency: 10,
minConcurrency: 3,
// ...
});
Tune based on Run details. If CPU stays below 50%, increase maxConcurrency. If memory spikes, decrease it.
Autoscaling: when to override
Apify's default autoscaler adjusts concurrency based on system load. For predictable workloads, you can disable it and fix concurrency:
const crawler = new PlaywrightCrawler({
autoscaledPoolOptions: {
autoscale: false, // Fixed concurrency
},
maxConcurrency: 8,
// ...
});
Use fixed concurrency when the default autoscaler over-provisions and wastes CUs. Use autoscale when load varies widely.
Dataset and KV store write batching
Frequent pushData calls add I/O overhead. Crawlee batches internally, but avoid push-per-element in hot loops when possible.
// Better: batch in memory, push periodically
const batch: Record<string, unknown>[] = [];
for (const item of items) {
batch.push(item);
if (batch.length >= 100) {
await Actor.pushData(batch);
batch.length = 0;
}
}
if (batch.length > 0) await Actor.pushData(batch);
For very large runs, consider writing to a KV store in chunks and exporting at the end. Reduces dataset API calls.
Monitoring memory in Console
Run → Details shows peak memory and CPU. Compare to your Actor's allocation. If peak is 400 MB and you're on 2 GB, downgrade to 512 MB and re-run. Saves 75% on that dimension.
Complete optimization checklist
- Memory set to minimum viable (256 MB Cheerio, 512 MB–1 GB Playwright)
- Images/CSS/fonts blocked in Playwright where possible
- maxConcurrency tuned from Run details
- Cheerio used instead of Playwright where HTML is static
- Dataset writes batched for high-volume runs
Crawler type comparison
| Crawler | Memory | CPU | Cost/1K pages | Best For |
|---|---|---|---|---|
| CheerioCrawler | 256 MB | Low | $ | Static HTML, APIs |
| PlaywrightCrawler (no images) | 512 MB | Medium | $$ | JS pages, light rendering |
| PlaywrightCrawler (full) | 1–2 GB | High | $$$ | Screenshots, PDFs, complex SPAs |
Best for cost: Cheerio. Best for fidelity: Playwright with resource blocking. See Actor testing guide to validate before scaling.
Default to Cheerio. Switch to Playwright only when pages need rendering. You'll often cut costs by 50% or more.
Apify bills by Compute Units (CUs) = Memory (GB) × Runtime (hours). A 1 GB Actor running 2 hours costs 2 CUs. Free plan gives ~$5/month; paid plans are usage-based.
Cheerio typically needs 256 MB. Playwright needs 512 MB–2 GB depending on page complexity. Disable images and CSS to reduce Playwright memory by 30–50%.
Use Cheerio for static HTML or when the target serves content without JS. Use Playwright when you need to render JavaScript, take screenshots, or interact with the page. Cheerio is 3–5× cheaper.
Block images, stylesheets, fonts, and media via page.route(). Use --disable-extensions and --no-sandbox in launch args. Reduce maxConcurrency if memory spikes.
Open Run details in Apify Console. The metrics section shows peak memory and CPU. Compare to your Actor's allocation — if usage is low, downsize to save cost.
Batching reduces I/O overhead and can slightly lower runtime. The main savings come from memory sizing and crawler choice. Still, batch when pushing thousands of items.