Apify Error Handling: Build Resilient Scrapers That Never Crash

Contains affiliate links. Disclosure.

Scrapers fail constantly: HTTP 429s, navigation timeouts, elements that never appear, Cloudflare blocks. Without proper error handling, a single bad request can kill your entire run or silently drop data. This guide covers every tool Apify and Crawlee give you to build scrapers that survive real-world failures — retries, dead-letter queues, session pools, and webhook alerting. Start building resilient Actors on the Apify platform.

Why scrapers fail (and why it matters)

Most scraper failures fall into five buckets:

Failure Type	Cause	Typical Fix
HTTP errors	403, 404, 429, 503 from target sites	Retries with backoff, proxy rotation
Navigation timeouts	Page loads slowly or hangs	Increase navigationTimeoutSecs, skip problematic URLs
Element not found	DOM structure changed or page blocked	try/catch in handler, fallback selectors
Blocked requests	IP banned, CAPTCHA, bot detection	Session pool, residential proxies
Storage failures	Dataset/KV write limit or outage	Batch writes, retry pushData

By default, Crawlee retries failed requests up to 3 times before giving up. That helps with transient blips but not with patterns like "every 10th request gets 429." You need explicit configuration.

maxRequestRetries and exponential backoff

maxRequestRetries controls how many times Crawlee retries a request before calling failedRequestHandler. The default is 3. For flaky targets, 5–7 is safer.

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  maxRequestRetries: 5,
  requestHandlerTimeoutSecs: 60,
  async requestHandler({ page, request }) {
    const title = await page.title();
    await Actor.pushData({ url: request.url, title });
  },
});

Crawlee uses exponential backoff between retries. You can't configure the backoff curve directly, but increasing maxRequestRetries gives the target time to recover from temporary overload.

try/catch inside requestHandler

Errors inside your handler (e.g., selector not found) are not retried. Wrap logic in try/catch and either push partial data or re-throw to trigger a retry.

async requestHandler({ page, request }) {
  try {
    await page.waitForSelector('.product-title', { timeout: 10_000 });
  } catch (e) {
    Actor.log.warning(`Selector timeout: ${request.url}`);
    await Actor.pushData({ url: request.url, error: 'element_missing' });
    return; // Don't retry — likely page structure change
  }

  const title = await page.locator('.product-title').first().innerText();
  await Actor.pushData({ url: request.url, title });
}

Re-throwing propagates the error to Crawlee, which counts it as a failed request and retries up to maxRequestRetries.

failedRequestHandler: dead-letter behavior

When retries are exhausted, failedRequestHandler runs. Use it to log failures, push to a separate dataset, or enqueue for a later run. This is your dead-letter queue.

async failedRequestHandler({ request, log }) {
  log.error(`Permanent failure: ${request.url} after ${request.retryCount} retries`);

  await Actor.pushData({
    _failed: true,
    url: request.url,
    retryCount: request.retryCount,
    lastError: request.errorMessages?.pop(),
  });
}

Store failed URLs in a Dataset or KV store. A separate scheduled Actor can re-queue them days later when rate limits have reset.

Session pool for IP bans

When a site bans your IP mid-crawl, rotating user sessions helps. Session pools give each "logical user" a consistent cookie jar and optionally a proxy session.

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  useSessionPool: true,
  sessionPoolOptions: {
    maxPoolSize: 50,
    sessionOptions: {
      maxUsageCount: 10, // Rotate session after N requests
    },
  },
  proxyConfiguration: await Actor.createProxyConfiguration(),
  // ... requestHandler, failedRequestHandler
});

Configure a proxy in Apify Console or via input. Residential proxies reduce ban rates; see the proxy configuration guide.

Error grouping in Apify Console

Apify groups failed requests in Run → Log. Filter by "Failed" to see which URLs errored and the last error message. The Storage tab shows your dataset — if you push failed URLs via failedRequestHandler, they appear there for later analysis.

Alerting via webhooks

Set up webhooks to fire when a run fails. Use ACTOR.RUN.FAILED to notify Slack, PagerDuty, or a custom endpoint. See the webhooks guide for payload structure and authentication.

// Webhook config in Apify: Actor → Integrations → Webhooks
{
  "eventTypes": ["ACTOR.RUN.FAILED"],
  "requestUrl": "https://your-api.com/alert"
}

Add a filter for failure rate (e.g., alert only if >10% of requests failed) in your handler to avoid noisy alerts on minor blips.

Complete example: resilient PlaywrightCrawler

import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';

await Actor.init();

const crawler = new PlaywrightCrawler({
  maxRequestRetries: 5,
  requestHandlerTimeoutSecs: 90,
  useSessionPool: true,
  sessionPoolOptions: { maxPoolSize: 20 },
  proxyConfiguration: await Actor.createProxyConfiguration(),

  async requestHandler({ page, request }) {
    try {
      await page.waitForSelector('main', { timeout: 15_000 });
    } catch {
      Actor.log.warning(`Timeout on ${request.url}`);
      await Actor.pushData({ url: request.url, status: 'timeout' });
      return;
    }

    const title = await page.title();
    await Actor.pushData({ url: request.url, title });
  },

  async failedRequestHandler({ request }) {
    await Actor.pushData({
      _failed: true,
      url: request.url,
      retries: request.retryCount,
    });
  },
});

await crawler.run((await Actor.getInput())?.startUrls ?? [{ url: 'https://example.com' }]);
await Actor.exit();

Error handling strategy comparison

Strategy	When to use	Trade-off
Increase maxRequestRetries	Transient 5xx, rate limits	More CUs if many retries
failedRequestHandler + Dataset	Audit trail, later retry	Need second Actor to process
Session pool + proxy	IP bans, bot detection	Proxy cost
try/catch + partial push	DOM changes, optional fields	May miss data
Webhooks	Production alerting	Requires endpoint

Winner for production: Combine all five. Retries handle blips, failedRequestHandler captures permanent failures, session pool reduces bans, try/catch handles page variance, and webhooks keep you informed.

Error handling checklist

• maxRequestRetries set to 5+ for production
• failedRequestHandler logs and stores failed URLs
• requestHandler uses try/catch for selector/timeout errors
• Session pool enabled for ban-prone targets
• Webhook on ACTOR.RUN.FAILED for alerting
• Run details checked in Console for failure patterns

Make your Actors production-ready

Error handling is the highest-impact upgrade for any scraper. Spend 15 minutes adding failedRequestHandler and webhooks — your future self will thank you when a run fails at 2 AM.

Build resilient Actors on Apify | Configure webhooks

Frequently Asked Questions

maxRequestRetries controls how many times Crawlee retries a failed request before calling failedRequestHandler. Default is 3. For flaky sites, set 5–7. Crawlee uses exponential backoff between retries.

failedRequestHandler runs when a request has exhausted all retries (maxRequestRetries). Use it to log failures, push to a dataset for later retry, or trigger external alerting. It's your dead-letter queue.

Wrap selector logic in try/catch. If the element is optional, push partial data and return. If it's required, re-throw to trigger a Crawlee retry. Use waitForSelector with a reasonable timeout to avoid hanging.

Session pools give each logical user a consistent cookie/proxy session. Combined with proxy rotation, they spread requests across IPs. Set maxUsageCount to rotate sessions before they burn.

Yes. Create a webhook with event type ACTOR.RUN.FAILED. Your endpoint receives a JSON payload with runId and status. Wire it to Slack, PagerDuty, or a custom handler.

In the Run details, open the Log tab and filter by 'Failed'. Each entry shows the URL and last error. If you push failed URLs in failedRequestHandler, they also appear in the Storage/Dataset tab.