How to resume Apify scraping after a crash or timeout

Contains affiliate links. Disclosure.

Quick answer

Resume interrupted Apify Actor runs using request queues: Crawlee’s RequestQueue (backed by Apify storage when named) persists pending requests and marks handled ones, so a restarted Actor picks up exactly where it left off instead of re-enqueueing the whole seed list.

Open Apify · Crawlee docs

Long crawls stop for many reasons: pay-per-result caps, max cost per run, timeouts, OOM, or transient site errors. Store owners and developers use three layers of resilience: platform resume where the Actor supports it, input offsets for Store Actors that expose cursors, and persistent request queues (plus optional key-value checkpoints) for custom Crawlee code.

Why runs stop

Reason	What happened	Mitigation
Maximum charged results	Pay-per-result Actor hit its cap	Raise limit or chain runs
Maximum cost per run	Pay-per-event spend cap reached	Raise cap or split work
Timeout	Run exceeded max duration	Increase timeout or shard URLs
Out of memory	RAM too low for page/browser	More memory or smaller concurrency
Platform / network	Rare infrastructure blip	Retry; queues make retries safe

Defaults

If you do not set caps, a run continues until completion or until account credits are exhausted, whichever comes first.

Store Actors: resume and offsets

Built-in resume

Many published Actors expose Resume / Continue on a stopped run in the Apify Console when the author implemented stateful storage. Check the run’s Logs and the Actor README for whether resume is supported.

Manual restart with cursor / offset

If there is no resume button:

1. Open the stopped run and note how many results landed in the dataset.
2. Start a new run with the Actor’s start cursor, offset, skip already processed, or equivalent input (names vary by Actor).
3. Deduplicate downstream if the Actor cannot guarantee idempotent writes.

This pattern is Actor-specific; always read the input schema on the Store page.

Developer approach: persistent request queues

Crawlee’s request queue stores pending and handled requests in Apify when you use a named queue via Actor.openRequestQueue('name'). After a crash or intentional stop, starting another run with the same queue name reloads state: handled URLs are skipped, pending URLs are processed.

JavaScript (CheerioCrawler)

import { Actor } from "apify";
import { CheerioCrawler } from "crawlee";

await Actor.init();

const queue = await Actor.openRequestQueue("my-brand-crawl-v1");

const crawler = new CheerioCrawler({
  requestQueue: queue,
  maxRequestRetries: 2,
  async requestHandler({ request, $ }) {
    const title = $("title").text();
    await Actor.pushData({ url: request.url, title });
  },
});

await crawler.run(["https://example.com/start"]);
await Actor.exit();

Adding requests without losing resume

Use queue.addRequests() (or crawler helpers) so new URLs enter the same named queue. Avoid recreating the queue under a new name unless you intend a fresh crawl.

Python (Crawlee for Python)

Use a named RequestQueue so the same frontier is restored on the next run (when using Apify storage):

import asyncio

from apify import Actor
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.storages import RequestQueue

async def main() -> None:
    async with Actor:
        rq = await RequestQueue.open(name="my-brand-crawl-v1")
        crawler = BeautifulSoupCrawler(request_manager=rq, max_request_retries=2)

        @crawler.router.default_handler
        async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
            title = context.soup.title.string if context.soup.title else ""
            await Actor.push_data({"url": context.request.url, "title": title})

        await crawler.run(["https://example.com/start"])

if __name__ == "__main__":
    asyncio.run(main())

Queue names must follow Apify naming rules (lowercase letters, digits, hyphens). See Apify Python SDK and Crawlee for Python if your template uses different imports.

Checkpoints with key-value store

For progress that is not a URL (e.g. last listing page number, OAuth token rotation), persist JSON with the key-value store:

await Actor.setValue("CHECKPOINT", { lastPage: 42, updatedAt: Date.now() });
const checkpoint = await Actor.getValue("CHECKPOINT");

Combine named request queue (URL frontier) with occasional checkpoints (high-level cursor) for large sites with pagination quirks.

Best practices

1. Name your storage: named request queues and datasets survive across runs; default run-scoped storage does not help the next run.
2. Idempotent pushData: if you might reprocess a URL after a partial failure, include stable primary keys and dedupe in your pipeline, or use upsert logic downstream.
3. Tune retries: set maxRequestRetries (and backoff where available) so transient 5xxs do not permanently drop URLs.
4. Set explicit limits: use max results / max cost while testing so stops are predictable; widen for production.
5. Alert on failures: hook Slack or monitoring so you restart quickly while the queue still holds pending work.

FAQ

Frequently Asked Questions

For custom Actors, use Crawlee with a named RequestQueue backed by Apify storage: pending requests remain after a stop, and the next run continues the frontier. Some Store Actors also offer a Resume button or cursor inputs.

Yes when you use Actor.openRequestQueue with a stable name: the queue lives in your Apify storage and is shared by every run that uses that name.

If a URL is already marked handled in the queue, Crawlee will not rerun it. If you push duplicate logical records outside the queue or use multiple datasets, dedupe downstream with a stable key.

Yes: start a new run that continues from the queue state or cursor. The previous run stops cleanly at the cap; the queue still contains unhandled requests unless the Actor drains it differently.

Implement the same idea: persist a work queue and a done-set (database, Apify queue, or KV) and make your handler safe to retry.

Persist often enough that a timeout only loses in-flight pages, not the whole frontier: named queues plus optional KV checkpoints.