Testing scrapers and Apify Actors with fixtures, mocks, and staging targets. Catch selector drift and rate limits early so production runs stay stable.
Testing scrapers with fixtures, mocks, and staging targets catches selector drift and rate-limit issues before they hit production. These guides cover building a test harness for Apify Actors and crawlers.
Tests against saved page snapshots make brittle scrapers far more reliable as sites change. Below you will find patterns for testing extraction logic and keeping production runs stable.
Test your Apify Actors before they run in production. Use unit tests for extraction logic, integration tests for live URLs, and apify run for end-to-end validation. Test in the Apify Console or run locally.
A linear Python script with requests and a for loop over 500 URLs is not a production system. In real deployments, markup changes, socket timeouts, and bad proxy exits eventually break naive runs.
To move from a side project to production, your pipeline needs fault tolerance, state, and observability.
This guide covers four practical building blocks for running high-volume extraction reliably.
The best web scraping tools in 2026 are Apify (#1 all-in-one platform, $5 free credit, Starter $29/mo), Scrape.do (#2 best-value API, $29/mo Hobby = 250,000 successful requests, 1,000 free, no card), Bright Data (best enterprise unblocking and proxy network), Firecrawl (best for LLM and RAG markdown, $16/mo Hobby), and Crawlee (best open-source library, free MIT).
The best tool depends on whether you want a hosted platform, a drop-in API, a code library you run yourself, or raw proxies. This roundup compares thirteen mature options with honest pros and cons, a single comparison table (type, best-for, free tier, pricing floor, skill level), and a short decision framework.
Unit test parsing functions against fixture HTML files to catch selector failures without network requests. Integration test full actor runs against a curated URL set. Schema validate all outputs. Run regression tests against archived HTML snapshots when selectors change. Add smoke tests that run daily against live sites and alert on failure.
Fixture-based unit tests for every parser function, integration tests for full run coverage, schema validation tests that run on every output, negative tests for missing fields and malformed HTML, and performance tests that catch concurrency regressions. The test suite should run in CI on every commit and complete in under 5 minutes.
Record HTTP responses from live sites as fixtures and replay them in tests—VCR cassette libraries do this for Python and Node. This avoids flaky tests from site changes while ensuring tests reflect real response shapes. Refresh fixtures monthly or when sites change structure. Keep a small set of live integration tests that run less frequently.
Jest or Vitest for unit testing TypeScript actors, pytest for Python actors, the Apify CLI for local actor execution, and the Apify API for running test builds against staging environments. Use the Actor.pushData mock in local testing to validate dataset writes without consuming platform credits during development.
