Scraping dynamic websites with Playwright, Puppeteer, and Crawlee

Contains affiliate links. Disclosure.

To scrape dynamic, JavaScript-rendered websites, drive a real browser with Playwright or Puppeteer through Crawlee, wait for the content to render (waitForSelector or network idle), then read the live DOM or intercept the site's own JSON API calls. Run it as an Apify Actor for managed browsers, proxies, and autoscaling.

Single-page apps built with React, Vue, or Angular often ship an almost empty HTML shell and render content in the browser. Pure HTTP clients never execute that JavaScript, so they miss the DOM you see in DevTools. That gap is one of the most common web scraping challenges teams hit when they outgrow simple scripts.

Quick answer

Scrape JavaScript-heavy websites (React, Vue, Angular) using Playwright or Puppeteer with Crawlee. Run full browser automation in Apify Actors to get the fully rendered DOM, or capture the XHR/JSON responses the page fetches for the cleanest data.

On Apify, package your crawler as an Actor so you get scalable headless browsers, storage, proxy rotation, and scheduling without operating your own Chrome farm.

Why JavaScript rendering matters

Signal	HTTP-only scraper	Real browser (Playwright / Puppeteer)
HTML contains data	Sometimes (SSR/static)	After JS runs, yes for SPAs
Client-side routing	Usually breaks	Can navigate like a user
Lazy-loaded lists	Misses below-the-fold content	Scroll / wait / intercept network
Canvas/WebGL widgets	Hard	Accessible where legal/allowed

If View Page Source is tiny but Inspect Element shows the data, you need a browser.

Cheerio (HTTP + static parse) vs Playwright (browser automation)

Cheerio (often paired with got, fetch, or Crawlee’s HTTP crawlers) parses HTML returned by the server. It is fast, cheap in CPU/RAM, and ideal when the JSON you need is already embedded in the first response or in predictable API calls you can replay.

Playwright drives a real Chromium/WebKit/Firefox instance: it executes JavaScript, fires events, reads the live DOM, and can capture network responses (often the cleanest data source on SPAs).

Rule of thumb: start by checking XHR/fetch payloads in DevTools. If a stable JSON endpoint exists and you may call it ethically, prefer HTTP. If the site only materializes data in the DOM after JS, use PlaywrightCrawler (or PuppeteerCrawler) from Crawlee.

Playwright vs Puppeteer for scraping

Crawlee ships both PlaywrightCrawler and PuppeteerCrawler, so you can pick either driver and keep the same request queue, session pool, and Apify integration. The differences come down to browser coverage and API surface.

Factor	Playwright	Puppeteer
Browser engines	Chromium, Firefox, WebKit	Chromium (and Chrome); experimental Firefox
Maintainer	Microsoft	Chrome DevTools team (Google)
Auto-waiting	Built into most actions (click, fill)	Mostly manual (waitForSelector)
Multi-language	Node, Python, Java, .NET	Node (Python via unofficial ports)
Network capture	page.on("response"), expect_response	page.on("response"), request interception
Best fit	New projects, cross-browser, Python crawlers	Existing Chrome-only Node code

Default pick: Playwright for new work (broader engine support, first-class Python). Keep Puppeteer when you inherit a working Node Actor and have no reason to migrate. Choose one driver per project rather than mixing them.

Waiting for content to render

The single most common dynamic-scraping mistake is reading the DOM before the framework has painted it. Three reliable signals:

• waitForSelector / wait_for_selector waits for a specific element to appear. Prefer this over fixed sleeps because it returns as soon as the node exists.
• Network idle (page.wait_for_load_state("networkidle")) waits until no requests have fired for a short window. Useful after triggering lazy loads, but slower on sites that poll in the background.
• expect_response / page.waitForResponse resolves when a matching URL responds, which is the most precise option when you know the API call you depend on.

Infinite scroll and click interactions

Lazy lists and "Load more" buttons only reveal data through user-like actions. The pattern is the same in both drivers: trigger the action, wait for new nodes or a network response, repeat until growth stops or a sentinel disappears. The Python example below uses a click loop; for infinite scroll, replace the click with a page.mouse.wheel(0, height) or window.scrollTo(0, document.body.scrollHeight) step inside page.evaluate, then re-measure document.body.scrollHeight and stop when it stabilizes. Always attach a deduplication key so repeated nodes do not create duplicate rows.

Intercepting XHR/JSON endpoints (often the fastest route)

Most SPAs render from an internal JSON API. Capturing those responses gives you clean, typed data with no fragile DOM selectors, and it is usually far cheaper than parsing rendered HTML. In Playwright Python you subscribe to the response event and filter by URL:

captured: list[dict] = []

async def request_handler(context) -> None:
    page = context.page

    async def on_response(response) -> None:
        if "/api/products" in response.url and response.status == 200:
            captured.extend((await response.json())["items"])

    page.on("response", on_response)
    await page.goto(context.request.url)
    await page.wait_for_load_state("networkidle")

    for item in captured:
        await Actor.push_data(item)

You can also wait for a specific call with page.expect_response("**/api/products*") when the data loads after a click. Only replay endpoints you are permitted to use, and respect the site's terms and robots.txt. For the bot-detection side of this, see anti-scraping techniques.

Python example: PlaywrightCrawler with a “Load more” loop

Below, the crawler waits for product cards, clicks Load more until it disappears, then reads titles from the rendered DOM. Adapt selectors to your target.

import asyncio

from apify import Actor
from crawlee.crawlers import PlaywrightCrawler


async def main() -> None:
    async with Actor:
        crawler = PlaywrightCrawler(max_requests_per_crawl=50)

        @crawler.router.default_handler
        async def request_handler(context) -> None:
            page = context.page
            log = context.log
            log.info("Processing %s", context.request.url)

            await page.wait_for_selector(".product-item", timeout=10_000)

            while await page.query_selector(".load-more-button"):
                log.info('Clicking "Load more"')
                await page.click(".load-more-button")
                await page.wait_for_timeout(500)
                await page.wait_for_load_state("networkidle")

            titles = await page.eval_on_selector_all(
                ".product-item .title",
                "els => els.map(el => el.textContent?.trim()).filter(Boolean)",
            )
            for title in titles:
                await Actor.push_data({"title": title})

        await crawler.run(["https://example.com/products"])


if __name__ == "__main__":
    asyncio.run(main())

Prefer condition-based waits

Replace wait_for_timeout with waits on element counts, specific responses, or mutation when you can. Static sleeps are a last resort.

Node.js example: Crawlee PlaywrightCrawler with fingerprints

Use this pattern when you need consistent browser fingerprints and session pools on harder sites (often combined with proxies):

import { PlaywrightCrawler } from 'crawlee';
import { Actor } from 'apify';

await Actor.init();

const crawler = new PlaywrightCrawler({
    useSessionPool: true,
    browserPoolOptions: {
        useFingerprints: true,
        fingerprintOptions: {
            fingerprintGeneratorOptions: {
                browsers: ['chrome'],
                devices: ['desktop'],
                operatingSystems: ['windows'],
            },
        },
    },
    async requestHandler({ page, request, log }) {
        log.info(`Processing ${request.url}`);
        await page.waitForSelector('article', { timeout: 15_000 });
        const items = await page.$$eval('article h2', (nodes) =>
            nodes.map((n) => n.textContent?.trim()).filter(Boolean),
        );
        for (const title of items) {
            await Actor.pushData({ title });
        }
    },
});

await crawler.run(['https://example.com/news']);
await Actor.exit();

Deploying on Apify gives you managed browsers, proxy configuration, and Compute Unit billing aligned with real resource use.

Which technique should I use?

Situation	Technique	Why
Data is in the first HTML response	Cheerio / HTTP crawler	Fastest, cheapest, no browser needed
Data loads from a JSON API you may call	Intercept response or call the endpoint	Clean structured data, low cost
Data only exists in the DOM after JS	PlaywrightCrawler + waitForSelector	Renders the SPA before extraction
"Load more" buttons or infinite scroll	Browser + click/scroll loop	Triggers lazy content like a user
Heavy bot detection or geo-blocks	Browser + Apify proxies + fingerprints	Rotates identity and IP per session

Scaling dynamic scrapers on Apify

Running headless browsers reliably at volume is the hard part: you need a pool of Chromium instances, rotating proxies, retries, and storage. Crawlee handles autoscaling and the request queue locally, and deploying the same crawler as an Apify Actor adds the managed infrastructure honestly:

• Managed browsers: the Apify base Docker images ship Playwright/Puppeteer and matching browser binaries, so local runs match production.
• Proxy rotation: attach Apify residential or datacenter proxies through the Crawlee proxyConfiguration so blocked requests retry on a fresh IP.
• Autoscaling: Crawlee scales concurrency up or down based on available CPU and memory, and Apify bills via Compute Units tied to real usage.
• Storage and scheduling: push records to the Actor dataset and run on a cron schedule from the Apify Console.

See the Apify SDK JavaScript tutorial and the Crawlee Python tutorial for end-to-end Actor builds.

Cost and performance notes

• Playwright/Puppeteer Actors consume more RAM and CPU than Cheerio-style crawlers, so raise memory when pages are heavy.
• Block images/fonts if you do not need them (route.abort) to cut bandwidth and speed up navigation.
• Concurrency: each browser context has a price; scale down before you scale up when debugging.
• Intercept over parse when a JSON API is available; it avoids rendering cost entirely.

Further reading

• Crawlee Playwright guide: crawlee.dev (Node and Python sections)
• Build an Apify Actor: packaging and deployment
• Crawlee Python tutorial: beginner path

Technical details were checked against current Crawlee and Apify documentation; verify APIs in upstream docs when upgrading versions.

Start building browser Actors on Apify →

FAQ

Frequently Asked Questions

Use Playwright when the data is produced client-side, requires clicks, infinite scroll, or authenticated SPA navigation. Use Cheerio or HTTP crawlers when the server returns complete HTML or when you can legally call a stable JSON API directly.

Yes. Crawlee supports PuppeteerCrawler as well as PlaywrightCrawler. Playwright tends to offer broader browser support and a modern API, but Puppeteer remains common in existing Actors, so pick one stack per project for maintainability.

Loop: scroll to the bottom, wait for new nodes or network idle, stop when height stops growing or a sentinel appears. Combine with deduplication keys so you do not push duplicate records.

Common causes are missing dependencies in Dockerfile, different default timeouts, missing proxies for blocked regions, and memory limits that are too low for Chromium. Check Actor logs, increase memory, and mirror Apify base images recommended in the docs.

Yes. Headless browsers use more compute units than plain HTTP crawlers. Mitigate by blocking heavy assets, lowering concurrency, and only using browsers on URLs that truly need them.

Subscribe to the browser's response event (page.on("response", ...) in Playwright), filter by the API URL, and read response.json(). For data that loads after a click, use page.expect_response (Python) or page.waitForResponse (Node) to wait for the matching call. This is often cleaner and cheaper than scraping the rendered DOM.

waitForSelector waits for a specific element to appear and returns as soon as it exists, which is precise and fast. networkidle waits until no network requests fire for a short window, which is broader but slower and unreliable on pages that poll in the background. Prefer selector or response waits over fixed sleeps.