Browser automation handles JS-heavy pages, auth, and infinite scroll beyond raw HTTP. Apify runs Playwright and Puppeteer with proxies and dataset storage.
Browser automation handles what plain HTTP requests cannot: JavaScript rendering, logins, infinite scroll, and clicking through interactive flows. These guides cover Playwright, Puppeteer, and Selenium for driving real browsers in a scraping pipeline.
Running browsers is heavier than HTTP, so use them where the page truly needs it and add proxies plus human-paced timing to avoid blocks. Apify runs browser actors with proxy and dataset storage built in. Below you will find tutorials and patterns for reliable browser-based extraction.
When simple HTTP requests fail — because content is JavaScript-rendered, login is required, or pagination is AJAX-driven — you need a real browser. Playwright, Puppeteer, and Selenium are the three dominant tools. This deep dive covers when to use each, advanced techniques (network interception, CDP access, fingerprint evasion), and how to run them at scale on Apify.
Default to Playwright for new browser automation: one install drives Chromium, Firefox, and WebKit with built-in auto-wait, ~50–80 MB per context, and Crawlee/Apify integration. Pick Puppeteer 25 for Chrome-only Node services that benefit from raw CDP or the new WebDriver BiDi transport. Keep Selenium 4 when you need Java/C#, Selenium Grid, or BiDi network logging across legacy suites.
In 2026, three tools dominate browser automation: Playwright (Microsoft), Puppeteer 25 (Google, now with WebDriver BiDi), and Selenium 4 (cross-vendor, BiDi-capable). The right choice depends on your use case: new projects should default to Playwright; Puppeteer fits Chrome-focused, lightweight needs; Selenium remains for Java/C# teams and legacy systems. Run Playwright on Apify.
Stagehand is BrowserBase's open-source framework that extends Playwright with LLM-powered natural language commands. You can page.act("click the login button") or page.extract({ instruction: "...", schema: {...} }) instead of writing brittle selectors. Stagehand shines on complex, dynamic UIs; plain Playwright remains better for high-volume scraping where LLM latency and cost matter. Use Apify for scalable browser automation.
browser-use is an open-source Python library that gives an LLM control of a real browser (via Playwright). It runs a perceive–act loop: the page DOM is pruned and tagged, the model chooses actions like click or type, and the loop repeats until the task finishes. It shines when layouts change often and fixed selectors break; it costs more in tokens and time than traditional scrapers, and it is a poor fit for hard WAFs or huge deterministic crawls. For production, run it in containers on Apify with proxy rotation.
Traditional automation (Playwright or Puppeteer) depends on stable selectors. If a team hardcodes .submit-btn and the site renames classes, the job fails.
browser-use inverts that: you describe the goal in natural language, the library feeds a sanitized view of the page to an LLM, and the model plans clicks, typing, and extractions through Playwright.
This guide covers the architecture, where it breaks in production, and how to pair it with Apify Actors when you need cloud browsers and proxies.
Use real browsers when sites rely on client-side rendering, complex interactions, or anti-bot checks that block simple HTTP clients. Login flows, infinite scroll, and shadow DOM often need Playwright or Puppeteer. Apify hosts these stacks with scaling and storage so you are not babysitting headless Chrome on a laptop.
Prefer resilient selectors, wait for network idle judiciously, and capture screenshots on failure. Version browser images and test upgrades in staging. Apify datasets with error fields help you triage flaky steps without reproducing every failure manually. Log enough context for on-call engineers to diagnose failures quickly.
RAM and CPU per page exceed plain HTTP fetches, and concurrency must stay lower to avoid melting targets or your budget. Session reuse reduces login overhead but risks cross-talk if state leaks. Monitor memory in long sessions and restart browsers periodically to prevent gradual heap growth.
Yes, and you should when lists load over JSON but details need a browser. Fetch cheap endpoints first, then launch browsers only for records that require it. This hybrid pattern is common in Apify actors optimizing cost versus coverage. Schedule periodic access reviews for datasets that include personal data.
