Drive real browsers with WebDriver across languages: legacy-friendly automation for complex flows, grids, and enterprise scraping integrated with Apify.
Selenium drives real browsers through WebDriver across many languages, making it a long-standing choice for scraping complex, interactive sites. These guides cover automating clicks, forms, and waits, plus running browser grids for parallel jobs.
Selenium handles legacy and enterprise flows well, though newer tools like Playwright offer faster, more resilient APIs for fresh projects. Either way, proxies and human-paced timing reduce blocks. Below you will find tutorials and patterns for integrating browser automation with Apify cloud runs.
When simple HTTP requests fail — because content is JavaScript-rendered, login is required, or pagination is AJAX-driven — you need a real browser. Playwright, Puppeteer, and Selenium are the three dominant tools. This deep dive covers when to use each, advanced techniques (network interception, CDP access, fingerprint evasion), and how to run them at scale on Apify.
Default to Playwright for new browser automation: one install drives Chromium, Firefox, and WebKit with built-in auto-wait, ~50–80 MB per context, and Crawlee/Apify integration. Pick Puppeteer 25 for Chrome-only Node services that benefit from raw CDP or the new WebDriver BiDi transport. Keep Selenium 4 when you need Java/C#, Selenium Grid, or BiDi network logging across legacy suites.
In 2026, three tools dominate browser automation: Playwright (Microsoft), Puppeteer 25 (Google, now with WebDriver BiDi), and Selenium 4 (cross-vendor, BiDi-capable). The right choice depends on your use case: new projects should default to Playwright; Puppeteer fits Chrome-focused, lightweight needs; Selenium remains for Java/C# teams and legacy systems. Run Playwright on Apify.
The best Udemy Selenium courses in 2026 are Selenium WebDriver with Python & PyTest (4.7★, beginners), Selenium WebDriver Java – Complete Automation Testing Master (4.5★, 20K+ students), and Python From Scratch & Selenium WebDriver QA Automation (4.6★, 20K+ students). Selenium skills transfer to web scraping; Playwright is the modern alternative for new projects.
For new scraping projects in 2026, Playwright wins: it runs Chromium, Firefox, and WebKit from one install with built-in auto-wait and trace viewer, hitting ~35–55 pages/min sequentially on static URLs. Puppeteer 25 is a tighter Chrome/Firefox CDP wrapper with WebDriver BiDi support and lower idle RAM. Selenium 4 still leads when WebDriver Grid, Java/C#, or BiDi network logging are non-negotiable.
If you are choosing a driver for web scraping and automation in 2026, the decision is mostly about protocol, waiting model, and browser coverage—not brand loyalty. This guide compares Selenium, Playwright, and Puppeteer feature by feature, sketches realistic performance expectations, shows minimal starter code for each, and ends with Playwright on Apify Crawlee as the default production path.
Playwright is the best choice for web scraping in 2026 — faster than Selenium, better supported than Puppeteer, with built-in auto-waiting and multi-browser support. Selenium 4 is best for legacy test suites or BiDi-mandated environments.
Use Puppeteer when you are Chrome-only (or Chrome + Firefox via BiDi), Node-only, and want a minimal CDP wrapper. Use Selenium when you must integrate with existing WebDriver-based QA, non-Node stacks, or Selenium Grid that already standardised on WebDriver.
Selenium is the original browser automation framework, now on version 4 with improved API and W3C standardization. It remains the most widely used in enterprise test automation and is supported by all major browsers. For scraping, it is slower than Playwright but familiar to many QA teams who repurpose test suites for data extraction.
Playwright has a more modern API, better async support, network interception, and faster multi-browser setup. Selenium is more universal—it works with every language binding and every browser. For new scraping projects, Playwright is generally preferred; Selenium is worth using if you already have an existing test infrastructure.
Use WebDriverWait with expected_conditions to wait for specific elements rather than time.sleep. Explicit waits are more reliable and faster than implicit waits. For infinite scroll, execute JavaScript scrollTo calls and wait for network requests to settle. Log what the page looks like at failure time using driver.save_screenshot.
Yes, use headless Chrome or Firefox mode with ChromeOptions or FirefoxOptions. In containerized environments, Selenium Grid Docker images handle browser provisioning. Apify offers Selenium-based actor templates for teams that prefer the Selenium ecosystem over Playwright without managing browser infrastructure themselves.
