Data engineering: guides & tutorials

Firecrawl returns clean Markdown, structured JSON, and rich metadata from any URL — but by default it stores nothing. Every crawl result exists only for the duration of your API response. If you want to search past crawls, track changes over time, or build a content pipeline, you need persistent storage.

Supabase provides a managed PostgreSQL instance with a REST API, real-time subscriptions, and a built-in vector extension (pgvector) — making it the most developer-friendly database target for Firecrawl output. This tutorial walks through the complete pipeline: schema design, automated ingestion, full-text search, and incremental re-crawl updates.

Maintaining a separate scraper stack for every site—one-off Puppeteer for real estate, another pipeline for maps, another for social—does not scale. You end up babysitting browsers, proxies, and parsers instead of shipping.

The Apify Store is a catalog of 19,000+ ready-made Actors: packaged scrapers and automations (often built on Crawlee, Playwright, or similar) that someone else keeps patched as sites change. You pick an Actor, pass inputs, and run it in the cloud instead of owning the whole stack.

Below is a use-case map with links to our deeper category and collection pages.

Extracting POI (Point of Interest) and Local Business data from Google Maps at scale requires complex architectural engineering to bypass the platform's hard 120-result pagination limit per search query.

The Google Maps Scrapers collection on Apify hosts dedicated serverless Actors that use geographic grid partitioning to support high-coverage ingestion for city-wide or nation-wide location targeting.

Outbound sales lives or dies on clean, structured lead lists. Manual research does not scale; running scrapers from a single IP often hits WAFs (Cloudflare, DataDome, and similar) before you get volume.

The Lead Generation category on Apify is a set of hosted Actors aimed at pulling contacts, profiles, and company signals from directories and social surfaces—without you hosting browsers and proxy pools yourself.

Retail property investors often see prices move only after the market has already shifted. Quant teams build steady extraction pipelines from listing sites (Zillow, Realtor.com) so they can model cap rates, price-to-rent gaps, and days-on-market before those signals show up everywhere else.

Since the official Zillow API was retired, getting that data means getting past serious WAFs and running geographic searches at scale—often on the Apify Serverless Cluster.

If you are choosing a driver for web scraping and automation in 2026, the decision is mostly about protocol, waiting model, and browser coverage—not brand loyalty. This guide compares Selenium, Playwright, and Puppeteer feature by feature, sketches realistic performance expectations, shows minimal starter code for each, and ends with Playwright on Apify Crawlee as the default production path.

Quick verdict​

Playwright is the best choice for web scraping in 2026 — faster than Selenium, better supported than Puppeteer, with built-in auto-waiting and multi-browser support. Selenium 4 is best for legacy test suites or BiDi-mandated environments.

Use Puppeteer when you are Chrome-only (or Chrome + Firefox via BiDi), Node-only, and want a minimal CDP wrapper. Use Selenium when you must integrate with existing WebDriver-based QA, non-Node stacks, or Selenium Grid that already standardised on WebDriver.

Lumping these three platforms under the umbrella of "Web Scraping Tools" is a fundamental engineering error. They operate at entirely distinct layers of the OSI and Application stack.

Choosing the incorrect architecture guarantees extreme technical debt: either locking your team into managing headless Chrome clusters locally (when you thought you bought a scraping API), or overpaying for Serverless execution when you strictly needed raw TCP proxy pipes.

This guide provides a rigid architectural differentiation between Apify, Bright Data, and ScrapingBee for Data Engineering teams in 2026.

Retrieval-Augmented Generation (RAG) is entirely dependent on the structural integrity of its underlying Vector Database. Attempting to hydrate a Pinecone index by recursively dumping raw, unparsed HTML <div> nodes into an embedding model guarantees catastrophic semantic hallucination upon retrieval.

In 2026, the data ingestion pipeline is the critical engineering bottleneck. This guide details the explicit architectural flow required to extract complex JavaScript Single-Page Applications (SPAs), mathematically flatten the DOM into clean Markdown, and synchronize the vectors utilizing the Apify Serverless Infrastructure.

Companies scrape the open web when there is no stable API, when coverage matters more than a single vendor snapshot, or when you need fresh data on a schedule. Below is a numbered list with concrete examples and links into Apify use cases and our guides.

In 2026, constructing custom web scrapers utilizing default requests and BeautifulSoup libraries represents an extreme engineering anti-pattern. Modern targets are fiercely protected by Datadome and Cloudflare Enterprise WAFs, enforcing strict TLS fingerprint validation and executing dynamic JavaScript hydration payloads specifically designed to destroy headless execution.

Use Apify is an independent architectural playbook for Data Engineers. We focus exclusively on shifting data extraction workloads off local Python scripts and onto the Apify Serverless Cluster, allowing teams to treat the public web as an immediately queryable, structured database.