Guide: guides & tutorials

Enterprise competitive intelligence tools — Crayon, Klue, Kompyte, Similarweb — charge $300–$2,000/month (quote-based, plan-dependent) for competitive monitoring dashboards. This same monitoring can be built with Apify for data collection, Claude for analysis, n8n for orchestration, and a free dashboard — for under $50/month (starting cost; scales with competitor count, Actor fees, and proxy usage).

This guide builds it step by step: from identifying what to monitor, to automated daily scrapes, to AI-powered change detection that alerts your team in Slack when competitors make moves that matter.

TL;DR:

Scrapers fail constantly: HTTP 429s, navigation timeouts, elements that never appear, Cloudflare blocks. Without proper error handling, a single bad request can kill your entire run or silently drop data. This guide covers every tool Apify and Crawlee give you to build scrapers that survive real-world failures — retries, dead-letter queues, session pools, and webhook alerting. Start building resilient Actors on the Apify platform.

SEO in 2026 means data: SERP positions, competitor content changes, broken links, duplicate titles. Manual tracking doesn't scale. Apify automates it — run Actors on schedule, export to Google Sheets, and build dashboards without writing crawlers from scratch. Browse SEO Actors in the Apify Store.

Apify bills by Compute Units: Memory (GB) × runtime (hours). Oversizing Actors wastes money; undersizing causes OOM crashes. This guide covers memory sizing, Cheerio vs Playwright trade-offs, Playwright resource blocking, concurrency tuning, and dataset batching so you cut costs by 30–50% without losing throughput. Optimize your Actors in the Apify Console.

Bright Data Scraping Browser is a cloud-hosted headless Chrome that connects via Chrome DevTools Protocol (CDP). It's Playwright- and Puppeteer-compatible, with built-in fingerprinting, CAPTCHA solving, and IP rotation — no proxy configuration required. For JavaScript-heavy sites behind Cloudflare, LinkedIn, or Amazon-level anti-bot, it's the fastest path to reliable extraction. Try Bright Data Scraping Browser.

When simple HTTP requests fail — because content is JavaScript-rendered, login is required, or pagination is AJAX-driven — you need a real browser. Playwright, Puppeteer, and Selenium are the three dominant tools. This deep dive covers when to use each, advanced techniques (network interception, CDP access, fingerprint evasion), and how to run them at scale on Apify.

Competitive intelligence (CI) in 2026 is data-driven: pricing changes, hiring signals, SEO shifts, product releases, and PR sentiment. Manual monitoring doesn't scale. A production CI system combines Apify for data collection, PostgreSQL for storage, Python/dbt for analysis, Grafana or Metabase for dashboards, and Slack for alerts. This guide covers architecture, five core signals, scheduling, and cost estimates.

E-commerce data collection in 2026 spans four core types: product catalog, pricing, reviews and ratings, and inventory. Each has different crawl frequency needs, anti-bot considerations, and target-specific approaches. This guide covers data types, price monitoring architecture, target breakdowns (Amazon, Shopify, WooCommerce), review scraping strategies, inventory tracking, Apify Store Actors, data schemas, and a price alert system. For pre-built scrapers, the Apify Store offers Amazon, Shopify, and Walmart Actors. For anti-bot heavy targets, Bright Data provides Scraping Browser and datasets.

The Firecrawl Node.js SDK (@mendable/firecrawl-js) turns any URL into markdown, HTML, or structured JSON with one call. Install it, pass your API key, and use scrape(), crawl(), map(), or extract() — no browser setup required. Get your Firecrawl API key.

Static HTML scrapers fail on modern sites: content loaded via AJAX, infinite scroll feeds, and single-page apps (SPAs) render in the browser, not the initial response. View Source shows a near-empty shell; Inspect Element reveals the full DOM. This guide covers three approaches—API interception (fastest), Playwright waits (reliable), and scroll/click triggers (for infinite scroll)—plus SPA-specific techniques and a Playwright vs httpx vs Crawlee comparison. Deploy on Apify for managed execution.