Guide: guides & tutorials

Enterprise competitive intelligence tools — Crayon, Klue, Kompyte, Similarweb — charge $300–$2,000/month (quote-based, plan-dependent) for competitive monitoring dashboards. This same monitoring can be built with Apify for data collection, Claude for analysis, n8n for orchestration, and a free dashboard — for under $50/month (starting cost; scales with competitor count, Actor fees, and proxy usage).

This guide builds it step by step: from identifying what to monitor, to automated daily scrapes, to AI-powered change detection that alerts your team in Slack when competitors make moves that matter.

TL;DR:

The Model Context Protocol (MCP) is an open standard (originated by Anthropic in November 2024, now open and community-governed) that lets AI assistants like Claude call external tools, read resources, and use reusable prompts. Instead of hardcoding integrations, you run MCP servers, lightweight processes that expose capabilities over a simple protocol. Claude (Desktop and Code), ChatGPT, VS Code, Cursor, and many other MCP hosts connect to these servers, so your AI can scrape the web, query databases, or run APIs on your behalf. Think of MCP as a USB-C port for AI: one standard plug instead of a custom cable per integration. Explore the Apify MCP server.

Scrapers fail constantly: HTTP 429s, navigation timeouts, elements that never appear, Cloudflare blocks. Without proper error handling, a single bad request can kill your entire run or silently drop data. This guide covers every tool Apify and Crawlee give you to build scrapers that survive real-world failures — retries, dead-letter queues, session pools, and webhook alerting. Start building resilient Actors on the Apify platform.

Apify bills by Compute Units: Memory (GB) × runtime (hours). Oversizing Actors wastes money; undersizing causes OOM crashes. This guide covers memory sizing, Cheerio vs Playwright trade-offs, Playwright resource blocking, concurrency tuning, and dataset batching so you cut costs by 30–50% without losing throughput. Optimize your Actors in the Apify Console.

Competitive intelligence (CI) in 2026 is data-driven: pricing changes, hiring signals, SEO shifts, product releases, and PR sentiment. Manual monitoring doesn't scale. A production CI system combines Apify for data collection, PostgreSQL for storage, Python/dbt for analysis, Grafana or Metabase for dashboards, and Slack for alerts. This guide covers architecture, five core signals, scheduling, and cost estimates.

JavaScript and Node.js power some of the most capable web scrapers in 2026. The ecosystem spans Axios and node-fetch for HTTP, Cheerio for HTML parsing, Playwright and Puppeteer for browser automation, and Crawlee as the full framework that powers Apify Actors. This guide covers the JS scraping stack, comparison tables, TypeScript patterns, data output options, and a complete Crawlee TypeScript Actor example. Try Apify to run Crawlee Actors in the cloud.

B2B lead gen pulls from company sites, contact pages, LinkedIn, Google Maps, Crunchbase, job boards, and similar public surfaces. Scraping automates the collection step; you still layer enrichment (Hunter.io, Clearbit), scoring (rules or a model), and CRM push (HubSpot, Salesforce). This guide maps sources, GDPR/CCPA basics, pipeline shape, and how Apify compares to Apollo.io and ZoomInfo for that stack.

For the evergreen reference version with field-by-field source tables and a full Apify Store walkthrough, see the B2B lead generation with web scraping use-case guide.

Real estate investors and analysts need data: market comps, rental yields, price trends, days on market, and new listing alerts. Public listing sites (Zillow, Realtor.com, Redfin, Rightmove, Idealista) hold this data, but their official APIs are limited or expensive. Web scraping fills the gap. Apify offers Zillow and Realtor.com scrapers in the Store. This guide covers use cases, data schema, workflows, legal considerations, and MLS alternatives.

SEO in 2026 means data: SERP positions, competitor content changes, broken links, duplicate titles. Manual tracking doesn't scale. Apify automates it — run Actors on schedule, export to Google Sheets, and build dashboards without writing crawlers from scratch. Browse SEO Actors in the Apify Store.

When simple HTTP requests fail — because content is JavaScript-rendered, login is required, or pagination is AJAX-driven — you need a real browser. Playwright, Puppeteer, and Selenium are the three dominant tools. This deep dive covers when to use each, advanced techniques (network interception, CDP access, fingerprint evasion), and how to run them at scale on Apify.