Architecture: guides & tutorials

Many RAG (Retrieval-Augmented Generation) projects fail in production not because the technology doesn't work, but because teams skip the hard parts: chunking strategy, embedding model selection, retrieval quality measurement, and stale data management. Validate each step on your own corpus — field names, SDK versions, and Actor outputs change over time.

The pipeline: crawl websites → chunk intelligently → embed → store in vector DB → retrieve with reranking → generate answers with citations.

TL;DR:

Prerequisites:

• Python 3.10+ or Node.js 18+
• Apify account (sign up)
• Vector database (Qdrant Cloud free tier or self-hosted)
• LLM API key (Claude, GPT-4, or self-hosted Ollama)

Web scraping architectures evolve from a single script to queue-based, distributed, and managed platforms. Each level trades simplicity for scalability, reliability, and maintainability. This guide describes four architecture levels, when to move between them, data pipeline integration, anti-detection at the architecture level, observability, and a Level 2 code example with Crawlee, Redis, and PostgreSQL. For managed infrastructure that handles queues and scaling for you, Apify provides a Level 4 option out of the box.

Quick answer: Production-grade web scraping requires (1) checking for APIs first, (2) choosing the right tool for the site's rendering complexity, (3) respecting rate limits with delays and backoff, (4) using proxy rotation for blocked targets, (5) building schema-validated extraction with retries, and (6) logging structured metrics for monitoring. The practices below are ranked by impact.

Production-grade web scraping is engineering, not just writing selectors. This guide covers the patterns that separate throwaway scripts from scrapers that run reliably for months.

Maintaining a separate scraper stack for every site—one-off Puppeteer for real estate, another pipeline for maps, another for social—does not scale. You end up babysitting browsers, proxies, and parsers instead of shipping.

The Apify Store is a catalog of 19,000+ ready-made Actors: packaged scrapers and automations (often built on Crawlee, Playwright, or similar) that someone else keeps patched as sites change. You pick an Actor, pass inputs, and run it in the cloud instead of owning the whole stack.

Below is a use-case map with links to our deeper category and collection pages.

A linear Python script with requests and a for loop over 500 URLs is not a production system. In real deployments, markup changes, socket timeouts, and bad proxy exits eventually break naive runs.

To move from a side project to production, your pipeline needs fault tolerance, state, and observability.

This guide covers four practical building blocks for running high-volume extraction reliably.

In April 2026, Walmart has become the primary battleground for retail arbitrage and market intelligence. But for developers, it remains one of the most difficult targets on the web.

Unlike other retailers that block you with a clear 403 Forbidden, Walmart is famous for the "Silent 200." You send a request, you get a "Success" status code, but the HTML payload is missing the actual price, stock, and SKU data. To your monitoring script, everything looks fine; to your business, the data is useless.

This guide explains how to build a production-grade Walmart scraper using Apify that bypasses shadow-bans and delivers verified data.

If you trade on price spreads or run arbitrage between marketplaces, you need Walmart numbers you can trust—not a pretty HTTP status with junk in the body.

Walmart’s edge is aggressive bot filtering. A naive script may “succeed” on paper while returning HTML that omits the real product state.

This guide walks through how that failure mode shows up and how to pull reliable pricing and availability with the Apify platform.

A Google News scraper collects public index data from Google News—headlines, links, outlets, and timestamps—for PR, competitive intel, risk monitoring, and NLP pipelines. Parsing news.google.com yourself is brittle (obfuscated markup, UI churn). A maintained Apify Actor returns stable fields and repeatable runs so you can ship alerts, dashboards, and models faster.

Instagram is a heavily protected ecosystem. Teams still need public metrics like follower counts, reel velocity, and comment signals for influencer vetting, but automated collection is actively restricted.

Running an Instagram extraction workflow in 2026 means understanding SPA behavior, GraphQL/mobile endpoints, and IP reputation controls.

This guide outlines common failure modes in basic scripts and shows a more durable path using the Apify Platform.

HTML scraping breaks often for a boring reason: layout churn. You wire up precise CSS selectors (div.product-card > span.price-wrapper > span.value), then the site ships an A/B test or a Tailwind refactor and classes become text-sm font-bold. The scraper still runs, but the data is wrong—or empty—and that quietly poisons anything downstream.

One way to reduce that fragility is to stop depending on the visual tree and read semantic metadata that many sites already embed for search engines.

The usual format is Schema.org vocabulary serialized as JSON-LD (JSON for Linked Data).