Best Apify Actors for AI and RAG 2026

Contains affiliate links. Disclosure.

RAG pipelines fail on data quality long before they fail on the model. The ingestion layer has to strip boilerplate, preserve semantic links, and emit chunk-ready Markdown, not a wall of HTML with nav, cookie banners, and tracking pixels that shred your token budget. Three Apify Actors cover almost every ingestion shape a team actually needs: Website Content Crawler for bulk corpora, RAG Web Browser for live fetch at inference time, and the Apify MCP Server so agents can reach every other Store Actor through one tool schema.

Top AI & RAG Actors comparison

Actor	Pick this if…	Output	Cost signal
Website Content Crawler	You're indexing a whole domain (docs, KB, blog) on a schedule	Markdown + text + HTML + page metadata per URL	$0.20/1K raw HTTP · $0.50–$5/1K headless (128K users, 4.6★)
RAG Web Browser	Your agent needs post-training-cutoff info at query time	Google SERP + scraped Markdown bodies in one JSON response	Pay-per-usage (platform compute), ~sub-cent per query (89K users)
Apify MCP Server	Your MCP client (Claude, Cursor, ChatGPT) should pick the right scraper itself	MCP tool calls → Actor runs	Free server, pay per Actor run

1. Website Content Crawler: production RAG ingestion

Pick this if your monthly ingest is more than a few hundred pages and a daily/weekly index is acceptable. Otherwise use RAG Web Browser.

• Emits Markdown and text and the cleaned HTML per URL, plus url, crawl.loadedUrl, metadata.title, metadata.description, metadata.openGraph, and metadata.jsonLd. That is already a chunk record, so you don't need a separate parser step in LangChain or LlamaIndex.
• Strips nav, footer, header, cookie banners, and repeated side columns on every page, so embeddings aren't poisoned by the same 800 tokens of chrome on every chunk. Internal <a> hrefs are kept as Markdown links, preserving graph signal for link-aware retrievers.
• Cost anchor: $0.20 per 1,000 pages on the Cheerio (raw HTTP) crawler, $0.50–$5 per 1,000 on Playwright/Firefox. For reference, Firecrawl's published /scrape rate sits around $1–$2 per 1,000 pages on JS-rendered sites, so the Cheerio tier is 5 to 10 times cheaper when the site isn't React/Vue/Next.
• Use maxCrawlPages, includeUrlGlobs, and a sitemap seed. A single un-scoped crawl of a marketing site with infinite facet URLs is where bills come from.

2. RAG Web Browser: live agent browsing

Pick this if your agent answers questions where the answer changes between crawls: pricing, release notes, news, score lines, stock prices. Otherwise Website Content Crawler + a vector DB is cheaper and lower-latency.

One request returns a Google SERP and the scraped Markdown bodies of the top N results, shaped to drop straight into a context window (no dataset round-trip, no second fetch). Good default for a web_search tool in an agent: hand the user's query in, get back [{url, title, markdown}].

3. Apify MCP Server: one endpoint, every Actor

Pick this if you're building an agent that routes scraping tasks across many target sites and you don't want to write a tool schema per Actor. The server exposes Actor discovery + run as MCP tools. Claude or Cursor picks compass/crawler-google-places for Maps queries, apidojo/tweet-scraper for X, apify/website-content-crawler for arbitrary docs, and you don't wire it.

Scheduled sync pattern (the only RAG diagram you need)

1. Apify Schedule → Website Content Crawler on your docs domain, scoped with includeUrlGlobs.
2. Native integration (Pinecone, Qdrant, Weaviate, Chroma) upserts each dataset row as a vector, keyed by url. New pages get added, changed pages overwrite the prior chunks, deleted pages can be purged with a tombstone pass.
3. Your assistant retrieves against the live index. No glue service.

For JS-heavy targets (React docs, Notion-hosted KBs, single-page apps), force Playwright + Firefox in the crawler input. Cheerio will return the empty shell and your vectors will be garbage.

Build smarter AI

Don't feed an LLM messy HTML. Start with clean Markdown and you skip half the retrieval tuning.

Start your first RAG crawl for free on Apify →

Frequently Asked Questions

Website Content Crawler for bulk indexing, RAG Web Browser for live agent lookups. WCC converts any domain into clean Markdown with page metadata (url, title, description, OpenGraph, JSON-LD) that is already a chunk record for LangChain or LlamaIndex.

Yes. Pinecone, Qdrant, Weaviate, Chroma, and Milvus all have native Apify integrations. Set them up as run-finished webhooks on the crawler; rows land as upserts keyed by URL with no middleware service.

Put the Website Content Crawler on an Apify Schedule (daily for docs, hourly for pricing or release notes) and wire the vector DB integration to upsert by URL. Stale chunks get overwritten on the next run without manual intervention.

General scrapers return structured fields (price, rating, SKU). WCC returns the main article body as Markdown with nav, footer, and cookie-banner DOM stripped, plus page metadata. It is optimized for embedding, not for row-based analytics.

Apify's raw HTTP crawler is $0.20 per 1,000 pages and the headless tier is $0.50–$5 per 1,000 depending on memory and render time. Firecrawl's /scrape endpoint is typically $1–$2 per 1,000 pages on JS-rendered sites, so for static docs the Cheerio tier is materially cheaper; for heavy JS the two are in the same band.