Best Apify Actors for RAG 2026
Two Actors cover almost all RAG ingestion on Apify. Website Content Crawler turns a whole domain into clean Markdown for about $0.20 per 1,000 pages on its raw HTTP crawler. RAG Web Browser searches Google and scrapes the results at request time, for agents that need something newer than your last crawl. Choose by one question: does the answer change between crawls? This page sticks to RAG; for picks across every other category, the best Apify Actors list is the wider starting point.
What changed, checked 2026-09-09. Earlier versions of this page listed the Apify MCP Server as a third Actor. It is not an Actor any more. The old Store URL apify.com/apify/actors-mcp-server now answers with a 308 redirect to mcp.apify.com, and the capability lives at that hosted endpoint mcp.apify.com plus a local npm package, documented under Apify's MCP integration docs. If MCP is what you came for, the MCP server Actors guide covers the transport, auth and tool-pinning properly. It is not a pick on this page, because it is not something you run.
Which Apify Actor should I use for RAG ingestion?
| Actor | Use it when | Output per page | Platform cost |
|---|---|---|---|
| Website Content Crawler | You are indexing a domain (docs, KB, blog) on a schedule | markdown, text, html, plus url, crawl.loadedUrl and metadata.title / description / openGraph | ~$0.20 per 1,000 pages raw HTTP; $0.50-$5 per 1,000 headless |
| RAG Web Browser | The answer changes between crawls: pricing, release notes, news | Google results plus the scraped bodies, in Markdown, text or HTML | Compute units only, no published per-result price |
Both are first-party Apify Actors on pay-per-usage, so you are billed for platform consumption rather than a rental fee. As of 2026-09-09 their Store pages show 155,185 total users and 4.61 out of 5 for the crawler, and 169,448 users and 4.71 out of 5 for RAG Web Browser.
What does Website Content Crawler actually cost?
Apify publishes two rates on the Actor's own page: about $0.20 per 1,000 pages on the raw HTTP crawler, and $0.50 to $5 per 1,000 with a headless browser, depending on the site. The spread inside the headless band is render time, so a heavy SPA sits near the top of it.
The single input that decides which rate you pay is crawlerType. Set it to cheerio for raw HTTP, playwright:firefox for a real browser, or playwright:adaptive to let the Actor switch per page. Leaving a browser on for static documentation is the most common way to pay up to 25 times more for identical Markdown.
On the Free plan you get $5 of included usage per month, which is roughly 25,000 pages at the Cheerio rate before proxy and storage consumption. Starter is $19 per month with $19 of included usage ($17 per month billed annually), verified on Apify's pricing page on 2026-09-09.
$0.20 per 1,000 pages subject to change · verified 2026-09-09One caveat on parallelism, because Apify's own two sources disagree. apify.com/pricing lists 5 maximum concurrent runs on the Free plan; docs.apify.com/platform/limits lists 25 for the same plan. Both were read on 2026-09-09. Plan a large first crawl against the lower number and you will not be surprised either way.
The Website Content Crawler walkthrough covers input tuning and where a bill escapes in more detail.
When is RAG Web Browser the better choice?
Use it when a scheduled index would already be stale by the time the question is asked. One call takes a query, runs the search, scrapes the top organic results and returns the bodies in your chosen outputFormats, so an agent gets a usable answer without a dataset round-trip.
Its scrapingTool input defaults to browser-playwright. The alternative, raw-http, is documented as about twice as fast, and it fails on JavaScript-rendered pages. For a web_search tool in an agent, start on the default and drop to raw-http only for sources you know serve server-rendered HTML.
For bulk work it is the wrong tool. Crawling 50,000 documentation pages one query at a time through a browser costs more and takes longer than one scheduled crawl into a vector index.
Is Apify cheaper than Firecrawl for RAG crawls?
For static pages, yes, by a wide margin. Beyond that it depends on how hard your target is to render, and there is a band where Firecrawl wins outright.
Firecrawl bills 1 credit per page for a standard scrape and does not charge extra for JavaScript rendering; the documented surcharge is 4 extra credits per page for its JSON, Question and Highlight formats. Working its published plans out per 1,000 pages, on annual billing (firecrawl.dev/pricing, read 2026-09-09):
| Firecrawl plan | Price per month (annual) | Credits per month | Cost per 1,000 pages |
|---|---|---|---|
| Free | $0 | 1,000 | n/a |
| Hobby | $16 ($19 monthly) | 5,000 | $3.20 ($3.80 monthly) |
| Standard | $83 ($99 monthly) | 100,000 | $0.83 ($0.99 monthly) |
| Growth | $333 ($399 monthly) | 500,000 | $0.67 ($0.80 monthly) |
| Scale | $599 ($749 monthly) | 1,000,000 | $0.60 ($0.75 monthly) |
Against Apify's $0.20 raw-HTTP rate, no Firecrawl tier comes close, and the cheapest Firecrawl per-page rate needs a $599-per-month commitment on annual billing to reach.
Now the other direction. Because Firecrawl's credit is flat, a page that costs Apify $3 in headless render time still costs Firecrawl one credit. Anywhere your headless crawl settles above roughly $0.83 per 1,000 pages, Firecrawl's Standard plan is the cheaper way to fetch it, and the harder the target, the wider that gap gets. If your corpus is mostly JavaScript-heavy applications rather than server-rendered docs, price both before committing. The honest split is: Apify wins static bulk ingestion on price and wins on everything downstream of the crawl; Firecrawl wins predictability on difficult renders.
Can Apify write straight into my vector database?
Yes, through separate integration Actors that run when your crawl finishes. Set one up as a run-finished webhook and dataset rows land as upserts keyed by URL with no service of your own in between.
Read the adoption numbers before you build on this. Verified on each Actor's Store page on 2026-09-09:
| Integration Actor | Total users | Rating |
|---|---|---|
| apify/pinecone-integration | 558 | 3.21 / 5 |
| apify/qdrant-integration | 61 | 5.00 / 5 |
| apify/milvus-integration | 10 | 4.50 / 5 |
| apify/weaviate-integration | 7 | 4.99 / 5 |
| apify/chroma-integration | 4 | 4.50 / 5 |
Only Pinecone has real usage, and it carries the worst rating of the five. The other four are effectively unproven: four to sixty-one users is not a load-bearing sample. Apify's own integrations directory lists Milvus, Pinecone and Qdrant under vector databases and does not list Weaviate or Chroma, even though both Actors exist and are not deprecated.
The practical read: if you already run Pinecone, try the integration and check the recent reviews first. Otherwise, pull the finished dataset with the API and write to your vector store from your own code. It is thirty lines, and you own the failure mode. The RAG pipeline guide walks through both routes, and the LangChain integration loads datasets directly if you are already in that framework.
How do I keep a RAG index from going stale?
- Put the crawler on an Apify Schedule. Daily is right for documentation; hourly is right for pricing and release notes.
- Scope the crawl with
includeUrlGlobs(for example/docs/**) and seed fromsitemap.xml, so a schedule cannot quietly turn into an unbounded crawl. - Upsert by
url. Changed pages overwrite their previous chunks instead of adding a second, contradictory copy. - Run a periodic tombstone pass for deleted pages. Nothing in the integration Actors removes vectors for URLs that stopped existing, and stale deletions are what produce confidently wrong citations.
- Watch
crawl.loadedTimein the dataset. If it stops moving, the schedule failed and your index is frozen at a date nobody noticed.
Run the first crawl on cheerio against 100 pages and read the Markdown before you scale. The Free plan's $5 of monthly usage covers roughly 25,000 raw-HTTP pages, which is enough to prove the ingestion quality without a card.
Website Content Crawler for bulk indexing, RAG Web Browser for live agent lookups. The crawler converts a domain into clean Markdown with per-page metadata (url, crawl.loadedUrl, metadata.title, description, openGraph) that is already close to a chunk record for LangChain or LlamaIndex.
About $0.20 per 1,000 pages on the raw HTTP (cheerio) crawler, and $0.50 to $5 per 1,000 with a headless browser depending on render time, per the Actor's Store page on 2026-09-09. The crawlerType input decides which rate you pay.
No. As of 2026-09-09 the old Store URL apify.com/apify/actors-mcp-server returns a 308 redirect to mcp.apify.com. Apify now runs MCP as a hosted endpoint there, with a local @apify/actors-mcp-server npm package for clients that need stdio. There is no Actor to run.
Pinecone, Qdrant, Milvus, Weaviate and Chroma all have integration Actors, but adoption is thin: 558 users on Pinecone, 61 on Qdrant, 10 on Milvus, 7 on Weaviate and 4 on Chroma as of 2026-09-09. Apify's integrations directory lists only Milvus, Pinecone and Qdrant.
Apify's raw HTTP crawler at $0.20 per 1,000 pages undercuts every Firecrawl tier, whose best rate is $0.60 per 1,000 on the $599 Scale plan. But Firecrawl charges one flat credit per page regardless of rendering, so on JavaScript-heavy sites where Apify's headless tier climbs above roughly $0.83 per 1,000, Firecrawl's Standard plan is cheaper.
Common mistakes and fixes
Crawler output still contains nav, footer, or cookie-banner text.
Set `removeCookieWarnings: true` in the input. If the site is React, Vue or another SPA, switch `crawlerType` from `cheerio` to `playwright:firefox`. Cheerio only ever sees the empty shell, so there is no main content for the boilerplate stripper to find.
Embeddings drift and answers cite pages that no longer say that.
Re-crawl on an Apify Schedule and upsert keyed by `url` so changed pages overwrite their old chunks instead of adding duplicates. Deleted pages need a separate tombstone pass; none of the vector-database integration Actors purge for you.
A crawl that should have been cheap cost several dollars.
Check `crawlerType` first. Headless runs cost $0.50-$5 per 1,000 pages against $0.20 on raw HTTP, so an unnecessary browser is a 2.5x to 25x markup. Then cap `maxCrawlPages` and scope with `includeUrlGlobs`; an unscoped crawl of a site with faceted URLs has no natural end.