How to feed live web data into a RAG pipeline
To feed live web data into a RAG pipeline, crawl the source with Apify's Website Content Crawler, which returns clean Markdown per URL. Split that Markdown on its headings with Markdown RAG Chunker, embed the chunks, and upsert them into a vector store keyed by a stable chunk ID. Re-crawl on a schedule and re-embed only the pages that changed. The script below does all of it in about 50 lines of Python.
If an answer has to reflect a page as it is right now, skip the index entirely. RAG Web Browser takes a query or a URL and returns cleaned text straight into the prompt.
Who does what: Apify runs collect → clean → chunk. Embedding and vector storage are yours, in whichever model and database you already use.
RAG ingestion is one entry in the wider set of Apify use cases — lead lists, price tracking, and SERP data among them — so if web data feeds a different job than a vector store, that page is the better starting point.
Which Actor should you use for RAG data?
| You are ingesting | Actor | What comes back | Where it goes |
|---|---|---|---|
| A whole docs site, help centre or blog | Website Content Crawler | markdown, text, url, metadata (title, description, language) per page | Chunker, then embeddings |
| Crawled pages, PDFs or Word files to split for embedding | Markdown RAG Chunker | One chunk per section, ≤512 tokens, with embedding_text, header_path and a stable chunk_id | Embeddings, upsert by chunk_id |
| A page you need read at question time | RAG Web Browser | Markdown, plain text or HTML per request | Prompt context, nothing stored |
| A known list of URLs, no crawl config | Firecrawl | LLM-oriented Markdown | Chunker, then embeddings |
Website Content Crawler and RAG Web Browser carry the FREE pricing model: no rental fee and no per-result charge, so you pay platform usage only (api.apify.com/v2/store, checked 2026-10-20 for Website Content Crawler, 2026-09-09 for RAG Web Browser). Markdown RAG Chunker is pay per event: $0.80 per 1,000 chunks plus $0.001 per run on the Free and Bronze plans, with compute and storage included.
Website Content Crawler has three crawler modes worth using, set with crawlerType: playwright:adaptive switches between a headless browser and raw HTTP on its own and is the Console's recommended choice; playwright:firefox handles JavaScript-heavy sites; cheerio, the raw HTTP client, is the fastest and cheapest (input schema, checked 2026-10-20). When you start the crawler from code, set crawlerType explicitly: the API falls back to Firefox, not adaptive. Use raw HTTP once you confirm the pages render server-side, because that single switch is the largest cost lever on this page.
For setup screenshots and input fields, see how to scrape website content with Apify. For a wider field of RAG-adjacent tools, see the best AI data Actors roundup.
What does a RAG crawl actually cost?
Apify's own estimates, published on the Website Content Crawler page and checked 2026-09-09:
| Crawl mode | Apify's stated cost | 10,000-page site, one full crawl |
|---|---|---|
| Raw HTTP (Cheerio) | $0.20 per 1,000 pages | ~$2 |
| Headless browser | $0.50–$5 per 1,000 pages | $5–$50 |
| AI summarisation add-on | ~$2–$3 per 1,000 pages | $20–$30 on top |
Platform usage is billed in compute units, and the rate falls as the plan grows: $0.20 per CU on Free and Starter, $0.16 on Scale, $0.13 on Business (apify.com/pricing, checked 2026-09-09). The Free plan includes $5 of usage a month; Starter is $19/month subject to change · verified 2026-09-09 — $17 a month billed annually — with $19 of included usage (apify.com/pricing, checked 2026-09-09). See how compute units are counted if the number on your usage chart does not match your expectation.
Chunking with Markdown RAG Chunker adds $0.001 per page plus $0.80 per 1,000 chunks. One page of Apify's own documentation produced 13 chunks in our test run (October 2026); if your pages average about four, a 10,000-page site is roughly $42 for the first full pass ($10 of runs, $32 of chunks). Later refreshes cost nothing for unchanged pages, because the script compares hashes before calling the chunker; you pay only for pages that changed.
So a 10,000-page documentation site crawled on raw HTTP fits inside the free monthly credit. The same site on the browser crawler, re-crawled weekly, is $20–$200 a month and is what pushes you onto a paid plan. Crawling is rarely the expensive half of a RAG pipeline. Embeddings and vector hosting are billed by their own vendors and are not included in any figure above.
One conflict to be aware of before you schedule parallel crawls. Apify's two tier-1 sources disagree on Free-plan concurrency: apify.com/pricing lists 5 concurrent Actor runs, while docs.apify.com/platform/limits lists 25 (both checked 2026-09-09). Plan against 5, and confirm on your own account's Limits page before you depend on it.
How do you turn crawled pages into embeddings?
- Crawl. Run Website Content Crawler with your start URLs and a page limit. Use the
markdownfield as the body text. - Filter. Deduplicate by URL, drop pages under a word-count floor, and filter by language. Every page you drop here is an embedding you never pay for.
- Chunk by headings. Split each page on its
#,##and###headings first, then cap long sections at about 512 tokens. Keep the URL and the section path on every chunk. - Embed and upsert. Embed each chunk and upsert it under a stable chunk ID. The
sourceURL is what lets the model cite its answer. - Schedule. Put the crawl on an Apify schedule and run the incremental refresh below.
Why 512 tokens, split by headings? Chroma's chunking evaluation (July 2024) found 400-token recursive chunks reached 89.5% recall with text-embedding-3-large, and NVIDIA's 2025 chunking study recommends 256 to 512 tokens for factoid questions and 1,024 for analytical ones. Documentation questions are mostly factoid, and a chunk that stays inside one section matches one topic instead of two.
Markdown is the better default for embedding free-form pages, because it keeps headings and lists intact while dropping HTML chrome. Reach for structured JSON only when you need typed fields — price, author, publish date — as filterable metadata on the vector.
Crawl, chunk and index in one script
This script crawls a site with Website Content Crawler, splits every page with Markdown RAG Chunker, and upserts the chunks into Qdrant through LangChain. Install the latest stable versions (October 2026) and start Qdrant:
pip install apify-client==3.3.0 langchain-qdrant==1.1.0 langchain-openai==1.7.2
docker run -p 6333:6333 qdrant/qdrant
Set APIFY_TOKEN (get one from Apify Console) and OPENAI_API_KEY, then run:
import hashlib
import os
import uuid
from concurrent.futures import ThreadPoolExecutor
from apify_client import ApifyClient
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
apify = ApifyClient(os.environ["APIFY_TOKEN"])
qdrant = QdrantClient(url=os.environ.get("QDRANT_URL", "http://localhost:6333"))
# 1. Crawl the site to Markdown
crawl = apify.actor("apify/website-content-crawler").call(run_input={
"startUrls": [{"url": "https://docs.apify.com/platform/storage"}],
"crawlerType": "cheerio", # raw HTTP, cheapest; "playwright:adaptive" for JavaScript-heavy sites
"maxCrawlPages": 50,
"proxyConfiguration": {"useApifyProxy": True},
})
# 2. Chunk only new or changed pages by their headings
def page_hash(markdown): # the same value the chunker returns as document_hash
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]
def indexed_hashes(): # {url: document_hash} already in Qdrant
if not qdrant.collection_exists("web_rag"):
return {}
hashes, offset = {}, None
while True:
points, offset = qdrant.scroll("web_rag", limit=1000, offset=offset, with_payload=["metadata"])
hashes.update({p.payload["metadata"]["source"]: p.payload["metadata"]["document_hash"] for p in points})
if offset is None:
return hashes
def chunk_page(page):
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": page["markdown"],
"source_id": page["url"], # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
if run is None or run.status != "SUCCEEDED":
return []
return [(page, chunk) for chunk in apify.dataset(run.default_dataset_id).iterate_items()]
pages = list(apify.dataset(crawl.default_dataset_id).iterate_items())
crawled_urls = {page["url"] for page in pages}
indexed = indexed_hashes()
todo = [p for p in pages if p.get("markdown") and indexed.get(p["url"]) != page_hash(p["markdown"])]
docs, ids = [], []
with ThreadPoolExecutor(max_workers=4) as pool: # parallel runs: same cost, less waiting
for results in pool.map(chunk_page, todo):
for page, chunk in results:
docs.append(Document(page_content=chunk["embedding_text"], metadata={
"source": chunk["source"],
"section": chunk["header_path"],
"title": page["metadata"].get("title", ""),
"document_hash": chunk["document_hash"],
}))
# Qdrant point IDs must be UUIDs: derive one from the stable chunk_id
ids.append(str(uuid.uuid5(uuid.NAMESPACE_URL, chunk["chunk_id"])))
# 3. Embed and upsert: the same chunk_id always maps to the same point
if docs:
QdrantVectorStore.from_documents(
docs,
OpenAIEmbeddings(model="text-embedding-3-small"),
url=os.environ.get("QDRANT_URL", "http://localhost:6333"),
collection_name="web_rag",
ids=ids,
)
print(f"Crawled {len(crawled_urls)} pages, re-chunked {len(todo)}, upserted {len(docs)} chunks")
Each chunk embeds embedding_text, which is the section path followed by the content, so a chunk from "Key-value store > Basic usage" carries that context into its vector. Each page is one chunker run at the default 1 GB of memory, so one $0.001 start event plus $0.0008 per chunk on the Free plan. The script runs four at a time, below the Free plan's concurrency limit, which costs the same as running them one after another and finishes far sooner. Pages whose Markdown hash already matches the index are never sent, and max_chunks stops a runaway page before any chunk is billed.
For a Firecrawl version of the same pipeline with hybrid (keyword + vector) search and a FastAPI endpoint, see the Firecrawl + Qdrant tutorial and its tested code on GitHub.
Does Apify write to your vector store for you?
Partly. Apify publishes six official integration Actors, but adoption is concentrated in one of them. User counts and last-modified dates from api.apify.com/v2/acts/..., checked 2026-10-20:
| Integration Actor | Users | Last modified |
|---|---|---|
apify/pinecone-integration | 558 | 2026-03-19 |
apify/qdrant-integration | 61 | 2025-06-09 |
apify/pgvector-integration | 16 | 2025-04-13 |
apify/milvus-integration | 10 | 2025-08-15 |
apify/weaviate-integration | 7 | 2025-04-13 |
apify/chroma-integration | 4 | 2025-08-25 |
None is deprecated. But Pinecone is the only one Apify lists on its own integrations page, the only one modified this year, and the only one with adoption in the hundreds. If your vector store is Qdrant, Weaviate, Milvus, Chroma or pgvector, treat the integration Actor as a convenience you should be ready to replace with twenty lines of your own upsert code.
How do you keep the index fresh without re-embedding everything?
Re-embedding a whole corpus on every run is the default mistake. The script above already avoids most of it:
- Unchanged pages cost nothing. The script hashes each page's Markdown and compares it with the
document_hashalready stored in Qdrant. Matching pages are never sent to the chunker or the embedding model. - Unchanged sections keep their ID. When a page does change, sections you did not edit get the same
chunk_idas before, so the upsert overwrites them in place. - Stale chunks need one cleanup step. Delete the chunks of edited sections and of pages that disappeared from the site. Run this after the script, on every scheduled run:
from qdrant_client import QdrantClient, models
qdrant = QdrantClient(url=os.environ.get("QDRANT_URL", "http://localhost:6333"))
def match(key, value):
return models.FieldCondition(key=f"metadata.{key}", match=models.MatchValue(value=value))
# Changed pages: drop chunks left over from the previous version
for url, doc_hash in {d.metadata["source"]: d.metadata["document_hash"] for d in docs}.items():
qdrant.delete("web_rag", points_selector=models.FilterSelector(filter=models.Filter(
must=[match("source", url)],
must_not=[match("document_hash", doc_hash)],
)))
# Removed pages: drop chunks of URLs the crawl no longer finds
qdrant.delete("web_rag", points_selector=models.FilterSelector(filter=models.Filter(
must_not=[models.FieldCondition(key="metadata.source", match=models.MatchAny(any=list(crawled_urls)))],
)))
The second delete assumes the collection holds only this crawl, and that the crawl's page limit covers the whole site. If you raise maxCrawlPages later, the pages beyond the old limit simply get added.
How much this saves depends on how often your sources change. Log how many pages return chunks on your first three scheduled runs: that share is what you actually re-embed. A release-notes page will look nothing like a marketing site.
How do you use the chunks in LangChain or LlamaIndex?
The script above already produces LangChain Document objects and a Qdrant vector store. To query it from your app:
import os
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
store = QdrantVectorStore.from_existing_collection(
collection_name="web_rag",
embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
url=os.environ.get("QDRANT_URL", "http://localhost:6333"),
)
for doc in store.as_retriever(search_kwargs={"k": 4}).invoke("How do I read a record from a key-value store?"):
print(doc.metadata["source"], "|", doc.metadata["section"])
Pass the returned chunks to your LLM as context, and show source and section as the citation. Swap langchain-qdrant for langchain-pinecone, langchain-postgres (pgvector) or another LangChain vector store if you already run one; the Document objects and IDs stay the same. More patterns in the Apify LangChain integration guide.
For LlamaIndex (pip install llama-index-core==0.14.25), build a TextNode per chunk inside the chunking loop instead of a LangChain Document, and keep the chunk ID as the node ID:
from llama_index.core.schema import TextNode
def to_node(chunk: dict) -> TextNode:
return TextNode(
id_=chunk["chunk_id"],
text=chunk["embedding_text"],
metadata={"source": chunk["source"], "section": chunk["header_path"]},
)
How do you give an agent live web access instead of a static index?
Two mechanisms, and they are not interchangeable.
Standby HTTP. RAG Web Browser runs in Apify's Standby mode, so it answers plain GET requests instead of requiring a run to be started and polled:
https://rag-web-browser.apify.actor/search?token=<APIFY_API_TOKEN>&query=hello+world
It searches Google, scrapes the top results in a real browser, and returns Markdown, plain text or HTML. That is a normal HTTP call your agent can make inside a tool handler. See Standby mode for how the warm-instance billing works.
MCP. Apify's Model Context Protocol surface is a hosted endpoint, https://mcp.apify.com, not an Actor you run. Connect over Streamable HTTP with OAuth or a bearer token; for local development, npx @apify/actors-mcp-server speaks stdio. Setup is documented for Cursor, VS Code and Claude Desktop, and the Apify CLI can write the config for claude-code, cursor, vscode, vscode-insiders, codex, kiro and antigravity (docs.apify.com/platform/integrations/mcp, checked 2026-09-09). Our MCP server setup guide walks through a client config.
Static RAG answers from what you already indexed. MCP and Standby answer from what is on the page this second, and charge you per lookup. Most production systems end up with both: an index for the corpus that changes weekly, live calls for the handful of sources that change hourly.
When is Firecrawl the better tool?
If you already have the list of URLs and you just want Markdown back, Firecrawl is less work than configuring a crawler. Its free tier is 1,000 credits a month and the cheapest paid tier, Hobby, is $16 a month billed annually for 5,000 credits (firecrawl.dev/pricing, checked 2026-09-09). For a one-off ingest of a few thousand known pages, that is the shorter path and you should take it.
Apify earns its place when the job is a repeating one: full-domain discovery rather than a URL list, scheduling, proxy rotation for sites that block you, and the long tail of awkward sources that need a site-specific Actor rather than a generic crawler. The two also mix cleanly — Apify for the scheduled domain crawl, Firecrawl for the supplementary URLs someone drops in Slack. A fuller breakdown lives in Apify vs Firecrawl. Either way the chunking step is the same: pass each Firecrawl page to Markdown RAG Chunker exactly as the script above does with Website Content Crawler rows.
What to check before you embed
- Deduplicate URLs and near-identical bodies.
- Drop pages under a word-count floor; they are usually errors or placeholders.
- Confirm the
markdownfield is populated, not justhtml. - Language-filter if the model serves one locale.
- Keep
source, the section path and crawl time on every chunk, for citations and for freshness SLAs. - Check robots.txt and the site's terms; prefer public documentation you have the right to ingest. Is Apify legal covers the ground rules.
- Minimise personal data in anything bound for training or analytics (GDPR, CCPA).
If you need the hosted chatbot layer on top of the corpus rather than building retrieval yourself, compare Chatbase and the alternatives in Chatbase vs Botpress, eesel AI and Intercom before you write any of this code.
Start the crawl
The realistic first move is small: point Website Content Crawler at one documentation domain on the raw HTTP client, look at the markdown field of twenty rows, and chunk them before you spend anything on embeddings. On Apify's stated $0.20 per 1,000 pages the crawl costs cents, twenty chunker runs cost a few cents more, and the Free plan's $5 monthly credit covers both with room left over. Then run the script above on the whole site.
Run Website Content Crawler → | Chunk pages with Markdown RAG Chunker →
Crawl the sources with Apify's Website Content Crawler, which returns clean Markdown per page with navigation and boilerplate stripped. Split that Markdown on its headings into chunks of up to about 512 tokens (Markdown RAG Chunker on Apify does this and returns a stable chunk ID), embed each chunk, and upsert it into a vector store keyed by that ID with the source URL in metadata. Then schedule the crawl and re-embed only the pages that changed. For answers that must reflect a page right now, call RAG Web Browser at query time instead of reading from the index.
Website Content Crawler for anything that looks like a site: docs, help centres, blogs. It crawls a domain and outputs markdown, text, url, title and metadata per page, with nav, headers, footers, ads and cookie banners removed. Chunk its output with Markdown RAG Chunker, which splits each page by headings and adds section paths and stable IDs. Add RAG Web Browser when an answer needs a live page rather than an indexed copy. Website Content Crawler and RAG Web Browser carry Apify's FREE pricing model, so you pay only platform usage; Markdown RAG Chunker costs $0.80 per 1,000 chunks plus $0.001 per run.
Apify's own estimates on the Website Content Crawler page are $0.20 per 1,000 pages on the raw HTTP client and $0.50 to $5 per 1,000 pages with a headless browser, with AI summarisation adding roughly $2 to $3 per 1,000 pages (checked 2026-09-09). A 10,000-page docs site is therefore about $2 per full crawl on raw HTTP. Platform usage is $0.20 per compute unit on the Free and Starter plans, dropping to $0.16 on Scale and $0.13 on Business; the Free plan includes $5 of usage a month and Starter is $19 a month. Chunking with Markdown RAG Chunker adds $0.001 per page and $0.80 per 1,000 chunks. Embeddings and vector hosting are billed separately by their own vendors.
Connect to Apify's hosted MCP endpoint, https://mcp.apify.com, over Streamable HTTP with OAuth or a bearer token. For local development, npx @apify/actors-mcp-server speaks stdio. Setup is documented for Cursor, VS Code and Claude Desktop, and the Apify CLI can write the client config for you.
Split each page on its headings first, then cap long sections at about 512 tokens, and keep the source URL and section path on every chunk. Chroma's 2024 evaluation found 400-token recursive chunks reached 89.5% recall, and NVIDIA's 2025 study recommends 256 to 512 tokens for factoid questions. Markdown RAG Chunker on Apify does the heading split, counts tokens exactly, and returns a stable chunk_id to upsert by.
Read the Website Content Crawler dataset with apify-client, chunk each page with Markdown RAG Chunker, and turn each chunk into a LangChain Document whose page_content is embedding_text and whose metadata holds the source URL and section path. Upsert with any LangChain vector store, such as langchain-qdrant, using the chunk ID as the document ID, then query it with as_retriever().
It can, for six stores. Apify publishes integration Actors for Pinecone, Qdrant, pgvector, Milvus, Weaviate and Chroma, none of them deprecated. Adoption is lopsided: Pinecone has 558 users and was last modified in March 2026, while the other five sit between 4 and 61 users and were last touched in 2025 (api.apify.com, checked 2026-10-20). Pinecone is also the only one Apify documents on its own integrations page. For the others, plan to write your own upsert.
If you already have the URLs and just want Markdown back once, yes, Firecrawl is less setup. Its free tier is 1,000 credits a month and Hobby is $16 a month billed annually for 5,000 credits (checked 2026-09-09). Apify is the better fit when the job repeats: full-domain crawling rather than a URL list, scheduling, proxy rotation, and site-specific Actors for sources a generic crawler cannot read. Plenty of pipelines use both.
Markdown for the body text. It keeps headings and list structure while dropping HTML chrome, which produces cleaner chunk boundaries. Use structured JSON when you need typed fields such as price, author, publish date or category as filterable metadata on the vector, for example to restrict retrieval to recent or in-stock items. A common shape is Markdown as the embedded text with a few JSON fields alongside it as metadata.
Match the schedule to how fast the source actually changes, and keep the cost down with a content hash rather than by crawling less often. Compare each page's Markdown hash with the stored document_hash so unchanged pages are never re-chunked or re-embedded, and upsert by the stable chunk ID so unchanged sections are overwritten rather than duplicated. Documentation and marketing sites usually justify weekly; release notes, status pages and pricing pages justify daily. Log how many pages return chunks on your first few runs and set the interval from that.
Common mistakes and fixes
Retrieved chunks are navigation and cookie banners, not body text.
Use the `markdown` field, not `html`. Website Content Crawler already strips nav, headers, footers, ads and cookie warnings; if boilerplate still survives, drop pages under a minimum word count before you chunk.
The first sync costs more in embeddings than in crawling.
That is normal. Crawling 10,000 pages on the raw HTTP client is about $2 of Apify platform usage; embedding those same pages is billed by your model provider. Cap crawl depth, dedupe URLs, and embed with a small model until the retrieval quality is proven.
Page titles come back empty.
Website Content Crawler stores the title under `metadata.title`, not at the top level of each row. Read `page["metadata"]["title"]`.
A scheduled run re-chunks nothing.
Every page's Markdown hash matches the document_hash already in Qdrant, so nothing changed and nothing was sent to the chunker. Those pages cost $0. To force a full re-index (for example after changing embedding models), index into a new collection.
Qdrant rejects the point ID.
Qdrant IDs must be unsigned integers or UUIDs. Derive one from the chunk ID with `uuid.uuid5(uuid.NAMESPACE_URL, chunk_id)`, as the script on this page does.