Firecrawl for RAG: Build a Knowledge Base from Any Website 2026
To build a RAG knowledge base from a website with Firecrawl, you run six steps: map the site's URLs, crawl the pages to Markdown, split the Markdown into chunks, embed each chunk, store the vectors with their source URL, and retrieve the closest chunks for each question. Firecrawl handles the first two steps. Chunking is the step that most affects answer quality, and it is the one Firecrawl leaves to you.
This guide gives you a tested Python pipeline for all six steps with the current Firecrawl SDK, plus two ways to chunk: a free local splitter and a header-aware chunker that adds section paths and stable IDs.
What is a RAG knowledge base?
A RAG (retrieval-augmented generation) knowledge base is a vector index of your documents that an LLM searches before it answers. Instead of relying on what the model memorised, your app finds the passages most similar to the question and puts them in the prompt, so answers cite your current content.
Building one from a website has three hard parts:
- Getting clean text. Raw HTML is full of navigation, cookie banners and scripts. Firecrawl renders the page and returns the main content as Markdown.
- Cutting it into retrievable pieces. Chunks that cross section boundaries, or lose track of which section they came from, retrieve poorly.
- Keeping it current. Sites change. Without stable chunk IDs, every re-crawl duplicates the index.
The pipeline at a glance
| Step | Tool in this guide | Output |
|---|---|---|
| 1. Discover | Firecrawl map | List of URLs on the site |
| 2. Extract | Firecrawl crawl | One Markdown document per page |
| 3. Chunk | LangChain splitter, or Markdown RAG Chunker | Chunks of up to 512 tokens with metadata |
| 4. Embed | OpenAI text-embedding-3-small (or any embedding model) | One vector per chunk |
| 5. Store | Chroma via langchain-chroma | Persistent vector index |
| 6. Retrieve | LangChain retriever with a metadata filter | Top chunks for the LLM prompt |
Install the packages:
pip install firecrawl-py==4.53.2 apify-client==3.3.0 langchain-text-splitters==1.1.3 \
tiktoken==0.14.0 langchain-chroma==1.1.0 langchain-openai==1.7.2
These are the latest stable versions as of October 2026; the code below targets them.
Step 1: Map the site
map returns the URLs Firecrawl can find on a site, which tells you what a crawl will cost and which paths to include before you spend credits on it.
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
site = firecrawl.map("https://docs.example.com", limit=500)
for link in site.links[:10]:
print(link.url, "-", link.title)
print(f"{len(site.links)} URLs found")
Step 2: Crawl pages to Markdown
from firecrawl import ScrapeOptions
job = firecrawl.crawl(
"https://docs.example.com",
limit=100,
include_paths=["^/guides/.*"], # regex on the URL path
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True),
)
pages = [p for p in job.data if p.markdown]
print(f"Crawled {len(pages)} pages")
# p.markdown is the page text; p.metadata.source_url is its URL
crawl waits for the job to finish and returns a CrawlJob whose data is a list of Document objects. Each page's text is p.markdown and its URL is p.metadata.source_url.
A basic crawl costs 1 Firecrawl credit per page, and the free plan includes 1,000 credits a month (Firecrawl pricing), so a 100-page docs site fits comfortably.
Step 3: Chunk the Markdown
Embedding a whole page as one vector blurs everything on it together. Split each page into chunks of a few hundred tokens, and keep the source URL with every chunk.
What chunk size should you use?
| Source | Finding |
|---|---|
| Chroma, Evaluating Chunking Strategies (July 2024) | RecursiveCharacterTextSplitter at 400 tokens with no overlap reached 89.5% recall with text-embedding-3-large, slightly above the same size with 200-token overlap (88.1%) |
| NVIDIA, Finding the Best Chunking Strategy (June 2025) | 256–512 tokens for factoid questions and varied collections, 1,024 tokens for complex analytical questions; page-level chunking was the most consistent default |
For a docs or help-centre knowledge base, where most questions are "how do I…" lookups, 512 tokens with a small overlap is a sound default. Measure in tokens, not characters: 1,200 characters is roughly 300 tokens of English prose, but more for code and far more for Chinese or Japanese.
Option A: Split locally with LangChain (free)
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
# from_tiktoken_encoder measures chunk_size in tokens; the plain
# constructor would measure characters.
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=512,
chunk_overlap=50,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
docs, ids = [], []
for page in pages:
url = page.metadata.source_url
for i, text in enumerate(splitter.split_text(page.markdown)):
docs.append(Document(page_content=text, metadata={"source_url": url, "chunk_index": i}))
ids.append(f"{url}#{i}")
The separators make the splitter prefer heading breaks, but it does not guarantee them. A chunk can still start in one section and end in the next, nothing records which section it came from, and the IDs are positions: if a paragraph is added near the top of a page, every later chunk on that page gets a new ID.
Option B: Header-aware chunks with Markdown RAG Chunker
Markdown RAG Chunker is an Apify Actor that fixes those three gaps. It splits each page on its #, ## and ### headings first, so a chunk never straddles two sections, then re-splits only the sections longer than 512 tokens (counted with tiktoken, 10% overlap). This is the two-step pattern LangChain's own Markdown splitter docs recommend, run for you. Every chunk carries:
| Field | Use |
|---|---|
embedding_text | Section path + content (Guides > Authentication\n\n...). Embed this. |
header_path | Section path for metadata filters and citations |
chunk_id | Stable upsert key: unchanged on re-crawl if the section did not change |
source, document_hash | Find and delete a page's stale chunks after it changes |
token_count | Exact count, so no chunk exceeds your embedding model's limit |
import hashlib
from concurrent.futures import ThreadPoolExecutor
from apify_client import ApifyClient
from langchain_core.documents import Document
apify = ApifyClient("YOUR_APIFY_TOKEN")
def page_hash(markdown: str) -> str:
# The same value the chunker returns as document_hash for markdown_text input
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]
def chunk_page(page) -> list[dict]:
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": page.markdown,
"source_id": page.metadata.source_url, # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
if run is None or run.status != "SUCCEEDED":
print(f"Skipped {page.metadata.source_url}: {run.status_message if run else 'run did not start'}")
return []
return list(apify.dataset(run.default_dataset_id).iterate_items())
indexed = {} # {source_url: document_hash} already stored; filled on re-runs (Step 6)
todo = [p for p in pages if indexed.get(p.metadata.source_url) != page_hash(p.markdown)]
docs, ids = [], []
with ThreadPoolExecutor(max_workers=4) as pool: # parallel runs: same cost, less waiting
for items in pool.map(chunk_page, todo):
for c in items:
docs.append(Document(page_content=c["embedding_text"], metadata={
"source_url": c["source"],
"section_path": c["header_path"],
"document_hash": c["document_hash"],
"chunk_index": c["chunk_index"],
}))
ids.append(c["chunk_id"])
Each run chunks one document. Pricing is $0.80 per 1,000 chunks plus $0.001 per run on Apify's free plan, compute included. As a worked example, a first pass over 100 pages that produce 400 chunks costs 400 × $0.0008 + 100 × $0.001 = $0.42. After that you pay only for pages whose Markdown changed: unchanged pages never reach the Actor, so they cost $0.
Which option? Use Option A for a one-off prototype. Use Option B when the knowledge base is re-crawled on a schedule, when you want section paths in your citations, or when the content is not in English.
Step 4: Embed and store in Chroma
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
vectorstore = Chroma(
collection_name="website_kb",
embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"),
persist_directory="./chroma_db",
)
if docs:
vectorstore.add_documents(docs, ids=ids)
Only chunks from new or changed pages are embedded, so re-runs don't pay OpenAI for pages you already indexed. Passing ids matters: Chroma overwrites a document with the same ID instead of adding a duplicate. With Option B's chunk_id, unchanged sections keep their ID across crawls.
Step 5: Retrieve with metadata filters
retriever = vectorstore.as_retriever(
search_kwargs={"k": 4, "filter": {"source_url": "https://docs.example.com/guides/auth"}}
)
for doc in retriever.invoke("How do I rotate an API key?"):
print(doc.metadata["source_url"], doc.metadata.get("section_path", ""))
print(doc.page_content[:200], "\n")
Drop the filter to search the whole knowledge base. Pass the returned chunks to your LLM as context, and show source_url (plus section_path from Option B) as the citation.
Step 6: Keep the knowledge base fresh
- Re-crawl on a schedule that matches how often the site changes: daily for changelogs, weekly for reference docs.
- Upsert by ID. Re-adding documents with the same IDs updates them in place.
- Skip unchanged pages. Before Step 3, load the hashes already in the store so unchanged pages are never sent to the chunker or the embedding model:
stored = vectorstore.get(include=["metadatas"])["metadatas"]
indexed = {m["source_url"]: m["document_hash"] for m in stored if "document_hash" in m}
- Delete what changed. After Step 4, remove each re-chunked page's old chunks:
for url, doc_hash in {d.metadata["source_url"]: d.metadata["document_hash"] for d in docs}.items():
vectorstore.delete(where={"$and": [
{"source_url": url},
{"document_hash": {"$ne": doc_hash}},
]})
A nightly refresh then costs only the changed pages: one $0.001 run plus $0.0008 per chunk for each, and $0 for the rest.
The LangChain loader shortcut
If you prefer LangChain's document loaders, use Firecrawl's official LangChain package:
# pip install langchain-firecrawl==0.1.0
from firecrawl import ScrapeOptions
from langchain_firecrawl import FirecrawlLoader
loader = FirecrawlLoader(
url="https://docs.example.com",
mode="crawl",
params={"limit": 100, "scrape_options": ScrapeOptions(formats=["markdown"], only_main_content=True)},
)
pages = loader.load() # page_content is Markdown; metadata includes source_url
The loader hands you whole pages as LangChain Documents, so you still need Step 3. In that code, read page.page_content instead of page.markdown and page.metadata["source_url"] instead of page.metadata.source_url, and keep copying only the fields you need into each chunk's metadata.
Vector database options
| Database | Hosting | Good fit |
|---|---|---|
| Chroma | Embedded, self-hosted or cloud | Prototypes and single-machine apps |
| Qdrant | Self-hosted or cloud | Metadata filtering and built-in hybrid (keyword + vector) search |
| Pinecone | Managed only | Teams that want no infrastructure |
| pgvector | Inside Postgres | Apps that already run Postgres |
All four have LangChain integrations, and the chunk IDs and metadata above work the same way in each. For a Qdrant version of this pipeline with hybrid search and a FastAPI endpoint, see Firecrawl + Qdrant vector search. The complete, runnable project is on GitHub: firecrawl-qdrant-vector-search.
Firecrawl or Apify for the crawl step?
| Firecrawl | Apify Website Content Crawler | |
|---|---|---|
| Output | Clean Markdown per page | Clean Markdown per page |
| Pricing | Credits: 1 per basic page, 1,000 free a month | Apify platform usage |
| Scheduling | You run it (cron, webhook) | Built-in schedules and integrations |
| Chunking | Left to you | Chain Markdown RAG Chunker on the same platform |
| Best for | Quick API-first prototypes | Scheduled pipelines and sites that need logins or heavy anti-bot handling |
Both produce Markdown that the chunking and storage steps above accept unchanged. For the Apify-native walkthrough, see how to feed live web data into a RAG pipeline. For evaluation, reranking and monitoring once the prototype works, read RAG production architecture.
Crawl 20 to 50 pages first and write ten questions you know the answers to. If the right page is not in the top four results for most of them, fix chunking before you touch the prompt.
Try Markdown RAG Chunker on Apify → | Get the full example on GitHub
Map the site's URLs, crawl the pages to Markdown with a tool like Firecrawl, split the Markdown into chunks of a few hundred tokens, embed each chunk, store the vectors with their source URL in a vector database such as Chroma or Qdrant, and retrieve the closest chunks for each question.
Start with 512 tokens and a small overlap. Chroma's 2024 evaluation found 400-token recursive chunks reached 89.5% recall, and NVIDIA's 2025 study recommends 256 to 512 tokens for factoid questions and 1,024 for analytical ones. Measure chunk size in tokens, not characters.
No. Firecrawl returns clean Markdown per page; splitting it into chunks is up to you. Use a LangChain text splitter, or Markdown RAG Chunker on Apify for header-aware chunks with section paths and stable IDs.
A basic crawl costs 1 credit per page. The free plan includes 1,000 credits a month, and the Hobby plan is $16 a month for 5,000 credits when billed annually.
Any of them. Firecrawl outputs Markdown, so Chroma, Qdrant, Pinecone and pgvector all work. Store the source URL with every chunk for citations and filtering.
Re-crawl on a schedule, upsert chunks by a stable ID so unchanged sections are overwritten rather than duplicated, and delete a page's old chunks when its content hash changes. Compare each page's Markdown hash with the stored document_hash first, so unchanged pages are never re-chunked or re-embedded.
