Skip to main content

Firecrawl for RAG: Build a Knowledge Base from Any Website 2026

· 12 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

To build a RAG knowledge base from a website with Firecrawl, you run six steps: map the site's URLs, crawl the pages to Markdown, split the Markdown into chunks, embed each chunk, store the vectors with their source URL, and retrieve the closest chunks for each question. Firecrawl handles the first two steps. Chunking is the step that most affects answer quality, and it is the one Firecrawl leaves to you.

This guide gives you a tested Python pipeline for all six steps with the current Firecrawl SDK, plus two ways to chunk: a free local splitter and a header-aware chunker that adds section paths and stable IDs.

What is a RAG knowledge base?​

A RAG (retrieval-augmented generation) knowledge base is a vector index of your documents that an LLM searches before it answers. Instead of relying on what the model memorised, your app finds the passages most similar to the question and puts them in the prompt, so answers cite your current content.

Building one from a website has three hard parts:

  1. Getting clean text. Raw HTML is full of navigation, cookie banners and scripts. Firecrawl renders the page and returns the main content as Markdown.
  2. Cutting it into retrievable pieces. Chunks that cross section boundaries, or lose track of which section they came from, retrieve poorly.
  3. Keeping it current. Sites change. Without stable chunk IDs, every re-crawl duplicates the index.

The pipeline at a glance​

StepTool in this guideOutput
1. DiscoverFirecrawl mapList of URLs on the site
2. ExtractFirecrawl crawlOne Markdown document per page
3. ChunkLangChain splitter, or Markdown RAG ChunkerChunks of up to 512 tokens with metadata
4. EmbedOpenAI text-embedding-3-small (or any embedding model)One vector per chunk
5. StoreChroma via langchain-chromaPersistent vector index
6. RetrieveLangChain retriever with a metadata filterTop chunks for the LLM prompt

Install the packages:

pip install firecrawl-py==4.53.2 apify-client==3.3.0 langchain-text-splitters==1.1.3 \
tiktoken==0.14.0 langchain-chroma==1.1.0 langchain-openai==1.7.2

These are the latest stable versions as of October 2026; the code below targets them.

Step 1: Map the site​

map returns the URLs Firecrawl can find on a site, which tells you what a crawl will cost and which paths to include before you spend credits on it.

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

site = firecrawl.map("https://docs.example.com", limit=500)
for link in site.links[:10]:
print(link.url, "-", link.title)
print(f"{len(site.links)} URLs found")

Step 2: Crawl pages to Markdown​

from firecrawl import ScrapeOptions

job = firecrawl.crawl(
"https://docs.example.com",
limit=100,
include_paths=["^/guides/.*"], # regex on the URL path
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True),
)

pages = [p for p in job.data if p.markdown]
print(f"Crawled {len(pages)} pages")
# p.markdown is the page text; p.metadata.source_url is its URL

crawl waits for the job to finish and returns a CrawlJob whose data is a list of Document objects. Each page's text is p.markdown and its URL is p.metadata.source_url.

A basic crawl costs 1 Firecrawl credit per page, and the free plan includes 1,000 credits a month (Firecrawl pricing), so a 100-page docs site fits comfortably.

Step 3: Chunk the Markdown​

Embedding a whole page as one vector blurs everything on it together. Split each page into chunks of a few hundred tokens, and keep the source URL with every chunk.

What chunk size should you use?​

SourceFinding
Chroma, Evaluating Chunking Strategies (July 2024)RecursiveCharacterTextSplitter at 400 tokens with no overlap reached 89.5% recall with text-embedding-3-large, slightly above the same size with 200-token overlap (88.1%)
NVIDIA, Finding the Best Chunking Strategy (June 2025)256–512 tokens for factoid questions and varied collections, 1,024 tokens for complex analytical questions; page-level chunking was the most consistent default

For a docs or help-centre knowledge base, where most questions are "how do I…" lookups, 512 tokens with a small overlap is a sound default. Measure in tokens, not characters: 1,200 characters is roughly 300 tokens of English prose, but more for code and far more for Chinese or Japanese.

Option A: Split locally with LangChain (free)​

from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter

# from_tiktoken_encoder measures chunk_size in tokens; the plain
# constructor would measure characters.
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=512,
chunk_overlap=50,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)

docs, ids = [], []
for page in pages:
url = page.metadata.source_url
for i, text in enumerate(splitter.split_text(page.markdown)):
docs.append(Document(page_content=text, metadata={"source_url": url, "chunk_index": i}))
ids.append(f"{url}#{i}")

The separators make the splitter prefer heading breaks, but it does not guarantee them. A chunk can still start in one section and end in the next, nothing records which section it came from, and the IDs are positions: if a paragraph is added near the top of a page, every later chunk on that page gets a new ID.

Option B: Header-aware chunks with Markdown RAG Chunker​

Markdown RAG Chunker is an Apify Actor that fixes those three gaps. It splits each page on its #, ## and ### headings first, so a chunk never straddles two sections, then re-splits only the sections longer than 512 tokens (counted with tiktoken, 10% overlap). This is the two-step pattern LangChain's own Markdown splitter docs recommend, run for you. Every chunk carries:

FieldUse
embedding_textSection path + content (Guides > Authentication\n\n...). Embed this.
header_pathSection path for metadata filters and citations
chunk_idStable upsert key: unchanged on re-crawl if the section did not change
source, document_hashFind and delete a page's stale chunks after it changes
token_countExact count, so no chunk exceeds your embedding model's limit
import hashlib
from concurrent.futures import ThreadPoolExecutor
from apify_client import ApifyClient
from langchain_core.documents import Document

apify = ApifyClient("YOUR_APIFY_TOKEN")

def page_hash(markdown: str) -> str:
# The same value the chunker returns as document_hash for markdown_text input
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]

def chunk_page(page) -> list[dict]:
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": page.markdown,
"source_id": page.metadata.source_url, # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
if run is None or run.status != "SUCCEEDED":
print(f"Skipped {page.metadata.source_url}: {run.status_message if run else 'run did not start'}")
return []
return list(apify.dataset(run.default_dataset_id).iterate_items())

indexed = {} # {source_url: document_hash} already stored; filled on re-runs (Step 6)
todo = [p for p in pages if indexed.get(p.metadata.source_url) != page_hash(p.markdown)]

docs, ids = [], []
with ThreadPoolExecutor(max_workers=4) as pool: # parallel runs: same cost, less waiting
for items in pool.map(chunk_page, todo):
for c in items:
docs.append(Document(page_content=c["embedding_text"], metadata={
"source_url": c["source"],
"section_path": c["header_path"],
"document_hash": c["document_hash"],
"chunk_index": c["chunk_index"],
}))
ids.append(c["chunk_id"])

Each run chunks one document. Pricing is $0.80 per 1,000 chunks plus $0.001 per run on Apify's free plan, compute included. As a worked example, a first pass over 100 pages that produce 400 chunks costs 400 × $0.0008 + 100 × $0.001 = $0.42. After that you pay only for pages whose Markdown changed: unchanged pages never reach the Actor, so they cost $0.

Which option? Use Option A for a one-off prototype. Use Option B when the knowledge base is re-crawled on a schedule, when you want section paths in your citations, or when the content is not in English.

Step 4: Embed and store in Chroma​

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma(
collection_name="website_kb",
embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"),
persist_directory="./chroma_db",
)
if docs:
vectorstore.add_documents(docs, ids=ids)

Only chunks from new or changed pages are embedded, so re-runs don't pay OpenAI for pages you already indexed. Passing ids matters: Chroma overwrites a document with the same ID instead of adding a duplicate. With Option B's chunk_id, unchanged sections keep their ID across crawls.

Step 5: Retrieve with metadata filters​

retriever = vectorstore.as_retriever(
search_kwargs={"k": 4, "filter": {"source_url": "https://docs.example.com/guides/auth"}}
)
for doc in retriever.invoke("How do I rotate an API key?"):
print(doc.metadata["source_url"], doc.metadata.get("section_path", ""))
print(doc.page_content[:200], "\n")

Drop the filter to search the whole knowledge base. Pass the returned chunks to your LLM as context, and show source_url (plus section_path from Option B) as the citation.

Step 6: Keep the knowledge base fresh​

  1. Re-crawl on a schedule that matches how often the site changes: daily for changelogs, weekly for reference docs.
  2. Upsert by ID. Re-adding documents with the same IDs updates them in place.
  3. Skip unchanged pages. Before Step 3, load the hashes already in the store so unchanged pages are never sent to the chunker or the embedding model:
stored = vectorstore.get(include=["metadatas"])["metadatas"]
indexed = {m["source_url"]: m["document_hash"] for m in stored if "document_hash" in m}
  1. Delete what changed. After Step 4, remove each re-chunked page's old chunks:
for url, doc_hash in {d.metadata["source_url"]: d.metadata["document_hash"] for d in docs}.items():
vectorstore.delete(where={"$and": [
{"source_url": url},
{"document_hash": {"$ne": doc_hash}},
]})

A nightly refresh then costs only the changed pages: one $0.001 run plus $0.0008 per chunk for each, and $0 for the rest.

The LangChain loader shortcut​

If you prefer LangChain's document loaders, use Firecrawl's official LangChain package:

# pip install langchain-firecrawl==0.1.0
from firecrawl import ScrapeOptions
from langchain_firecrawl import FirecrawlLoader

loader = FirecrawlLoader(
url="https://docs.example.com",
mode="crawl",
params={"limit": 100, "scrape_options": ScrapeOptions(formats=["markdown"], only_main_content=True)},
)
pages = loader.load() # page_content is Markdown; metadata includes source_url

The loader hands you whole pages as LangChain Documents, so you still need Step 3. In that code, read page.page_content instead of page.markdown and page.metadata["source_url"] instead of page.metadata.source_url, and keep copying only the fields you need into each chunk's metadata.

Vector database options​

DatabaseHostingGood fit
ChromaEmbedded, self-hosted or cloudPrototypes and single-machine apps
QdrantSelf-hosted or cloudMetadata filtering and built-in hybrid (keyword + vector) search
PineconeManaged onlyTeams that want no infrastructure
pgvectorInside PostgresApps that already run Postgres

All four have LangChain integrations, and the chunk IDs and metadata above work the same way in each. For a Qdrant version of this pipeline with hybrid search and a FastAPI endpoint, see Firecrawl + Qdrant vector search. The complete, runnable project is on GitHub: firecrawl-qdrant-vector-search.

Firecrawl or Apify for the crawl step?​

FirecrawlApify Website Content Crawler
OutputClean Markdown per pageClean Markdown per page
PricingCredits: 1 per basic page, 1,000 free a monthApify platform usage
SchedulingYou run it (cron, webhook)Built-in schedules and integrations
ChunkingLeft to youChain Markdown RAG Chunker on the same platform
Best forQuick API-first prototypesScheduled pipelines and sites that need logins or heavy anti-bot handling

Both produce Markdown that the chunking and storage steps above accept unchanged. For the Apify-native walkthrough, see how to feed live web data into a RAG pipeline. For evaluation, reranking and monitoring once the prototype works, read RAG production architecture.

Start small

Crawl 20 to 50 pages first and write ten questions you know the answers to. If the right page is not in the top four results for most of them, fix chunking before you touch the prompt.



Try Markdown RAG Chunker on Apify → | Get the full example on GitHub

Frequently Asked Questions

Map the site's URLs, crawl the pages to Markdown with a tool like Firecrawl, split the Markdown into chunks of a few hundred tokens, embed each chunk, store the vectors with their source URL in a vector database such as Chroma or Qdrant, and retrieve the closest chunks for each question.

Start with 512 tokens and a small overlap. Chroma's 2024 evaluation found 400-token recursive chunks reached 89.5% recall, and NVIDIA's 2025 study recommends 256 to 512 tokens for factoid questions and 1,024 for analytical ones. Measure chunk size in tokens, not characters.

No. Firecrawl returns clean Markdown per page; splitting it into chunks is up to you. Use a LangChain text splitter, or Markdown RAG Chunker on Apify for header-aware chunks with section paths and stable IDs.

A basic crawl costs 1 credit per page. The free plan includes 1,000 credits a month, and the Hobby plan is $16 a month for 5,000 credits when billed annually.

Any of them. Firecrawl outputs Markdown, so Chroma, Qdrant, Pinecone and pgvector all work. Store the source URL with every chunk for citations and filtering.

Re-crawl on a schedule, upsert chunks by a stable ID so unchanged sections are overwritten rather than duplicated, and delete a page's old chunks when its content hash changes. Compare each page's Markdown hash with the stored document_hash first, so unchanged pages are never re-chunked or re-embedded.

Common mistakes and fixes

Retrieval returns irrelevant chunks

Chunk by headings instead of fixed windows, embed the section path with each chunk, and filter by source_url. Test with a fixed set of benchmark questions.

Stale content in knowledge base

Re-crawl on a schedule, upsert by a stable chunk ID, and delete chunks whose document_hash no longer matches the page.

Advertise on this page