Firecrawl for RAG: Build a Knowledge Base from Any Website 2026

Contains affiliate links. Disclosure.

Use Firecrawl to crawl a site, extract markdown, chunk it, embed, and store in a vector database. Then retrieve relevant chunks for RAG. Pipeline: map → crawl → chunk → embed → vector DB → retriever. Firecrawl gives clean markdown; LangChain or LlamaIndex handle the rest.

Start Firecrawl RAG →

RAG Architecture with Firecrawl

1. Discover — /map to scope URLs
2. Extract — /scrape or /crawl for markdown
3. Chunk — Split by size/overlap, preserve structure
4. Embed — OpenAI, Cohere, or local embeddings
5. Store — Pinecone, Qdrant, Chroma, etc.
6. Retrieve — Query with metadata filters (source_url, timestamp)

Minimal Ingestion (Python)

from firecrawl import Firecrawl
from langchain_text_splitters import RecursiveCharacterTextSplitter

app = Firecrawl(api_key="fc-YOUR-API-KEY")
doc = app.scrape("https://example.com/docs/page", formats=["markdown"])
md = doc.get("markdown", "")

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200,
    chunk_overlap=150
)
chunks = splitter.split_text(md)

Store each chunk with metadata: source_url, title, crawl_timestamp, section_path.

Full Pipeline Sketch

from langchain_community.document_loaders import FireCrawlLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

loader = FireCrawlLoader(url="https://example.com/docs", api_key="fc-KEY", mode="crawl")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=150)
chunks = splitter.split_documents(docs)

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")
retriever = vectorstore.as_retriever(k=4, search_kwargs={"filter": {"source": "example.com"}})

Retrieval Quality

• Remove nav/footer before embedding
• Keep canonical URL in metadata
• Use benchmark questions to evaluate precision
• Track stale chunk percentage

Chunking Strategy

Content type	chunk_size	overlap
Docs	1000–1500	150–200
Blog	800–1200	100–150
Dense prose	600–1000	100

Preserve section boundaries when possible.

Vector DB Options

• Chroma — Local, simple
• Pinecone — Managed, scalable
• Qdrant — Self-hosted, metadata filters

All work with LangChain. Preserve source_url and crawl_timestamp for citations and freshness.

Next step

Connect your RAG pipeline to Claude or Cursor via MCP for agent-driven research.

Frequently Asked Questions

Crawl or scrape with Firecrawl, chunk markdown, embed, store in a vector DB. Retrieve with metadata filters and feed to the LLM.

Any that supports metadata (Chroma, Pinecone, Qdrant). Preserve source_url and crawl_timestamp.

Retrieval precision on a benchmark set before optimizing generation quality.

Firecrawl: clean markdown, fast. Apify: more control, platform-specific actors. Both can feed RAG.