Skip to main content

Firecrawl + Qdrant: Build Vector Search for Any Website

· 15 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

You want semantic search over a website: "find everything about authentication errors" should match a page that only says "login failures." This tutorial builds it with Firecrawl for crawling, Markdown RAG Chunker for section-aware chunks, and Qdrant for hybrid (vector + keyword) search, all in about 200 lines of Python.

The pipeline: crawl a site to Markdown, split it into chunks that each stay inside one section, embed them, upsert to Qdrant with dense and BM25 vectors, and serve a FastAPI endpoint that fuses both rankings. Every snippet comes from a tested companion repo you can clone and run: github.com/yel-hadd/firecrawl-qdrant-vector-search.

Why Firecrawl + Qdrant?​

Most vector search tutorials skip the hardest part: getting clean text out of real websites. Raw HTML ruins embeddings. Cookie banners, navigation menus and <script> blocks dilute meaning and waste tokens.

Firecrawl handles extraction. It renders JavaScript, strips boilerplate, and returns clean Markdown. Qdrant handles storage and retrieval, with dense and sparse vectors in one collection, so hybrid search is a single query.

StageToolOutput
CrawlFirecrawl crawl()One Markdown document per page
ChunkMarkdown RAG Chunker or LangChain≤512-token chunks, one section each
EmbedOpenAI text-embedding-3-small or local FastEmbedDense vectors + BM25 sparse vectors
StoreQdrant collectionPoints with a stable ID and payload
SearchQdrant query_points with RRFRanked results from both signals

Prerequisites​

pip install firecrawl-py==4.53.2 apify-client==3.3.0 qdrant-client==1.19.1 \
langchain-text-splitters==1.1.3 tiktoken==0.14.0 fastembed==0.9.0 \
openai==3.35.1 fastapi==0.156.0 uvicorn==0.54.0

These are the latest stable versions as of October 2026; the code below targets them.

You need:

  • A Firecrawl API key. The free plan includes 1,000 credits a month, and a basic page costs 1 credit.
  • An Apify account for the chunker (Option B in Step 2). Skip it if you chunk locally.
  • An OpenAI API key, or nothing: the code falls back to a local embedding model.
  • Docker for Qdrant, or a Qdrant Cloud cluster.
docker run -p 6333:6333 qdrant/qdrant

Step 1 — Crawl a Website with Firecrawl​

crawl() follows links from a start URL, waits for the job to finish, and returns one Document per page.

import os
from firecrawl import Firecrawl, ScrapeOptions

firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])

job = firecrawl.crawl(
"https://docs.qdrant.tech",
limit=50, # max pages, 1 credit each
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True),
)

pages = [
{"url": doc.metadata.source_url, "markdown": doc.markdown}
for doc in job.data
if doc.markdown and doc.markdown.strip()
]
print(f"Crawled {len(pages)} pages")

only_main_content=True removes navigation, footers and banners before the Markdown comes back, which keeps your embeddings focused on the content. To stay inside one section of a large site, pass include_paths (regex patterns matched against the URL path) and max_discovery_depth.

Step 2 — Chunk the Markdown​

Embedding a whole page as one vector loses precision: a 2,000-word article becomes one point that represents everything and nothing. Split it into chunks of a few hundred tokens. Chroma's chunking evaluation found recursive splitting at around 400 tokens among the best-performing configurations, and NVIDIA's 2025 chunking study recommends 256 to 512 tokens for factoid questions. This tutorial uses 512.

Option A — Chunk locally with LangChain​

import hashlib
from langchain_text_splitters import RecursiveCharacterTextSplitter

# from_tiktoken_encoder measures chunk_size in tokens. The plain
# constructor measures characters, so chunk_size=512 there means
# roughly 130 tokens of English, not 512.
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=512,
chunk_overlap=50,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)

chunks = []
for page in pages:
doc_hash = hashlib.sha256(page["markdown"].encode()).hexdigest()[:16]
for i, text in enumerate(splitter.split_text(page["markdown"])):
chunks.append({
"id": f"{page['url']}#{i}",
"text": text, # embedded
"content": text, # shown in results
"url": page["url"],
"section": "",
"document_hash": doc_hash,
})

The separators list prefers splitting at H2 and H3 headings before falling back to paragraphs. That is a preference, not a guarantee: a chunk can still run from the end of one section into the next, nothing records which section a chunk came from, and the ID is the chunk's position, so an edit near the top of a page shifts every ID below it.

Option B — Section-aware chunks with Markdown RAG Chunker​

Markdown RAG Chunker is an Apify Actor that closes those three gaps. It splits each page on its #, ## and ### headings first, so a chunk never crosses a section boundary, then re-splits only the sections over 512 tokens (counted with tiktoken, 10% overlap). Each chunk comes back with:

FieldWhat it gives you
embedding_textSection path + content (Authentication > API keys\n\n...), so the vector knows where the text sits. Embed this.
chunk_idA stable ID from the source URL and the chunk's content. Unchanged sections keep it on the next crawl.
header_pathThe section path, for filters and citations
document_hashFingerprint of the page, for deleting stale chunks
token_countExact token count

Pass each Firecrawl page as markdown_text, with its URL as source_id. The helper below keeps the bill to what actually changed:

import hashlib
from concurrent.futures import ThreadPoolExecutor
from apify_client import ApifyClient

apify = ApifyClient(os.environ["APIFY_TOKEN"])

def page_hash(markdown: str) -> str:
# The same value the chunker returns as document_hash for markdown_text input
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]

def chunk_page(page: dict) -> list[dict]:
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": page["markdown"],
"source_id": page["url"], # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
if run is None or run.status != "SUCCEEDED":
print(f"Skipped {page['url']}: {run.status_message if run else 'run did not start'}")
return []
return list(apify.dataset(run.default_dataset_id).iterate_items())

indexed = {} # {url: document_hash} already in Qdrant; filled on re-runs (Step 4)
todo = [p for p in pages if indexed.get(p["url"]) != page_hash(p["markdown"])]

# 4 runs at a time (under the Free plan concurrency limit): same cost, less waiting
with ThreadPoolExecutor(max_workers=4) as pool:
items = [item for page_items in pool.map(chunk_page, todo) for item in page_items]

chunks = [
{
"id": item["chunk_id"],
"text": item["embedding_text"],
"content": item["content"],
"url": item["source"],
"section": item["header_path"],
"document_hash": item["document_hash"],
}
for item in items
]

Three things keep this cheap. Pages whose Markdown hash matches what is already indexed are never sent, so unchanged pages cost $0. max_chunks makes a runaway page (a huge changelog, say) fail before any chunk is billed. And the runs use the default 1 GB of memory, which is one $0.001 start event; more memory buys nothing here and adds a start event per GB.

For a page with two subsections, the Actor returns one chunk per section:

{
"content": "A 401 response means the key is missing or revoked.",
"header_path": "Authentication > Login errors",
"embedding_text": "Authentication > Login errors\n\nA 401 response means the key is missing or revoked.",
"chunk_id": "25273f9da6ab796e",
"source": "https://example.com/auth",
"document_hash": "a303951493cc6dcc",
"token_count": 12
}

Run the same input twice and the chunk_id values come back identical, which is what makes the upserts in Step 4 idempotent.

Which option? Each run chunks one page and costs $0.001, plus $0.80 per 1,000 chunks on the Apify free plan: about $0.42 for a first pass over 100 pages and 400 chunks, and only the changed pages after that. Option A is free. Choose Option B when you re-crawl on a schedule and want unchanged sections to keep their points, or when section context matters, as in API references where every endpoint has its own "Parameters" heading.

Step 3 — Generate Embeddings​

text-embedding-3-small produces 1536-dimension vectors and costs $0.02 per million tokens, so a 50-page site costs well under a cent. Embed text, which for Option B includes the section path.

from openai import OpenAI

openai = OpenAI()

def embed_dense(texts: list[str], batch_size: int = 100) -> list[list[float]]:
vectors = []
for start in range(0, len(texts), batch_size):
response = openai.embeddings.create(
model="text-embedding-3-small",
input=texts[start:start + batch_size],
)
vectors.extend(item.embedding for item in response.data)
return vectors

No OpenAI key? FastEmbed runs small models on CPU. BAAI/bge-small-en-v1.5 gives 384-dimension vectors:

from fastembed import TextEmbedding

local = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")

def embed_dense(texts: list[str]) -> list[list[float]]:
return [v.tolist() for v in local.embed(texts)]

Hybrid search also needs sparse keyword vectors. FastEmbed's Qdrant/bm25 model produces them:

from fastembed import SparseTextEmbedding

bm25 = SparseTextEmbedding(model_name="Qdrant/bm25")

texts = [c["text"] for c in chunks]
dense = embed_dense(texts)
sparse = list(bm25.embed(texts))

Step 4 — Upsert to Qdrant​

Create a collection with a named dense vector and a named sparse vector. The IDF modifier lets Qdrant weight BM25 terms by how rare they are across the collection.

import uuid
from qdrant_client import QdrantClient, models

qdrant = QdrantClient(url="http://localhost:6333")
COLLECTION = "website_search"

if not qdrant.collection_exists(COLLECTION):
qdrant.create_collection(
collection_name=COLLECTION,
vectors_config={"dense": models.VectorParams(size=1536, distance=models.Distance.COSINE)},
sparse_vectors_config={"sparse": models.SparseVectorParams(modifier=models.Modifier.IDF)},
)

points = [
models.PointStruct(
# Qdrant IDs must be integers or UUIDs: derive a UUID from the chunk ID.
id=str(uuid.uuid5(uuid.NAMESPACE_URL, c["id"])),
vector={
"dense": d,
"sparse": models.SparseVector(indices=s.indices.tolist(), values=s.values.tolist()),
},
payload={k: c.get(k, "") for k in ("content", "url", "section", "document_hash")},
)
for c, d, s in zip(chunks, dense, sparse)
]

for start in range(0, len(points), 100):
qdrant.upsert(collection_name=COLLECTION, points=points[start:start + 100])

Use size=384 if you embedded with FastEmbed. The collection_exists check keeps re-runs from wiping your data.

On the next crawl, load the hashes you already indexed so Step 2 skips unchanged pages:

def indexed_hashes() -> dict[str, str]:
if not qdrant.collection_exists(COLLECTION):
return {}
hashes, offset = {}, None
while True:
points, offset = qdrant.scroll(COLLECTION, limit=1000, offset=offset,
with_payload=["url", "document_hash"])
hashes.update({p.payload["url"]: p.payload["document_hash"] for p in points})
if offset is None:
return hashes

indexed = indexed_hashes()

Because the point ID derives from the chunk ID, re-running the pipeline overwrites points instead of duplicating them. With Option B, unchanged sections keep their IDs. To drop chunks from sections that were edited or removed, delete the points whose url matches a re-crawled page and whose document_hash differs from the new one:

for url, doc_hash in {c["url"]: c["document_hash"] for c in chunks}.items():
qdrant.delete(
collection_name=COLLECTION,
points_selector=models.FilterSelector(filter=models.Filter(
must=[models.FieldCondition(key="url", match=models.MatchValue(value=url))],
must_not=[models.FieldCondition(key="document_hash", match=models.MatchValue(value=doc_hash))],
)),
)

Check the result in the Qdrant dashboard at http://localhost:6333/dashboard.

Query with query_points and name the vector to search with using:

def semantic_search(query: str, limit: int = 5) -> list[dict]:
response = qdrant.query_points(
collection_name=COLLECTION,
query=embed_dense([query])[0],
using="dense",
limit=limit,
with_payload=True,
)
return [
{"score": p.score, "url": p.payload["url"], "section": p.payload["section"], "text": p.payload["content"][:300]}
for p in response.points
]

for hit in semantic_search("how to configure authentication"):
print(f"{hit['score']:.3f} | {hit['url']} | {hit['section']}")

Step 6 — Hybrid Search (Keyword + Semantic)​

Pure semantic search misses exact product names, error codes and version numbers. Hybrid search runs a dense query and a BM25 query, then merges the two ranked lists with Reciprocal Rank Fusion (RRF), which needs no score normalisation:

def hybrid_search(query: str, limit: int = 5) -> list[dict]:
q_sparse = next(iter(bm25.query_embed(query)))
response = qdrant.query_points(
collection_name=COLLECTION,
prefetch=[
models.Prefetch(query=embed_dense([query])[0], using="dense", limit=20),
models.Prefetch(
query=models.SparseVector(indices=q_sparse.indices.tolist(), values=q_sparse.values.tolist()),
using="sparse",
limit=20,
),
],
query=models.FusionQuery(fusion=models.Fusion.RRF),
limit=limit,
with_payload=True,
)
return [
{"score": p.score, "url": p.payload["url"], "section": p.payload["section"], "text": p.payload["content"][:300]}
for p in response.points
]

Use bm25.query_embed for the query and bm25.embed for documents; the BM25 model weights them differently.

Step 7 — Expose a FastAPI Search Endpoint​

from typing import Literal
from fastapi import FastAPI, Query
from pydantic import BaseModel

api = FastAPI(title="Website Search API")


class SearchResult(BaseModel):
score: float
url: str
section: str
text: str


@api.get("/search", response_model=list[SearchResult])
def search(
q: str = Query(..., description="Search query"),
limit: int = Query(5, ge=1, le=20),
mode: Literal["semantic", "hybrid"] = "hybrid",
):
return semantic_search(q, limit) if mode == "semantic" else hybrid_search(q, limit)
uvicorn main:api --reload
curl "http://localhost:8000/search?q=configure+authentication&mode=hybrid"

Complete Project on GitHub​

The companion repo wires these steps into a CLI and an API, with an offline test that runs without any API key (in-memory Qdrant, local embeddings):

firecrawl-qdrant-vector-search/
├── ingest.py # crawl → chunk → embed → upsert → delete stale
├── main.py # FastAPI /search (semantic | hybrid)
├── src/crawl.py # Firecrawl
├── src/chunk.py # Markdown RAG Chunker or LangChain
├── src/embed.py # OpenAI or FastEmbed, plus BM25
├── src/index.py # Qdrant collection, UUID IDs, stale cleanup
├── src/search.py # semantic and RRF hybrid search
└── tests/ # offline end-to-end test
python ingest.py https://docs.qdrant.tech --limit 50 --chunker apify

What It Costs​

For a 50-page documentation site at October 2026 list prices:

ItemCost
Firecrawl crawl50 credits (free plan: 1,000 a month)
Markdown RAG Chunker, ~190 chunks~$0.20 (50 runs × $0.001 + 190 × $0.0008)
OpenAI text-embedding-3-smallunder $0.01
Qdrantfree (Docker, or the Qdrant Cloud free tier)

With Option A and local embeddings, the only cost is the Firecrawl credits.

When to Use Apify Instead​

Firecrawl suits documentation sites, marketing pages and content-heavy domains where the crawl is straightforward. Use Apify instead when:

  • The target site requires authentication or session management
  • You need to handle JavaScript-heavy SPAs with complex interactions (infinite scroll, lazy loading)
  • The site uses aggressive anti-bot measures (Cloudflare Enterprise, DataDome)
  • You want crawling, chunking and scheduling on one platform

Apify's Website Content Crawler returns the same clean Markdown per page, so the chunking and indexing code above works unchanged: swap Step 1 only.

With both steps on Apify, the whole ingestion runs on one schedule: Website Content Crawler fetches the pages, and Markdown RAG Chunker chunks each one with Skip if unchanged turned on. A page that has not changed since the last run produces no chunks and costs only the $0.001 run fee, so you re-embed only what changed.

For more on RAG pipeline architecture, see the RAG pipeline guide.


FAQ​

What is Qdrant?​

Qdrant is an open-source vector database written in Rust. It stores vectors alongside JSON payloads and runs fast approximate nearest-neighbour search. It supports dense vectors (from neural embeddings) and sparse vectors (for keyword search) in the same collection, so hybrid retrieval is one query. Run it locally with Docker or use Qdrant Cloud.

How do I build vector search from a website?​

Five steps: (1) crawl the site to Markdown with Firecrawl or Apify's Website Content Crawler, (2) split the Markdown into chunks of a few hundred tokens, ideally one section per chunk, (3) embed each chunk, (4) upsert the vectors to Qdrant with the source URL in the payload, (5) at query time, embed the question and return the nearest chunks. The companion repo runs all five.

What chunk size should I use for Qdrant RAG?​

Start at 512 tokens and measure in tokens, not characters. Chroma's 2024 evaluation and NVIDIA's 2025 study both found a few hundred tokens per chunk retrieves best for factual questions. Keep each chunk inside one section of the page so its context stays intact.

Can I use Firecrawl with Qdrant?​

Yes. Firecrawl's crawl() returns one Markdown document per page, which feeds straight into chunking and embedding. Its only_main_content option removes navigation and boilerplate first, which improves embedding quality.

Do I need OpenAI for embeddings?​

No. FastEmbed runs models such as BAAI/bge-small-en-v1.5 (384 dimensions) on CPU for free. OpenAI's text-embedding-3-small usually retrieves better on English documentation, and at $0.02 per million tokens it costs little.

What is hybrid search and when should I use it?​

Hybrid search combines dense vector similarity (meaning) with sparse BM25 matching (exact terms) in one query. Use it when your content includes product names, error codes, API method names or version numbers, where semantic search alone misses exact matches. RRF fusion needs no tuning.

How do I stop duplicates when I re-crawl?​

Use a stable chunk ID as the point ID instead of the list position, as in Step 4. Markdown RAG Chunker's chunk_id stays the same for unchanged sections, so a re-crawl overwrites them in place, and the document_hash filter deletes the leftovers from edited sections.


Next step: clone the companion repo, add a Firecrawl API key and an Apify token, and index your first site in under 10 minutes.

Common mistakes and fixes

Qdrant rejects vectors with a wrong dimension error

The collection size must match the embedding model: 1536 for text-embedding-3-small, 384 for BAAI/bge-small-en-v1.5. Delete the collection and rerun after switching models.

Re-running ingestion duplicates results

Use a stable chunk ID as the Qdrant point ID (Step 4) instead of the list position, then delete stale points by document_hash.

Advertise on this page