Firecrawl + Qdrant: Build a Vector Search Engine for Any Website

You want semantic search over any website — not just keyword matching, but "find me everything about authentication errors" even when the page says "login failures." The two-tool stack that delivers this in under 200 lines of Python: Firecrawl for crawling and Qdrant for vector search.

This tutorial walks you from a blank Python project to a running semantic search API. You'll crawl a website with Firecrawl, chunk the markdown output, embed the chunks with OpenAI, upsert to a local Qdrant instance, and expose a FastAPI search endpoint that blends keyword and semantic results.

Contains affiliate links. Disclosure.

Why Firecrawl + Qdrant?

Most vector search tutorials skip the hardest part: getting clean text from real websites. Raw HTML ruins embeddings. Cookie banners, nav menus, and <script> blocks dilute semantic meaning and waste token budget.

Firecrawl solves the extraction layer. It handles JavaScript rendering, strips navigation boilerplate, and returns clean Markdown — exactly what an embedding model expects. Qdrant handles the storage and retrieval layer with native hybrid search (dense + sparse vectors), a Rust-powered engine, and a generous free tier.

Together they form a complete pipeline:

Stage	Tool	Output
Crawl	Firecrawl /crawl API	Array of {url, markdown} objects
Chunk	Python RecursiveCharacterTextSplitter	512-token text chunks
Embed	OpenAI text-embedding-3-small	1536-dim float vectors
Store	Qdrant collection	Indexed points with payload
Search	Qdrant hybrid query	Ranked results by relevance

Prerequisites

pip install firecrawl-py qdrant-client openai fastapi uvicorn langchain-text-splitters

You need:

• A Firecrawl API key (free tier: 500 pages/month)
• An OpenAI API key (or swap for a local model — covered below)
• Docker for running Qdrant locally (or use Qdrant Cloud)

Start Qdrant:

docker run -p 6333:6333 qdrant/qdrant

Step 1 — Crawl a Website with Firecrawl

Firecrawl's /crawl endpoint recursively fetches every page on a domain and returns clean Markdown. One API call replaces weeks of custom spider code.

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])

result = app.crawl_url(
    "https://docs.qdrant.tech",
    params={
        "limit": 100,          # max pages to crawl
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True,   # strips nav, footer, ads
        },
    },
)

pages = result.get("data", [])
print(f"Crawled {len(pages)} pages")
# [{'url': 'https://docs.qdrant.tech/...', 'markdown': '# Collections\n...'}]

onlyMainContent: True is the key flag. Firecrawl uses heuristic DOM analysis to remove boilerplate before returning Markdown — this keeps your embeddings focused on actual content.

Practical limits: The free Firecrawl tier allows 500 pages/month. For larger sites, use the maxDepth and includePaths parameters to focus the crawl on specific sections.

Step 2 — Chunk the Markdown

Embedding an entire page as one vector loses granularity. A 2,000-word article becomes a single point that represents everything and nothing specifically.

Chunk at 512 tokens with 50-token overlap. The overlap preserves context at chunk boundaries — a sentence explaining a concept won't be split from the code example that follows it.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)

chunks = []
for page in pages:
    url = page["url"]
    markdown = page.get("markdown", "")
    if not markdown.strip():
        continue

    splits = splitter.split_text(markdown)
    for i, text in enumerate(splits):
        chunks.append({
            "text": text,
            "url": url,
            "chunk_index": i,
        })

print(f"Created {len(chunks)} chunks from {len(pages)} pages")

The separators list respects Markdown structure: it prefers splitting at H2/H3 headers before falling back to paragraphs, then sentences.

Step 3 — Generate Embeddings

text-embedding-3-small costs $0.02 per million tokens and produces 1536-dimension vectors. For a 100-page documentation site, expect 200-400 chunks (~50K tokens, well under $0.01).

from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def embed_batch(texts: list[str], batch_size: int = 100) -> list[list[float]]:
    """Embed texts in batches to respect API rate limits."""
    all_embeddings = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i : i + batch_size]
        response = client.embeddings.create(
            model="text-embedding-3-small",
            input=batch,
        )
        all_embeddings.extend([e.embedding for e in response.data])
    return all_embeddings

texts = [c["text"] for c in chunks]
embeddings = embed_batch(texts)
print(f"Generated {len(embeddings)} embeddings of dim {len(embeddings[0])}")

Using a local model instead: If you prefer to avoid OpenAI costs, swap for sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384 dims, runs on CPU
embeddings = model.encode([c["text"] for c in chunks]).tolist()

The Qdrant setup below works identically — just change size=384 instead of size=1536.

Step 4 — Upsert to Qdrant

Create a collection and load your vectors. Qdrant stores each vector as a "point" with a numeric ID, a vector, and a JSON payload for metadata filtering.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client_q = QdrantClient(host="localhost", port=6333)

COLLECTION_NAME = "website_search"

# Create collection (idempotent)
client_q.recreate_collection(
    collection_name=COLLECTION_NAME,
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# Build points
points = [
    PointStruct(
        id=i,
        vector=embeddings[i],
        payload={
            "text": chunks[i]["text"],
            "url": chunks[i]["url"],
            "chunk_index": chunks[i]["chunk_index"],
        },
    )
    for i in range(len(chunks))
]

# Upsert in batches of 100
batch_size = 100
for i in range(0, len(points), batch_size):
    client_q.upsert(
        collection_name=COLLECTION_NAME,
        points=points[i : i + batch_size],
    )

print(f"Upserted {len(points)} points to Qdrant")

After uploading, verify in the Qdrant dashboard at http://localhost:6333/dashboard.

Step 5 — Semantic Search

Query by embedding the search string and finding nearest neighbors:

def semantic_search(query: str, limit: int = 5) -> list[dict]:
    query_vector = client.embeddings.create(
        model="text-embedding-3-small",
        input=[query],
    ).data[0].embedding

    results = client_q.search(
        collection_name=COLLECTION_NAME,
        query_vector=query_vector,
        limit=limit,
        with_payload=True,
    )

    return [
        {
            "score": r.score,
            "url": r.payload["url"],
            "text": r.payload["text"][:200],
        }
        for r in results
    ]

# Test it
hits = semantic_search("how to configure authentication")
for h in hits:
    print(f"{h['score']:.3f} | {h['url']}")
    print(f"  {h['text']}\n")

Step 6 — Hybrid Search (Keyword + Semantic)

Pure semantic search misses exact product names, error codes, and version numbers. Hybrid search blends dense vector similarity with sparse BM25-style keyword matching.

Qdrant supports this natively via sparse vectors. You'll need fastembed to generate sparse vectors:

pip install fastembed

from qdrant_client.models import (
    SparseVector,
    SparseVectorParams,
    NamedSparseVector,
    NamedVector,
    SearchRequest,
)
from fastembed import SparseTextEmbedding

sparse_model = SparseTextEmbedding(model_name="Qdrant/bm25")

# Recreate collection with both dense and sparse vectors
client_q.recreate_collection(
    collection_name=COLLECTION_NAME,
    vectors_config={"dense": VectorParams(size=1536, distance=Distance.COSINE)},
    sparse_vectors_config={"sparse": SparseVectorParams()},
)

# Generate sparse embeddings
sparse_embeddings = list(sparse_model.embed([c["text"] for c in chunks]))

# Rebuild points with both vector types
points = [
    PointStruct(
        id=i,
        vector={
            "dense": embeddings[i],
            "sparse": SparseVector(
                indices=sparse_embeddings[i].indices.tolist(),
                values=sparse_embeddings[i].values.tolist(),
            ),
        },
        payload={
            "text": chunks[i]["text"],
            "url": chunks[i]["url"],
        },
    )
    for i in range(len(chunks))
]

client_q.upsert(collection_name=COLLECTION_NAME, points=points)


def hybrid_search(query: str, limit: int = 5) -> list[dict]:
    # Dense query vector
    dense_vec = client.embeddings.create(
        model="text-embedding-3-small",
        input=[query],
    ).data[0].embedding

    # Sparse query vector
    sparse_vec = list(sparse_model.embed([query]))[0]

    results = client_q.query_points(
        collection_name=COLLECTION_NAME,
        prefetch=[
            {"query": dense_vec, "using": "dense", "limit": 20},
            {
                "query": SparseVector(
                    indices=sparse_vec.indices.tolist(),
                    values=sparse_vec.values.tolist(),
                ),
                "using": "sparse",
                "limit": 20,
            },
        ],
        query={"fusion": "rrf"},  # Reciprocal Rank Fusion
        limit=limit,
        with_payload=True,
    )

    return [
        {
            "score": r.score,
            "url": r.payload["url"],
            "text": r.payload["text"][:200],
        }
        for r in results.points
    ]

Reciprocal Rank Fusion (RRF) merges the two ranked lists without requiring score normalization — it's the standard approach for combining heterogeneous retrieval signals.

Step 7 — Expose a FastAPI Search Endpoint

Wire everything into a minimal REST API:

from fastapi import FastAPI, Query
from pydantic import BaseModel

api = FastAPI(title="Website Search API")


class SearchResult(BaseModel):
    score: float
    url: str
    text: str


@api.get("/search", response_model=list[SearchResult])
def search(
    q: str = Query(..., description="Search query"),
    limit: int = Query(5, ge=1, le=20),
    mode: str = Query("hybrid", description="semantic | hybrid"),
):
    if mode == "semantic":
        return semantic_search(q, limit)
    return hybrid_search(q, limit)


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(api, host="0.0.0.0", port=8000)

Run it:

uvicorn main:api --reload

Test:

curl "http://localhost:8000/search?q=configure+authentication&mode=hybrid"

Complete Project Structure

website-search/
├── crawl.py          # Firecrawl ingestion
├── embed.py          # Chunking + embedding
├── index.py          # Qdrant upsert
├── search.py         # Semantic + hybrid search functions
├── main.py           # FastAPI app
└── requirements.txt

requirements.txt:

firecrawl-py>=0.0.16
qdrant-client>=1.9.0
openai>=1.0.0
fastapi>=0.110.0
uvicorn>=0.29.0
langchain-text-splitters>=0.0.1
fastembed>=0.3.0

When to Use Apify Instead

Firecrawl is the right choice for documentation sites, marketing pages, and content-heavy domains where the crawl is straightforward. Use Apify instead when:

• The target site requires authentication or session management
• You need to handle JavaScript-heavy SPAs with complex interactions (infinite scroll, lazy loading)
• The site uses aggressive anti-bot measures (Cloudflare Enterprise, DataDome)
• You need scheduled recurring crawls with change detection

Apify's Website Content Crawler outputs the same clean Markdown format that Qdrant expects, so the chunking and indexing code above works unchanged — just swap the crawl step.

For more on RAG pipeline architecture including Qdrant ingestion patterns, see the RAG pipeline guide.

Performance Benchmarks

For a 100-page documentation site (Qdrant docs used as test case):

Metric	Value
Crawl time (Firecrawl)	~45 seconds
Pages crawled	100
Chunks generated	~380
Embedding time (OpenAI batch)	~8 seconds
Qdrant upsert time	~2 seconds
Search latency (semantic)	~120ms
Search latency (hybrid)	~180ms
Total pipeline cost	<$0.01

---

FAQ

What is Qdrant?

Qdrant is an open-source vector database written in Rust. It stores high-dimensional vectors alongside JSON payloads and provides fast approximate nearest-neighbor search. It supports both dense vectors (from neural embeddings) and sparse vectors (for keyword search), enabling hybrid retrieval in a single query. You can run it locally via Docker or use Qdrant Cloud for a managed instance.

How do I build vector search from a website?

The four-step process: (1) crawl the website with a tool like Firecrawl to get clean Markdown text, (2) chunk the Markdown into 400-600 token segments, (3) embed each chunk with a model like text-embedding-3-small, (4) upsert the vectors to Qdrant with the source URL as metadata. Queries are processed the same way — embed the query string and return the nearest chunks by cosine similarity.

Can I use Firecrawl with Qdrant?

Yes. Firecrawl's /crawl endpoint returns {url, markdown} objects that feed directly into a chunking and embedding pipeline. The onlyMainContent: True scrape option removes navigation and boilerplate before you start chunking, which directly improves embedding quality. The tutorial above covers the complete integration.

Do I need OpenAI for embeddings?

No. The sentence-transformers library provides CPU-friendly models like all-MiniLM-L6-v2 (384 dims) that run locally for free. The trade-off is retrieval quality: OpenAI's text-embedding-3-small consistently outperforms open-source models on English documentation retrieval benchmarks. For production use cases where search quality matters, OpenAI's $0.02/million token pricing is negligible.

What is hybrid search and when should I use it?

Hybrid search combines dense vector similarity (semantic understanding) with sparse BM25-style keyword matching in a single query. Use hybrid when your content includes exact product names, error codes, API method names, or version numbers — terms where semantic search alone underperforms because the model lacks domain-specific training. The Reciprocal Rank Fusion (RRF) fusion strategy used here requires no tuning and consistently outperforms either method alone.

---

Ready to build? Get a Firecrawl API key and have your first site indexed in under 10 minutes.