RAG in Production: From Website Crawl to Vector Search That Actually Works (2026)

April 7, 2026 · 11 min read

Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

Many RAG (Retrieval-Augmented Generation) projects fail in production not because the technology doesn't work, but because teams skip the hard parts: chunking strategy, embedding model selection, retrieval quality measurement, and stale data management. Validate each step on your own corpus — field names, SDK versions, and Actor outputs change over time.

The pipeline: crawl websites → chunk intelligently → embed → store in vector DB → retrieve with reranking → generate answers with citations.

TL;DR:

Stage	Tool	Key decisions
Crawl	Apify Website Content Crawler	Markdown output, max depth, content filtering
Chunk	LangChain `RecursiveCharacterTextSplitter`	~2000 character chunks, ~200 overlap (~500 tokens/chunk at ~4 chars/token — tune for your content)
Embed	OpenAI `text-embedding-3-small` or local `all-MiniLM-L6-v2`	Cost vs quality trade-off
Store	Qdrant or pgvector	Managed vs self-hosted
Retrieve	Dense vector search + Cohere rerank (sample below)	Top-k=20 candidates, rerank to top-5 — add BM25 / sparse hybrid in Qdrant if you need keyword-heavy queries
Generate	Claude Sonnet	With source citations

Prerequisites:

Python 3.10+ or Node.js 18+
Apify account (sign up)
Vector database (Qdrant Cloud free tier or self-hosted)
LLM API key (Claude, GPT-4, or self-hosted Ollama)

Why RAG fails: the actual failure modes

Before building, understand where RAG systems break. These failure modes are sourced from published research and production post-mortems:

Failure mode	Cause	Impact	Fix
Stale data	Documents not re-indexed	LLM cites outdated info	Scheduled re-crawling + version tags
Bad chunks	Naive splitting breaks context	Irrelevant retrieval	Semantic chunking, section-aware splitting
Wrong embedding model	Model doesn't match query style	Low recall	Benchmark against your actual queries
No reranking	Vector similarity is imprecise	First result isn't best	Add cross-encoder reranking
Hallucinated citations	LLM fabricates source references	Trust erosion	Enforce retrieval-only answers; return source URLs
Chunk boundary artifacts	Answer spans two chunks	Incomplete responses	Overlapping chunks + larger context windows

Stage 1: Crawl with Apify

The Website Content Crawler converts websites to clean Markdown — the ideal format for RAG ingestion.

Python implementation

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Configure the crawl
run = client.actor("apify/website-content-crawler").call(input={
    "startUrls": [{"url": "https://docs.example.com"}],
    "maxCrawlPages": 500,
    "crawlerType": "playwright:firefox",
    "saveMarkdown": True,  # saves crawled content as markdown (default: True)
    "removeElementsCssSelector": "nav, footer, .sidebar, .cookie-banner",
    "maxCrawlDepth": 5
})

# Fetch results
documents = []
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    documents.append({
        "url": item.get("url"),
        "title": item.get("metadata", {}).get("title", ""),
        "content": item.get("markdown") or item.get("text", ""),  # prefer markdown when available
        "crawled_at": item.get("crawledAt") or item.get("crawl", {}).get("finishedAt")
    })

print(f"Crawled {len(documents)} pages")

Dataset fields: Website Content Crawler output field names can change between Actor versions. Inspect item.keys() or a sample dataset row in Apify Console and map markdown / text / timestamps accordingly.

Alternative: Firecrawl MCP

For ad-hoc RAG ingestion via Claude Desktop, use the Firecrawl MCP server:

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": { "FIRECRAWL_API_KEY": "your_key" }
    }
  }
}

For large-scale crawling (1,000+ pages), Apify is more cost-effective. For quick URL-to-Markdown conversion (under 100 pages), Firecrawl is faster to set up.

Stage 2: Chunking strategy

Chunking is the most underrated decision in RAG. The right chunking strategy can improve retrieval accuracy significantly (commonly 10–40% in reported benchmarks — gains depend on your corpus and query distribution).

Recommended: Recursive character splitting

RecursiveCharacterTextSplitter splits by characters (approximately N tokens at ~4 chars/token — tune chunk_size for your embedding model's context window).

# LangChain ≥0.2: `pip install langchain-text-splitters` and use:
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Older installs may still use: `from langchain.text_splitter import RecursiveCharacterTextSplitter`

splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,  # characters (~500 tokens at ~4 chars/token — tune for your content)
    chunk_overlap=200,  # characters
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,  # counts characters; to count tokens instead, pass a tiktoken-based function
)

chunks = []
for doc in documents:
    doc_chunks = splitter.split_text(doc["content"])
    for i, chunk_text in enumerate(doc_chunks):
        chunks.append({
            "text": chunk_text,
            "metadata": {
                "source_url": doc["url"],
                "title": doc["title"],
                "chunk_index": i,
                "crawled_at": doc["crawled_at"]
            }
        })

print(f"Created {len(chunks)} chunks from {len(documents)} documents")

Chunking strategy comparison

Strategy	Chunk size	Overlap	Best for
Fixed-size	~2000 characters	~200 characters	General purpose, documentation
Paragraph-based	Variable	No overlap	Well-structured articles
Section-aware	By heading	No overlap	Technical docs with clear H2/H3 structure
Semantic	Variable (by topic)	No overlap	Long-form content with topic shifts

Start with ~2000 characters and ~200 overlap — this is a sensible default for character-based splitting. Optimize only after measuring retrieval quality on your actual queries.

Stage 3: Embedding

Option A: OpenAI embeddings (highest quality, per-token cost)

from openai import OpenAI

openai_client = OpenAI()

def embed_texts(texts: list[str]) -> list[list[float]]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",  # $0.02 per 1M tokens
        input=texts
    )
    return [item.embedding for item in response.data]

# Batch embed all chunks
batch_size = 100
all_embeddings = []
for i in range(0, len(chunks), batch_size):
    batch = [c["text"] for c in chunks[i:i+batch_size]]
    all_embeddings.extend(embed_texts(batch))

Option B: Local embeddings (free, good enough for most use cases)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384 dimensions

all_embeddings = model.encode(
    [c["text"] for c in chunks],
    batch_size=64,
    show_progress_bar=True
)

Embedding model comparison

Model	Dimensions	Quality (MTEB)	Cost	Speed
OpenAI `text-embedding-3-large`	3072	Highest	$0.13/1M tokens	Cloud
OpenAI `text-embedding-3-small`	1536	High	$0.02/1M tokens	Cloud
`all-MiniLM-L6-v2`	384	Good	Free (local)	Varies widely by CPU batch size
Cohere `embed-v3`	1024	High	$0.10/1M tokens	Cloud

Recommendation: Start with text-embedding-3-small for quality. Switch to local all-MiniLM-L6-v2 when embedding costs exceed $20/month. See the Self-Host AI Stack for local embedding setup.

Stage 4: Vector storage

Option A: Qdrant (purpose-built, best performance)

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct

client = QdrantClient(url="http://localhost:6333")

# Create collection — vector size MUST match your embedding model (1536 for text-embedding-3-small, 384 for all-MiniLM-L6-v2, etc.)
client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(
        size=1536,  # change to 384 if you indexed with all-MiniLM-L6-v2
        distance=Distance.COSINE
    )
)

# Upsert chunks
points = [
    PointStruct(
        id=i,
        vector=all_embeddings[i].tolist(),
        payload={
            "text": chunks[i]["text"],
            **chunks[i]["metadata"]
        }
    )
    for i in range(len(chunks))
]

client.upsert(collection_name="docs", points=points)
print(f"Indexed {len(points)} vectors")

Option B: pgvector (if you already use PostgreSQL)

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
    id SERIAL PRIMARY KEY,
    text TEXT NOT NULL,
    embedding vector(1536),  -- use (384) for all-MiniLM-L6-v2; must match embedder output
    source_url TEXT,
    title TEXT,
    chunk_index INTEGER,
    crawled_at TIMESTAMPTZ
);

CREATE INDEX ON doc_chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

Stage 5: Retrieval with reranking

Raw vector search returns approximate matches. Reranking with a cross-encoder can deliver meaningful improvements (gains vary by corpus — measure on your own data). The sample below uses the same embed_texts helper as Stage 3 (OpenAI path) — if you indexed with SentenceTransformer, use that model for query embeddings too.

from qdrant_client import QdrantClient
import cohere

qdrant = QdrantClient(url="http://localhost:6333")
co = cohere.Client("YOUR_COHERE_KEY")

def retrieve_and_rerank(query: str, top_k: int = 5) -> list[dict]:
    # Step 1: Embed the query
    query_embedding = embed_texts([query])[0]

    # Step 2: Vector search (retrieve more candidates)
    results = qdrant.search(
        collection_name="docs",
        query_vector=query_embedding,
        limit=20  # Retrieve 20, rerank to top 5
    )

    # Step 3: Rerank with Cohere
    texts = [r.payload["text"] for r in results]
    reranked = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=texts,
        top_n=top_k
    )

    # Step 4: Return reranked results with metadata
    return [
        {
            "text": results[r.index].payload["text"],
            "score": r.relevance_score,
            "source_url": results[r.index].payload["source_url"],
            "title": results[r.index].payload["title"]
        }
        for r in reranked.results
    ]

Stage 6: Generation with citations

import anthropic

client = anthropic.Anthropic()

def answer_with_rag(query: str) -> str:
    # Retrieve relevant chunks
    sources = retrieve_and_rerank(query, top_k=5)

    # Format context with source attribution
    context = "\n\n".join([
        f"[Source {i+1}: {s['title']}]({s['source_url']})\n{s['text']}"
        for i, s in enumerate(sources)
    ])

    # Generate answer with citations
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"""Answer the following question using ONLY the provided sources.
Cite sources using [Source N] notation. If the sources don't contain
enough information to answer, say so explicitly.

Sources:
{context}

Question: {query}"""
        }]
    )

    return response.content[0].text

Keeping data fresh: scheduled re-indexing

Stale data is the #1 production RAG failure. Set up automated re-crawling:

# Schedule this to run daily/weekly via cron or n8n — PSEUDOCODE (fill in clients, types, imports)
def reindex_pipeline():
    # 1. Crawl (Apify)
    documents = crawl_website("https://docs.example.com")

    # 2. Chunk
    chunks = chunk_documents(documents)

    # 3. Embed
    embeddings = embed_texts([c["text"] for c in chunks])

    # 4. ⚠️ delete_collection causes query downtime. For production, prefer blue/green:
    #    index into `docs_next`, then swap a collection alias — or use incremental upsert + tombstone old IDs.
    # qdrant.delete_collection("docs")
    # qdrant.create_collection("docs", vectors_config=VectorParams(...))

    # 5. Index new data
    # qdrant.upsert("docs", points=[...])

    print(f"Re-indexed {len(chunks)} chunks at {datetime.now()}")

For incremental updates (only re-index changed pages), track page hashes and compare:

import hashlib

def should_reindex(url: str, content: str) -> bool:
    current_hash = hashlib.sha256(content.encode()).hexdigest()
    stored_hash = get_stored_hash(url)  # from your database
    return current_hash != stored_hash

Measuring RAG quality

You can't improve what you don't measure. Three metrics that matter:

Metric	What it measures	How to compute
Recall@k	Are the right chunks in the top-k results?	Manual labeling of 50-100 query-document pairs
Answer accuracy	Is the generated answer correct?	Human evaluation or LLM-as-judge
Faithfulness	Does the answer match the retrieved sources?	Check if claims in the answer appear in source chunks

Frequently Asked Questions

Qdrant for performance and features (filtering, hybrid search, sparse vectors). pgvector if you want to avoid adding another database and already use PostgreSQL. Pinecone for fully managed, zero-ops deployments. For self-hosted setups, Qdrant is the clear winner.

With proper infrastructure: millions. Qdrant handles billions of vectors. The practical limit is embedding cost and re-indexing time. For 10,000 documents (typical for company knowledge bases), any setup works. For 1M+ documents, use batch embedding, sharded collections, and incremental indexing.

Apify for large-scale, scheduled crawling (1,000+ pages, daily re-indexing, anti-bot sites). Firecrawl for quick, ad-hoc URL-to-Markdown conversion (under 100 pages, one-time ingestion). Many teams use both: Apify for production pipelines, Firecrawl via MCP for interactive research.

For production RAG with quality requirements, yes. Reranking (e.g., Cohere rerank-v3.5) can deliver meaningful improvements (gains vary by corpus — measure on your own data) at the cost of ~50ms latency and $0.001-$0.01 per query. Skip reranking only for cost-sensitive internal tools where approximate answers are acceptable.

Use Apify's document processing Actors or dedicated PDF parsers (PyPDF2, pdf2image + Tesseract for scanned documents). Extract text, then feed into the same chunking → embedding → indexing pipeline. For mixed document types, normalize to Markdown before chunking.

Start with ~2000 characters and ~200 overlap for character-based splitting (approximately ~500 tokens per chunk at ~4 chars/token — tune for your embedding model). This balances context completeness (each chunk contains enough information to be useful) with retrieval precision (chunks are specific enough to match relevant queries). Test with your actual queries and adjust: smaller chunks for factoid QA, larger chunks for summarization.

Three defenses: (1) Explicitly instruct the LLM to answer ONLY from provided sources. (2) Post-process: verify that every [Source N] citation in the answer corresponds to an actual retrieved chunk. (3) Return source URLs alongside the answer so users can verify.

Yes. Llama 3.1 8B handles RAG generation adequately for internal tools. Quality is lower than Claude Sonnet for nuanced questions, but sufficient for documentation QA and knowledge base search. See our Self-Host AI Stack guide for Ollama setup.

RAG in production requires attention to crawling freshness, chunking quality, and retrieval precision — not just connecting a vector database to a language model. The pipeline above handles 80% of production RAG use cases. Measure, iterate, and resist the temptation to over-engineer before you have users testing the system.

Start crawling with Apify for your website data pipeline. For MCP-based interactive RAG, install the Firecrawl MCP server.

Why RAG fails: the actual failure modes​

Stage 1: Crawl with Apify​

Python implementation​

Alternative: Firecrawl MCP​

Stage 2: Chunking strategy​

Recommended: Recursive character splitting​

Chunking strategy comparison​

Stage 3: Embedding​

Option A: OpenAI embeddings (highest quality, per-token cost)​

Option B: Local embeddings (free, good enough for most use cases)​

Embedding model comparison​

Stage 4: Vector storage​

Option A: Qdrant (purpose-built, best performance)​

Option B: pgvector (if you already use PostgreSQL)​

Stage 5: Retrieval with reranking​

Stage 6: Generation with citations​

Keeping data fresh: scheduled re-indexing​

Measuring RAG quality​

Common mistakes and fixes