RAG in Production: From Website Crawl to Vector Search That Actually Works (2026)
· 11 min read
Many RAG (Retrieval-Augmented Generation) projects fail in production not because the technology doesn't work, but because teams skip the hard parts: chunking strategy, embedding model selection, retrieval quality measurement, and stale data management. Validate each step on your own corpus — field names, SDK versions, and Actor outputs change over time.
The pipeline: crawl websites → chunk intelligently → embed → store in vector DB → retrieve with reranking → generate answers with citations.
TL;DR:
| Stage | Tool | Key decisions |
|---|---|---|
| Crawl | Apify Website Content Crawler | Markdown output, max depth, content filtering |
| Chunk | LangChain RecursiveCharacterTextSplitter | ~2000 character chunks, ~200 overlap (~500 tokens/chunk at ~4 chars/token — tune for your content) |
| Embed | OpenAI text-embedding-3-small or local all-MiniLM-L6-v2 | Cost vs quality trade-off |
| Store | Qdrant or pgvector | Managed vs self-hosted |
| Retrieve | Dense vector search + Cohere rerank (sample below) | Top-k=20 candidates, rerank to top-5 — add BM25 / sparse hybrid in Qdrant if you need keyword-heavy queries |
| Generate | Claude Sonnet | With source citations |
Prerequisites:
- Python 3.10+ or Node.js 18+
- Apify account (sign up)
- Vector database (Qdrant Cloud free tier or self-hosted)
- LLM API key (Claude, GPT-4, or self-hosted Ollama)