RAG: guides & tutorials

TL;DR

• One docker-compose.yml: Dify (API + worker + web + sandbox) + Qdrant + Ollama + PostgreSQL + Redis + Caddy
• Measured idle RAM: ~7.4 GB (Llama 3 8B loaded, CPU inference); minimum server: 16 GB VPS
• Replaces: OpenAI Assistants API ($0.03/1k tokens) + Pinecone Starter ($70/mo) with a fully local, zero-egress alternative
• Your documents never leave the server — not during upload, not during embedding, not during inference

Retrieval-Augmented Generation (RAG) lets you ask questions against a private document corpus and get answers grounded in the actual content. The dominant hosted pattern — OpenAI Assistants + Pinecone — works, but every document chunk and every query travels to OpenAI's servers. For legal contracts, internal knowledge bases, or any non-public data, that is a compliance liability.

This guide deploys a fully self-hosted RAG stack: Dify as the orchestration and document management layer, Qdrant as the vector database, and Ollama serving both the embedding model and the chat LLM — all on a single Liquid Web VPS.

Many RAG (Retrieval-Augmented Generation) projects fail in production not because the technology doesn't work, but because teams skip the hard parts: chunking strategy, embedding model selection, retrieval quality measurement, and stale data management. Validate each step on your own corpus — field names, SDK versions, and Actor outputs change over time.

The pipeline: crawl websites → chunk intelligently → embed → store in vector DB → retrieve with reranking → generate answers with citations.

TL;DR:

Prerequisites:

• Python 3.10+ or Node.js 18+
• Apify account (sign up)
• Vector database (Qdrant Cloud free tier or self-hosted)
• LLM API key (Claude, GPT-4, or self-hosted Ollama)

A custom RAG MCP server gives Claude access to a private knowledge base built from your crawled web content. Firecrawl crawls URLs → markdown → chunk → embed → store in ChromaDB or pgvector. Your MCP server exposes a tool that queries the vector store and returns relevant chunks. Claude uses them to answer questions with your data. Start with Firecrawl.

Both Firecrawl and Jina Reader convert URLs to clean Markdown for LLMs — but their scope, pricing, and capabilities differ significantly.

TL;DR: Jina Reader is free and instant for single-URL lookups. Firecrawl is the production choice for full-site crawls, structured extraction, and RAG pipelines.

Retrieval Augmented Generation (RAG) lets your chatbot answer from your own documents instead of relying solely on the model's training data. This guide shows you how to build a fully local RAG pipeline: documents → embeddings → ChromaDB vector store → retrieval → Ollama LLM → answers. No cloud APIs. No per-token billing. Everything runs on your machine or a Liquid Web GPU VPS for heavier workloads.

Technical documentation is best consumed by Claude as clean markdown — no HTML tags, nav bars, or code-language headers. Firecrawl's crawl endpoint turns entire doc sites into LLM-ready markdown in minutes. Try Firecrawl for docs.

Use Firecrawl to crawl a site, extract markdown, chunk it, embed, and store in a vector database. Then retrieve relevant chunks for RAG. Pipeline: map → crawl → chunk → embed → vector DB → retriever. Firecrawl gives clean markdown; LangChain or LlamaIndex handle the rest.

Start Firecrawl RAG →

Firecrawl is a web scraping and crawling API that converts any website into clean, LLM-ready Markdown or structured JSON with a single API call. If you're building RAG pipelines, AI agents, or any application that needs web content without the headache of proxy management and JavaScript rendering, this review covers everything you need to decide whether Firecrawl is the right tool for your stack in 2026.

Feeding live web pages into an LLM or a RAG index sounds simple until you look at the token math. Raw HTML for a typical docs or product page can blow past 100,000 tokens once you count inline CSS, SVG, trackers, and nested <div>s—while the sentences you actually care about might sit closer to 3,000.

Shipping that HTML straight into inference burns latency and context budget, and it tends to bury the important lines—the familiar “lost in the middle” problem.

This guide compares the old playbook (sanitize HTML yourself) with API-first Markdown extraction aimed at chunking and embeddings, without changing the underlying tradeoffs.

Generative AI work is still gated by data quality and access. Pre-training needs very large, diverse text corpora. RAG needs fresh pages and documents so answers stay grounded. Agents need live web access when the task is not fully offline.

Bright Data began as a proxy vendor and now markets heavily to AI teams: curated datasets, an official MCP (Model Context Protocol) Server, and a cloud Browser API aimed at agent-style automation.

This guide outlines how those pieces fit together and how they compare to managed extraction platforms like Apify.