Build a Local RAG Chatbot with Ollama and ChromaDB: No Cloud Required (2026)

Contains affiliate links. Disclosure.

Retrieval Augmented Generation (RAG) lets your chatbot answer from your own documents instead of relying solely on the model's training data. This guide shows you how to build a fully local RAG pipeline: documents → embeddings → ChromaDB vector store → retrieval → Ollama LLM → answers. No cloud APIs. No per-token billing. Everything runs on your machine or a Liquid Web GPU VPS for heavier workloads.

Why RAG beats pure LLM for custom data: A raw LLM has no access to your PDFs, wikis, or internal docs. RAG injects relevant chunks at query time, grounding answers in your content while reducing hallucination. For knowledge bases, support docs, and proprietary data, RAG is the standard approach.

Deploy on a Liquid Web GPU server → | Apify for web scraping your docs →

What Is RAG? Architecture Overview

Retrieval Augmented Generation means: when a user asks a question, you first retrieve relevant chunks from a vector store, then pass those chunks + the question to the LLM. The model answers using the retrieved context instead of guessing from training data alone.

Pipeline flow:

1. Documents — PDF, TXT, Markdown, or scraped web pages
2. Chunk — Split into ~500–1000 character segments with overlap
3. Embed — Convert each chunk to a vector (Ollama nomic-embed-text)
4. Store — Save vectors + metadata in ChromaDB
5. Query — User question → embed query → similarity search → top-k chunks
6. Generate — Build prompt: Context: {chunks}\n\nQuestion: {query} → Ollama → answer

Prerequisites

• Python 3.11+
• Ollama running locally with:
  • llama3.2 (or similar) for generation
  • nomic-embed-text for embeddings
• pip packages: chromadb, langchain, langchain-community

ollama pull llama3.2
ollama pull nomic-embed-text
pip install chromadb langchain langchain-community

For production workloads or large document sets, a Liquid Web GPU VPS provides enough VRAM for both embedding and generation without slowdown.

---

Step 1: Load Documents

Use LangChain document loaders to load PDF, TXT, or Markdown files:

from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    UnstructuredMarkdownLoader,
)
from pathlib import Path

def load_documents(directory: str):
    loaders = {
        ".pdf": PyPDFLoader,
        ".txt": TextLoader,
        ".md": UnstructuredMarkdownLoader,
    }
    documents = []
    path = Path(directory)
    for file_path in path.rglob("*"):
        if file_path.suffix.lower() in loaders:
            loader = loaders[file_path.suffix.lower()](str(file_path))
            docs = loader.load()
            for doc in docs:
                doc.metadata["source"] = str(file_path)
            documents.extend(docs)
    return documents

docs = load_documents("./my_docs")
print(f"Loaded {len(docs)} document chunks")

For web pages, use Firecrawl or Apify to scrape and convert to Markdown, then load with the same pattern.

---

Step 2: Split Into Chunks

RecursiveCharacterTextSplitter keeps semantic boundaries (paragraphs, sentences) when possible:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_documents(docs)
print(f"Created {len(chunks)} chunks")

Tune chunk_size for your content: 500–800 for dense technical docs, 1200–1500 for narrative text.

---

Step 3: Embed with Ollama and Store in ChromaDB

Ollama's nomic-embed-text produces 768-dimensional embeddings. ChromaDB persists them locally:

from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OllamaEmbeddings(model="nomic-embed-text")

vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
    collection_name="my_rag_collection",
)

vector_store.persist()
print("Vector store persisted to ./chroma_db")

ChromaDB writes to ./chroma_db by default. For a fresh run, delete that folder first if you want to rebuild the collection.

---

Step 4: Build the Retrieval Chain

Query → retrieve top chunks → build prompt → call Ollama:

from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

llm = Ollama(model="llama3.2", temperature=0)

prompt_template = """Use the following context to answer the question.
If the context does not contain the answer, say "I don't have enough context."

Context:
{context}

Question: {question}

Answer:"""

PROMPT = PromptTemplate(
    template=prompt_template,
    input_variables=["context", "question"],
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vector_store.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT},
)

response = qa_chain.invoke({"query": "What is the refund policy?"})
print(response["result"])
# Optionally inspect retrieved chunks: response["source_documents"]

search_kwargs={"k": 5} returns the top 5 most similar chunks. Increase for broader context; decrease for focused answers.

---

Full End-to-End Python Script

Here is a complete, runnable script:

from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

DOCS_DIR = "./my_docs"
PERSIST_DIR = "./chroma_db"
COLLECTION = "rag_chatbot"

def main():
    # 1. Load docs
    loaders = {".pdf": PyPDFLoader, ".txt": TextLoader}
    docs = []
    for f in Path(DOCS_DIR).rglob("*"):
        if f.suffix.lower() in loaders:
            loader = loaders[f.suffix.lower()](str(f))
            loaded = loader.load()
            for d in loaded:
                d.metadata["source"] = str(f)
            docs.extend(loaded)

    if not docs:
        print("No documents found. Add PDF/TXT files to ./my_docs")
        return

    # 2. Chunk
    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
    chunks = splitter.split_documents(docs)

    # 3. Embed & store
    embeddings = OllamaEmbeddings(model="nomic-embed-text")
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=PERSIST_DIR,
        collection_name=COLLECTION,
    )
    vector_store.persist()

    # 4. Chain
    llm = Ollama(model="llama3.2", temperature=0)
    qa = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=vector_store.as_retriever(search_kwargs={"k": 5}),
        return_source_documents=True,
    )

    # 5. Query
    query = "What are the main topics covered?"
    result = qa.invoke({"query": query})
    print(f"Q: {query}\nA: {result['result']}\n")
    print("Sources:", [d.metadata.get("source") for d in result["source_documents"]])

if __name__ == "__main__":
    main()

Create ./my_docs, drop in a few PDFs or TXT files, and run. Adjust paths and model names as needed.

---

Testing with Sample Documents

Add a sample my_docs/faq.txt:

Refund policy: Full refund within 30 days. Contact support@example.com.
Shipping: 3–5 business days in the US. International: 7–14 days.

Run the script and ask: "What is the refund policy?" The retriever should surface that chunk and Ollama should answer from it.

---

Local ChromaDB vs Pinecone vs pgvector for RAG

Feature	ChromaDB (Local)	Pinecone	pgvector
Hosting	Local disk	Managed cloud	Self-hosted PostgreSQL
Setup	pip install chromadb	API key, index creation	Postgres extension
Scale	Millions of vectors on SSD	Billions, auto-scaling	Depends on Postgres tier
Latency	Sub-ms (local)	Low (global endpoints)	Low (same DB as app)
Cost	Free	Paid tiers	Postgres hosting cost
Best for	Prototyping, local RAG, small teams	Production at scale	Existing Postgres stack

Choose ChromaDB when you want zero cloud dependency and fast iteration. For production scale and multi-region, consider Pinecone or pgvector. See Firecrawl RAG knowledge base for ingestion pipelines that feed any of these stores. For a chat interface on top of Ollama, pair this RAG setup with Ollama + Open WebUI; for a multi-platform assistant, see OpenClaw + Ollama.

Start small

Begin with a handful of documents. Add more sources (PDFs, web crawls via Firecrawl or Apify) once retrieval quality looks good. Liquid Web GPU VPS handles larger collections and faster inference.

Frequently Asked Questions

Use nomic-embed-text: ollama pull nomic-embed-text. It produces 768-dim vectors and works well for semantic search. Ollama also supports other embedding models; nomic-embed-text is the recommended default.

Yes. Replace OllamaEmbeddings with OpenAIEmbeddings and pass your API key. The rest of the pipeline (chunking, ChromaDB, retrieval chain) stays the same. Local Ollama embeddings avoid API costs.

Start with k=5. Increase to 7–10 for complex questions or longer context windows. Too many chunks can dilute relevance and exceed the LLM context limit.

LangChain's PyPDFLoader extracts text only. For PDFs with images or complex layouts, use UnstructuredIO or Apache Tika. For scanned PDFs, add OCR (e.g. pytesseract) before chunking.

Load Chroma with persist_directory and collection_name, then call vector_store.add_documents(new_chunks). ChromaDB supports incremental adds without rebuilding the full index.

Check chunk size (500–1000 chars often works better than 2000+), embedding model (nomic-embed-text is strong for English), and metadata filters. Ensure your query is embedded with the same model used for ingestion.