use-apify.com
Vector database: guides & tutorials
Turn crawled pages into searchable vectors: chunking, embeddings, and sync to Pinecone, Qdrant, or pgvector. Apify pairs extraction with RAG-ready storage.
3 articles
View all tags
Vector databases store crawled content as embeddings so LLMs can retrieve it by meaning, not just keywords. These guides cover chunking, embedding, and syncing data to Pinecone, Qdrant, or pgvector.
A good vector store is only as useful as the pipeline feeding it, which is where clean, current crawls matter. Apify pairs extraction with RAG-ready output. Below you will find tutorials on building and maintaining vector search backends.
TL;DR
- One
docker-compose.yml: Dify (API + worker + web + sandbox) + Qdrant + Ollama + PostgreSQL + Redis + Caddy
- Measured idle RAM: ~7.4 GB (Llama 3 8B loaded, CPU inference); minimum server: 16 GB VPS
- Replaces: OpenAI Assistants API ($0.03/1k tokens) + Pinecone Starter ($70/mo) with a fully local, zero-egress alternative
- Your documents never leave the server — not during upload, not during embedding, not during inference
Retrieval-Augmented Generation (RAG) lets you ask questions against a private document corpus and get answers grounded in the actual content. The dominant hosted pattern — OpenAI Assistants + Pinecone — works, but every document chunk and every query travels to OpenAI's servers. For legal contracts, internal knowledge bases, or any non-public data, that is a compliance liability.
This guide deploys a fully self-hosted RAG stack: Dify as the orchestration and document management layer, Qdrant as the vector database, and Ollama serving both the embedding model and the chat LLM — all on a single Liquid Web VPS.
Large language models (the latest GPT and Claude Sonnet 4.6 models) have two practical limits you hit in production:
- Stale knowledge: Weights reflect the world only up to their training cutoff.
- No access to your data: They do not know your internal wikis, repos, or live systems unless you give them a path in.
Retrieval-Augmented Generation (RAG) is the usual fix. Instead of retraining or fine-tuning the model for every document change, RAG takes the user question, looks up relevant passages in an external store of facts (typically a vector database), and asks the LLM to answer using that retrieved text as ground context.
This guide focuses on the part that breaks most often: ingestion—getting web pages into a shape embeddings can use.
Retrieval-Augmented Generation (RAG) is entirely dependent on the structural integrity of its underlying Vector Database.
Attempting to hydrate a Pinecone index by recursively dumping raw, unparsed HTML <div> nodes into an embedding model guarantees catastrophic semantic hallucination upon retrieval.
In 2026, the data ingestion pipeline is the critical engineering bottleneck. This guide details the explicit architectural flow required to extract complex JavaScript Single-Page Applications (SPAs), mathematically flatten the DOM into clean Markdown, and synchronize the vectors utilizing the Apify Serverless Infrastructure.
