Skip to main content

Architecting RAG Ingestion Pipelines: DOM-to-Vector (2026)

· 4 min read
Yassine El Haddad
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

Large language models (the latest GPT and Claude Sonnet 4.6 models) have two practical limits you hit in production:

  1. Stale knowledge: Weights reflect the world only up to their training cutoff.
  2. No access to your data: They do not know your internal wikis, repos, or live systems unless you give them a path in.

Retrieval-Augmented Generation (RAG) is the usual fix. Instead of retraining or fine-tuning the model for every document change, RAG takes the user question, looks up relevant passages in an external store of facts (typically a vector database), and asks the LLM to answer using that retrieved text as ground context.

This guide focuses on the part that breaks most often: ingestion—getting web pages into a shape embeddings can use.

The ingestion bottleneck: noisy HTML​

For RAG you turn many URLs (for example a whole support site) into embeddings stored in something like Pinecone or Qdrant.

Where it goes wrong (bloat and bad flattening):
A common pattern is: HTTP GET the page, pass the raw <html> straight to the embedding API.

  • Wasted tokens: Modern pages are heavy on <svg>, CSS classes, inline scripts, and framework markup. A lot of that adds little meaning but eats context window.
  • Lost structure: Naive text extraction (e.g. BeautifulSoup.get_text()) can collapse tables and headings into one blob. A pricing <table> becomes hard to match at retrieval time.

A practical fix: clean Markdown​

Embeddings work better when the extractor turns the DOM into Markdown, which keeps hierarchy (# Headers, | Tables |, [Links]) and drops most presentational noise.

You can build DOM-to-Markdown yourself in Python, or use the Website Content Crawler on Apify’s serverless stack so you are not maintaining that path alone.

Running the crawl​

  1. Create an Apify API token.
  2. Point the Actor at a root domain (e.g. docs.stripe.com) and set crawl depth as needed.

Apify runs the crawl in its environment (WAF handling, headless rendering where needed), trims boilerplate like nav and cookie banners, and returns Markdown-oriented items in a dataset, for example:

[
  {
    "url": "https://docs.target.com/api/authentication",
    "markdown": "# API Authentication\nAll requests must include the `Bearer` token.\n\n### Error Codes\n| Code | Reason |\n|---|---|\n| 401 | Invalid Token |\n| 429 | Rate Limit Exceeded |"
  }
]

The Complete RAG Execution Flow​

Once the Markdown is extracted natively, the pipeline sequences predictably across standard Python integrations.

Step 1: Chunking the Input​

LLMs cannot embed an entire 40-page technical manual accurately in a singular tensor array. The Markdown string must be algorithmically fractured into Semantic Chunks.

# Example: split on Markdown headers (Python 3.12 + LangChain)
from langchain.text_splitter import MarkdownHeaderTextSplitter

# Keeps sections intact where possible
headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

blocks = [item['markdown'] for item in apify_dataset]
md_chunks = markdown_splitter.split_text("\n\n".join(blocks))

Step 2: Embeddings​

Send each chunk to an embedding API (e.g. text-embedding-3-small) to get a vector per chunk.

Step 3: Vector storage​

Upsert vectors with metadata (at least source URL) into Pinecone, Weaviate, or similar.

At query time​

For a question like "What happens if I exceed API limits?", embed the question, search the vector DB (e.g. cosine similarity), pull the top chunks—such as the table row | 429 | Rate Limit Exceeded | from your crawled Markdown—and pass only that text (plus the question) to the LLM.

Apify Affiliate Banner 728x90Apify Affiliate Banner 728x90Apify Affiliate Banner 300x50Apify Affiliate Banner 300x50

Conclusion​

Most RAG failures trace back to ingestion, not the model. Raw HTML or sloppy “strip tags and pray” text makes retrieval unreliable and invites hallucination when the model has no usable passages.

Running HTML-to-Markdown through a crawler built for that job—such as Apify with the Website Content Crawler—gives your vector DB a cleaner, more structured base to search.

Frequently Asked Questions

One giant embedding for a whole page mixes unrelated topics (auth, pricing, footers). Vectors get vague. Splitting on Markdown headers (for example `## Authentication`) keeps each vector closer to one idea, so similarity search can return the right paragraph.

Plain stripping often wrecks tables and lists. For pricing grids you usually want structured Markdown (or another stable format) so rows and columns stay aligned; otherwise you get ambiguous text the model cannot reliably ground answers in.