Scraping web data for AI, RAG, and LLM pipelines

Contains affiliate links. Disclosure.

Quick answer: Web scraping is the primary method to build RAG datasets. Apify's Website Content Crawler exports clean Markdown or HTML for embedding. Combine with Firecrawl for LLM-optimized crawling.

RAG answers user queries by pulling relevant chunks from a vector index into the prompt at inference time. Your retrieval quality is bounded by what you scraped, how you chunked, and whether you re-embed when the source changes. Apify covers the collect → structure → export half; you own the embed → store → refresh half.

This guide covers the Actors we actually use (Website Content Crawler, RAG Web Browser, MCP server), a scheduled re-embed pattern that doesn't blow up your embedding bill, and the full LangChain loader code. Actor details verified against the Apify Store, May 2026.

Start on Apify (affiliate link) →

---

What is RAG (in one minute)

RAG means: at question time, your system searches a vector index for text snippets related to the user query, injects them into the prompt, and the model answers using that context. Quality depends on retrieval, which depends on what you scraped, how you chunked, and what metadata you kept (especially source URLs for citations).

Approach	Best when	Updates
RAG	Docs, help centers, blogs, product pages; need citations	Re-crawl and re-embed on a schedule
Fine-tuning	Fixed style or task format; large labeled set	Costly to refresh
Pretraining	Foundation-model scale	Not a typical Apify workflow

For changing web content, RAG plus scheduled crawls is usually the right fit.

---

Why web scraping matters for RAG

• Coverage: Your knowledge base is only as good as the pages you ingest.
• Structure: Raw HTML is noisy; models do better with clean Markdown and stripped chrome.
• Freshness: Prices, policies, and docs change, so automated re-runs beat one-off exports.
• Governance: You need robots.txt, terms of use, and PII discipline before training or serving user-specific data.

Apify Actors output dataset rows you can feed straight into chunking and embedding jobs, or into LangChain / LlamaIndex loaders. For a side-by-side of the tools teams reach for, see the best AI data Actors roundup.

---

Recommended Apify Actors for RAG data

Actor	Role in RAG	Why teams use it
Website Content Crawler	Primary site and docs ingest	Crawls domains, outputs Markdown / text and metadata for vectors
RAG Web Browser	On-demand live web	Agent-style lookup when static index is not enough
Actors MCP Server	Tooling for agents	Exposes many Actors to MCP clients (see MCP guide)

For URL lists and LLM-first Markdown, Firecrawl is a strong complement: paste URLs, get rendered, cleaned Markdown with minimal setup.

For step-by-step setup of the primary crawler, see how to scrape website content with Apify.

Match the data type to the right Actor and output

Pick the Actor by what you are ingesting, then map its output format to your embedding step.

Data type	Recommended Actor	Output format	Goes into
Docs sites, help centers, blogs	Website Content Crawler	Clean Markdown / text + metadata	Chunker, then embeddings
Known list of URLs	Firecrawl	LLM-ready Markdown	Chunker, then embeddings
Live page at query time	RAG Web Browser	Cleaned text per request	Prompt context (no store)
Many niche sources via an agent	Actors MCP Server	Tool responses (JSON)	Agent tool calls

Markdown is the safer default for embeddings because it preserves headings and lists without HTML chrome. Reach for structured JSON when you need typed fields (price, author, date) as filterable metadata.

---

Pipeline overview: scrape → clean → embed → store

1. Scrape: Run Website Content Crawler with start URLs, max depth, and adaptive vs Playwright mode for JS-heavy sites. Alternatively, push URLs through Firecrawl when you already know the page list.
2. Clean: Deduplicate by URL and content hash, drop short pages, and filter language using crawler metadata when available.
3. Embed: Chunk (see below), call your embedding API (for example OpenAI text-embedding-3-small), attach source and title to every chunk.
4. Store: Upsert into Pinecone, Qdrant, Weaviate, Milvus, or self-hosted stores; optional native Apify integrations on the Actor when you use supported destinations.

The structured steps above are also expressed in <HowToSchema /> for search engines.

---

Chunking that actually helps retrieval

Strategy	Chunk size	Good for
Fixed window	500–1,500 tokens	General docs
Paragraph-first splits	Variable	Markdown from Website Content Crawler
Semantic / hierarchical	Parent + child	Long manuals

Practical defaults: chunk_size ~1000, overlap ~10–15%, split on \n\n first. Always keep source (URL) in metadata for citations in RAG answers.

---

Integration with LangChain

Use langchain_apify to map dataset rows to Document objects, then split and embed:

from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore

loader = ApifyDatasetLoader(
    dataset_id="YOUR_DATASET_ID",
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("markdown") or item.get("text") or "",
        metadata={"source": item.get("url"), "title": item.get("title")},
    ),
)
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(docs)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = PineconeVectorStore.from_documents(
    chunks, embeddings, index_name="YOUR_INDEX_NAME"
)

More detail: LangChain integration.

Incremental refresh (don't re-embed everything weekly)

Naive weekly full re-embed of a 10k-page docs site at text-embedding-3-small prices is cheap; at text-embedding-3-large it isn't. Cheaper pattern:

1. Store a contentHash (SHA-256 of item.markdown) alongside each Pinecone vector's metadata.
2. Every Monday, re-run Website Content Crawler on the same start URLs.
3. For each dataset row, compute contentHash and upsert only vectors where the hash changed or the URL is new. Delete vectors whose source URL no longer appears in the crawl.

import hashlib
def h(s: str) -> str:
    return hashlib.sha256(s.encode()).hexdigest()

current_urls = set()
for item in loader.load():
    url = item.metadata["source"]
    current_urls.add(url)
    new_hash = h(item.page_content)
    existing = index.fetch(ids=[url]).vectors.get(url)
    if existing and existing.metadata.get("contentHash") == new_hash:
        continue  # unchanged, skip embedding
    vec = embeddings.embed_query(item.page_content)
    index.upsert([(url, vec, {**item.metadata, "contentHash": new_hash})])

stale = [id for id in index.list(prefix="") if id not in current_urls]
if stale:
    index.delete(ids=stale)

Typical result on a docs site: 90–95% of URLs unchanged week-to-week, so embedding spend drops by the same factor.

---

Integration with LlamaIndex

LlamaIndex can read Apify datasets via the Apify dataset reader pattern (see current LlamaIndex docs for the ApifyDatasetLoader / reader module name in your version). The mapping is the same idea: markdown or text → Document text, url / title → metadata. Then build your VectorStoreIndex from those documents.

---

Firecrawl for LLM-optimized crawling

Firecrawl turns URLs into clean Markdown with JavaScript rendering and boilerplate removal, which maps well to embedding pipelines.

When to lean on Apify	When to add Firecrawl
Full-domain crawls, schedules, proxies, and many niche site Actors	Known URL batches, fast Markdown, minimal crawler config

You can mix both: Apify for deep crawls, Firecrawl for one-off pages or supplementary URLs.

---

Real-time and agent workflows

• RAG Web Browser: query or URL in, cleaned text out, useful when answers need live pages.
• MCP: Apify MCP Server lets compatible clients call Actors as tools (different from static RAG, but complementary).

---

Data governance checklist

• Respect robots.txt and site terms; prefer public documentation and marketing pages you have rights to use.
• Minimize personal data in corpora bound for training or analytics (GDPR / CCPA).
• Store provenance (URL, fetch time) for audit and citation.
• Deduplicate to reduce benchmark leakage and redundant chunks.

---

Data quality before you embed

• Dedupe URLs and near-duplicate bodies.
• Strip nav, cookie banners, and footers (crawler Markdown should already help).
• Language filter if the model is single-locale.
• Drop tiny pages (often errors or placeholders).
• Track crawl time for freshness SLAs.

If you also need the hosted chatbot layer on top of the corpus, compare the options in Chatbase Review and Chatbase vs Botpress / eesel AI / Intercom.

---

Related: Web scraping challenges · Apify for developers · MCP setup

Build your RAG dataset on Apify →

Frequently Asked Questions

Retrieval-Augmented Generation retrieves relevant text chunks from a vector store at query time and passes them to the LLM as context. Web scraping is how you populate that store from documentation, blogs, and product sites. Clean Markdown (from Apify Website Content Crawler or Firecrawl) improves chunk quality and reduces noise in retrieved context.

For many products, yes: you need up-to-date text from the public web or your own sites. Apify's Website Content Crawler exports clean Markdown or HTML for embedding, and Firecrawl is a strong option for LLM-oriented single-URL or batch Markdown extraction.

Start with Website Content Crawler for domain or docs crawls. Add RAG Web Browser when you need on-demand live pages, and consider the Apify MCP Server when your agent should call scrapers as tools. Pair with Firecrawl when you want minimal-config Markdown from explicit URLs.

Use langchain_apify's ApifyDatasetLoader to map dataset items to Documents, then chunk, embed, and push to your vector store. For LlamaIndex, use the Apify dataset reader for your version and build a VectorStoreIndex from the loaded documents. See our LangChain integration guide for step-by-step examples.

Scrape with Website Content Crawler or Firecrawl, normalize and dedupe text, chunk with overlap while preserving source metadata, embed each chunk, and upsert vectors into a database such as Pinecone or Qdrant. Refresh on a schedule so answers stay current.

Yes. Schedule Website Content Crawler to re-run on your start URLs (daily, weekly, or hourly), then re-embed only the pages whose content changed. Store a content hash with each vector, compare it on every run, and upsert just the new or modified pages while deleting vectors whose source URL disappeared. On a typical docs site, 90 to 95 percent of pages are unchanged week to week, so this keeps the corpus current without re-paying for embeddings you already have.

Markdown is the better default for free-form text like docs, blogs, and help articles because it keeps headings and lists intact while dropping HTML chrome, which makes cleaner chunks. Use structured JSON when you need typed fields (price, author, publish date, category) as filterable metadata on each vector, for example to restrict retrieval to recent or in-stock items. Many pipelines store the Markdown body as the embedded text and a few JSON fields alongside it as metadata.

Single-purpose extraction APIs like Firecrawl, ZenRows, and Scrapfly are quick when you already have a list of URLs and just want Markdown back. Apify covers that case too but adds full-domain crawling, scheduling, proxy rotation, native vector-store integrations, and a large library of site-specific Actors, so the same platform handles collection, refresh, and the long tail of awkward sources. We recommend Apify as the default and add Firecrawl for one-off or messy single URLs.

Apify is pay-per-use; crawl cost depends on pages and complexity. New accounts receive monthly free credits. Embedding APIs and vector hosting are billed separately; estimate both when budgeting.