Skip to main content

Scrape Docs to Markdown for Claude RAG (2026)

· 13 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

To give Claude reliable answers about a product's documentation, crawl the docs to clean Markdown, split the Markdown on its headings into chunks of a few hundred tokens, embed the chunks in a vector database, and pass only the matching chunks to Claude. This guide does each step with current code: Firecrawl or Apify's Website Content Crawler for the crawl, Markdown RAG Chunker for header-aware chunks, Chroma for storage and the Claude API for answers.

Why LLMs Need Docs in Markdown​

Claude and other LLMs perform better on structured text than raw HTML. Markdown preserves:

  • Hierarchy: headings mark where one topic ends and the next begins, which is exactly where chunks should split
  • Code blocks: syntax without DOM noise
  • Semantic density: far fewer tokens than the HTML for the same content
  • No boilerplate: navigation, footers, and cookie banners add noise and cost

Firecrawl strips boilerplate and returns Markdown by default. For RAG pipelines, this means higher retrieval quality and lower embedding costs. Our Firecrawl RAG guide covers full pipeline design.

Firecrawl Crawl for Documentation Sites​

Firecrawl's crawl() starts the job, waits until it finishes, and returns every page with its Markdown (.markdown) and URL (.metadata.source_url):

from firecrawl import Firecrawl, ScrapeOptions

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

job = firecrawl.crawl(
"https://docs.stripe.com/api",
limit=500,
include_paths=[r"^/api/.*"], # regex on the URL path
max_discovery_depth=3,
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True),
poll_interval=10,
)

pages = [(doc.metadata.source_url, doc.markdown) for doc in job.data if doc.markdown]
print(f"{job.completed}/{job.total} pages, {job.credits_used} credits")

include_paths and exclude_paths are regular expressions matched against the URL path, not globs, so /docs/** will not do what it looks like. max_discovery_depth counts link hops from the start URL (the start page and sitemap pages are depth 0). only_main_content is already on by default; it is set here to make the intent explicit.

For long crawls you don't want to block on, use start_crawl(...) with the same arguments and poll get_crawl_status(job.id) yourself.

Cost: a basic page costs 1 credit. Firecrawl's free plan includes 1,000 credits a month, and paid plans start at Hobby, $16 a month (billed annually) for 5,000 credits (pricing).

Handling Versioned Documentation​

Many doc sites (Stripe, React, FastAPI) use path-based versioning: /docs/v3/, /docs/v2/, and so on.

Strategy: start at the exact version URL and exclude the sibling versions:

job = firecrawl.crawl(
"https://docs.example.com/api/v3.2/",
limit=200,
include_paths=[r"^/api/v3\.2/.*"],
exclude_paths=[r"^/api/v2/.*", r"^/api/v4/.*"],
scrape_options=ScrapeOptions(formats=["markdown"]),
)

Escape the dots in version numbers (v3\.2), since . matches any character in a regex. Re-crawl when the docs update; the incremental updates section below shows how to skip unchanged pages.

Cleaning Output​

Firecrawl's main-content extraction removes navigation and footers. For extra control:

  • Strip frontmatter blocks: YAML frontmatter that some doc generators leave in the page can be removed before chunking
  • Normalize whitespace: collapse \n\n\n to \n\n so empty runs don't become near-empty chunks
  • Drop "Was this page helpful?" widgets and similar repeated footers with exclude_tags in the scrape options

For most technical docs, Firecrawl's default output is sufficient. See the Firecrawl API tutorial for advanced options.

Chunking Strategy for Claude​

Short answer: split on the document's headings first, then cap long sections at roughly 512 tokens, and keep the heading path with every chunk.

Current Claude models accept up to 1M tokens of context, but retrieval still works best with small, focused chunks: the vector search has to find the right passage before Claude ever sees it, and a chunk that mixes two topics matches neither well. Benchmarks from Chroma (2024) and NVIDIA (2025) found that chunks of a few hundred tokens retrieve best for documentation-style questions.

StrategyChunk sizeOverlapBest for
Heading-split, then token capSection, max ~512 tokens~10%, only inside long sectionsAPI reference, product docs
Token-aware recursive (tiktoken)~512 tokens50 tokensDocs without consistent headings
Sentence boundary~800 characters100 charactersProse-heavy guides

Option A: LangChain splitter (free, local)​

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=512, # tokens, because of from_tiktoken_encoder
chunk_overlap=50,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
chunks = splitter.split_text(markdown)

Use from_tiktoken_encoder: the plain RecursiveCharacterTextSplitter(chunk_size=...) constructor counts characters, so chunk_size=512 there is only about 130 tokens of English. The splitter also returns plain strings, so the heading path you want for filtering and citations has to be rebuilt yourself.

Option B: Markdown RAG Chunker (header-aware, on Apify)​

Markdown RAG Chunker is an Apify Actor that does the heading-split row of the table for you. It runs LangChain's MarkdownHeaderTextSplitter on #, ## and ###, then re-splits only the sections over 512 tokens (counted with tiktoken). Each chunk comes back with:

  • header_path, such as API Reference > Authentication
  • embedding_text: the header path plus the content, ready to embed
  • chunk_id: a stable ID scoped to the source URL, which stays the same on the next crawl if the section did not change
  • source, chunk_index, document_hash and an exact token_count
from apify_client import ApifyClient

apify = ApifyClient("YOUR_APIFY_TOKEN")
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": markdown, # one crawled page
"source_id": url, # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
chunks = list(apify.dataset(run.default_dataset_id).iterate_items())
# chunk["embedding_text"], chunk["header_path"], chunk["chunk_id"], chunk["token_count"]

Each run chunks one document, so call it once per page. Pricing is $0.80 per 1,000 chunks plus $0.001 per run on the free plan. API references benefit most, because every endpoint sits under its own heading.

LangChain splitterMarkdown RAG Chunker
CostFree$0.80 / 1,000 chunks + $0.001 / run
Respects section boundariesPrefers them, may crossAlways
Heading path per chunkNoYes (header_path)
Stable IDs for upsertsBuild your ownBuilt in (chunk_id)
Skips unchanged pagesBuild your owndocument_hash to compare, or Skip if unchanged
InputsText you pass inMarkdown, or a URL to HTML, PDF, DOCX, PPTX, XLSX, EPUB

Alternative: Apify Website Content Crawler​

Apify's Website Content Crawler produces similar Markdown. Use it when:

  • You need scheduling, webhooks, or Apify integrations
  • You already run Actors and want crawling and chunking on one platform and one bill
  • You prefer Crawlee-based extraction logic

Input: start URL, max pages, include/exclude patterns. Output: Markdown per page. Pass each page to Markdown RAG Chunker so crawling and chunking run on the same Apify schedule, or chain it with the LangChain Apify content pipeline for summarization.

Full Python Example​

End to end: Firecrawl crawl → header-aware chunks → Chroma → Claude answer with sources.

import hashlib
import os
from concurrent.futures import ThreadPoolExecutor

import chromadb
from anthropic import Anthropic
from apify_client import ApifyClient
from chromadb.utils import embedding_functions
from firecrawl import Firecrawl, ScrapeOptions

# 1. Crawl
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
job = firecrawl.crawl(
"https://docs.yourproduct.com",
limit=100,
max_discovery_depth=2,
scrape_options=ScrapeOptions(formats=["markdown"]),
)

# 2. Chunk only new or changed pages by headings (unchanged pages cost $0)
apify = ApifyClient(os.environ["APIFY_TOKEN"])
ef = embedding_functions.OpenAIEmbeddingFunction(
api_key=os.environ["OPENAI_API_KEY"], model_name="text-embedding-3-small"
)
coll = chromadb.PersistentClient(path="./chroma").get_or_create_collection(
"docs", embedding_function=ef
)

def page_hash(markdown): # the same value the chunker returns as document_hash
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]

indexed = {m["source"]: m["document_hash"] for m in coll.get(include=["metadatas"])["metadatas"]}
todo = [d for d in job.data
if d.markdown and indexed.get(d.metadata.source_url) != page_hash(d.markdown)]

def chunk_page(doc):
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": doc.markdown,
"source_id": doc.metadata.source_url,
"max_chunks": 600,
})
if run is None or run.status != "SUCCEEDED":
return []
return list(apify.dataset(run.default_dataset_id).iterate_items())

with ThreadPoolExecutor(max_workers=4) as pool: # parallel runs: same cost, less waiting
for items in pool.map(chunk_page, todo):
if not items:
continue
# 3. Upsert by chunk_id, then drop this page's chunks from older versions
coll.upsert(
ids=[c["chunk_id"] for c in items],
documents=[c["embedding_text"] for c in items],
metadatas=[{"source": c["source"], "section": c["header_path"],
"document_hash": c["document_hash"]} for c in items],
)
coll.delete(where={"$and": [
{"source": items[0]["source"]},
{"document_hash": {"$ne": items[0]["document_hash"]}},
]})

# 4. Retrieve and ask Claude
question = "How do I authenticate?"
hits = coll.query(query_texts=[question], n_results=5)
context = "\n\n".join(
f"[{m['source']} | {m['section']}]\n{d}"
for d, m in zip(hits["documents"][0], hits["metadatas"][0])
)

claude = Anthropic() # reads ANTHROPIC_API_KEY
msg = claude.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
messages=[{
"role": "user",
"content": f"Answer from these documentation excerpts and cite the source URL.\n\n{context}\n\nQuestion: {question}",
}],
)
print("".join(b.text for b in msg.content if b.type == "text"))

Install the latest stable versions (October 2026), which this example targets:

pip install firecrawl-py==4.53.2 apify-client==3.3.0 chromadb==1.5.9 anthropic==1.15.0 openai==3.35.1

To use Qdrant with hybrid (keyword + semantic) search instead of Chroma, see the Firecrawl + Qdrant tutorial and its companion repository on GitHub.

Comparison: Docs Crawling Tools​

AttributeFirecrawlApify Website Content CrawlerManual wget/curl
Markdown outputNative, cleanNativeRequires converter
Boilerplate removalAutomaticAutomaticManual
Rate limiting / proxiesBuilt inVia Apify ProxySelf-manage
Versioned docsPath regexesInclude/exclude patternsManual URL list
Change detectionchangeTracking formatScheduled runs + dataset diffManual
APIREST, Python/JS SDKApify API, Python/JS clientN/A
Cost1 credit per basic pagePay per usageFree (time cost)
Best forFast RAG setupScheduled pipelines on ApifyFull control

Fastest to a first result: Firecrawl, one call and Markdown out. Best for a pipeline that runs on a schedule: Apify, with Website Content Crawler and Markdown RAG Chunker on the same schedule.

Incremental Updates​

Docs change over time. To keep your RAG index fresh without re-embedding everything:

  • Firecrawl change tracking: add "changeTracking" to the scrape formats (alongside "markdown"). Each page then reports a changeStatus of new, same, changed or removed compared with your previous scrape, so you re-embed only new and changed pages. Basic change tracking costs no extra credits (docs).
  • Sitemap lastmod: if the doc site publishes <lastmod> dates, crawl only URLs whose date changed.
  • Content hashing: hash each page's Markdown and compare it with the document_hash already stored, as the full example above does. Unchanged pages never reach Markdown RAG Chunker or the embedding model, so they cost $0. When a page does change, sections you didn't edit keep their chunk_id, so upserting by that ID rewrites only the changed chunks; then delete the old chunks whose source matches and whose document_hash differs. On an Apify schedule with no local state, the chunker's Skip if unchanged option does the same check for the $0.001 run fee.
  • Schedule: run the crawl daily for changelogs and weekly for API reference, then push updates to the vector DB.

Use Cases​

  • RAG over Stripe docs: crawl the API reference, chunk, embed. Claude answers billing and API questions with sources.
  • Internal engineering wiki: crawl a public export, or chunk exported PDFs and Word files directly with Markdown RAG Chunker, which accepts document URLs.
  • API reference for a Claude coding agent: index the reference docs so the agent retrieves parameters and examples while it writes code.
  • Competitor feature tracking: crawl competitor changelogs with change tracking on, and review only what changed.
Start with a small crawl

Crawl 20–50 pages first, chunk them, and ask Claude ten real questions before scaling up. If answers cite the wrong section, fix chunking before anything else.



Try Markdown RAG Chunker | Try Firecrawl | Firecrawl RAG Pipeline

Frequently Asked Questions

Markdown is semantically dense: far fewer tokens than HTML, with the structure (headings, code blocks) preserved. HTML adds navigation, footers and tag overhead. Headings also give you natural chunk boundaries for retrieval.

Firecrawl can send custom headers (for example a session cookie) with its scrape options, but it cannot complete an interactive login for you. For login flows, use an Apify Actor that signs in first, or export the docs and chunk the exported files directly.

Split on headings, then cap sections at about 512 tokens with a small overlap. Use around 256 tokens for short factual lookups and up to 1,024 for long analytical answers. Test retrieval on your own questions before tuning further.

Not for documentation sites. Sending a whole doc set on every question costs far more and still dilutes the relevant passage. Retrieval with small, focused chunks keeps each request cheap and puts the right section in front of Claude.

Firecrawl is faster to set up: one REST call returns Markdown. Apify is better for scheduled pipelines: Website Content Crawler and Markdown RAG Chunker run on one schedule, with webhooks and storage included. Both output Markdown.

Yes. Add the changeTracking format to a Firecrawl crawl to get a changeStatus (new, same, changed or removed) per page, or compare each page's Markdown hash with the stored document_hash so unchanged pages are never re-chunked or re-embedded.

Common mistakes and fixes

Crawl returns partial or empty markdown

Check your include_paths regex matches the doc URLs, raise limit, and confirm the pages are public (no login wall). only_main_content is on by default.

Versioned docs include wrong version

Start the crawl at the version URL (e.g. /api/v3.2/) and add exclude_paths regexes for the other versions.

Chunks mix two unrelated sections

Split on headings before splitting by size. Markdown RAG Chunker does this and records the heading path on every chunk.

Advertise on this page