Scrape Docs to Markdown for Claude RAG (2026)
To give Claude reliable answers about a product's documentation, crawl the docs to clean Markdown, split the Markdown on its headings into chunks of a few hundred tokens, embed the chunks in a vector database, and pass only the matching chunks to Claude. This guide does each step with current code: Firecrawl or Apify's Website Content Crawler for the crawl, Markdown RAG Chunker for header-aware chunks, Chroma for storage and the Claude API for answers.
Why LLMs Need Docs in Markdown
Claude and other LLMs perform better on structured text than raw HTML. Markdown preserves:
- Hierarchy: headings mark where one topic ends and the next begins, which is exactly where chunks should split
- Code blocks: syntax without DOM noise
- Semantic density: far fewer tokens than the HTML for the same content
- No boilerplate: navigation, footers, and cookie banners add noise and cost
Firecrawl strips boilerplate and returns Markdown by default. For RAG pipelines, this means higher retrieval quality and lower embedding costs. Our Firecrawl RAG guide covers full pipeline design.
Firecrawl Crawl for Documentation Sites
Firecrawl's crawl() starts the job, waits until it finishes, and returns every page with its Markdown (.markdown) and URL (.metadata.source_url):
from firecrawl import Firecrawl, ScrapeOptions
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
job = firecrawl.crawl(
"https://docs.stripe.com/api",
limit=500,
include_paths=[r"^/api/.*"], # regex on the URL path
max_discovery_depth=3,
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True),
poll_interval=10,
)
pages = [(doc.metadata.source_url, doc.markdown) for doc in job.data if doc.markdown]
print(f"{job.completed}/{job.total} pages, {job.credits_used} credits")
include_paths and exclude_paths are regular expressions matched against the URL path, not globs, so /docs/** will not do what it looks like. max_discovery_depth counts link hops from the start URL (the start page and sitemap pages are depth 0). only_main_content is already on by default; it is set here to make the intent explicit.
For long crawls you don't want to block on, use start_crawl(...) with the same arguments and poll get_crawl_status(job.id) yourself.
Cost: a basic page costs 1 credit. Firecrawl's free plan includes 1,000 credits a month, and paid plans start at Hobby, $16 a month (billed annually) for 5,000 credits (pricing).
Handling Versioned Documentation
Many doc sites (Stripe, React, FastAPI) use path-based versioning: /docs/v3/, /docs/v2/, and so on.
Strategy: start at the exact version URL and exclude the sibling versions:
job = firecrawl.crawl(
"https://docs.example.com/api/v3.2/",
limit=200,
include_paths=[r"^/api/v3\.2/.*"],
exclude_paths=[r"^/api/v2/.*", r"^/api/v4/.*"],
scrape_options=ScrapeOptions(formats=["markdown"]),
)
Escape the dots in version numbers (v3\.2), since . matches any character in a regex. Re-crawl when the docs update; the incremental updates section below shows how to skip unchanged pages.
Cleaning Output
Firecrawl's main-content extraction removes navigation and footers. For extra control:
- Strip frontmatter blocks: YAML frontmatter that some doc generators leave in the page can be removed before chunking
- Normalize whitespace: collapse
\n\n\nto\n\nso empty runs don't become near-empty chunks - Drop "Was this page helpful?" widgets and similar repeated footers with
exclude_tagsin the scrape options
For most technical docs, Firecrawl's default output is sufficient. See the Firecrawl API tutorial for advanced options.
Chunking Strategy for Claude
Short answer: split on the document's headings first, then cap long sections at roughly 512 tokens, and keep the heading path with every chunk.
Current Claude models accept up to 1M tokens of context, but retrieval still works best with small, focused chunks: the vector search has to find the right passage before Claude ever sees it, and a chunk that mixes two topics matches neither well. Benchmarks from Chroma (2024) and NVIDIA (2025) found that chunks of a few hundred tokens retrieve best for documentation-style questions.
| Strategy | Chunk size | Overlap | Best for |
|---|---|---|---|
| Heading-split, then token cap | Section, max ~512 tokens | ~10%, only inside long sections | API reference, product docs |
| Token-aware recursive (tiktoken) | ~512 tokens | 50 tokens | Docs without consistent headings |
| Sentence boundary | ~800 characters | 100 characters | Prose-heavy guides |
Option A: LangChain splitter (free, local)
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=512, # tokens, because of from_tiktoken_encoder
chunk_overlap=50,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
chunks = splitter.split_text(markdown)
Use from_tiktoken_encoder: the plain RecursiveCharacterTextSplitter(chunk_size=...) constructor counts characters, so chunk_size=512 there is only about 130 tokens of English. The splitter also returns plain strings, so the heading path you want for filtering and citations has to be rebuilt yourself.
Option B: Markdown RAG Chunker (header-aware, on Apify)
Markdown RAG Chunker is an Apify Actor that does the heading-split row of the table for you. It runs LangChain's MarkdownHeaderTextSplitter on #, ## and ###, then re-splits only the sections over 512 tokens (counted with tiktoken). Each chunk comes back with:
header_path, such asAPI Reference > Authenticationembedding_text: the header path plus the content, ready to embedchunk_id: a stable ID scoped to the source URL, which stays the same on the next crawl if the section did not changesource,chunk_index,document_hashand an exacttoken_count
from apify_client import ApifyClient
apify = ApifyClient("YOUR_APIFY_TOKEN")
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": markdown, # one crawled page
"source_id": url, # scopes chunk IDs to this page
"max_chunks": 600, # budget guard: an oversized page fails for $0.001
})
chunks = list(apify.dataset(run.default_dataset_id).iterate_items())
# chunk["embedding_text"], chunk["header_path"], chunk["chunk_id"], chunk["token_count"]
Each run chunks one document, so call it once per page. Pricing is $0.80 per 1,000 chunks plus $0.001 per run on the free plan. API references benefit most, because every endpoint sits under its own heading.
| LangChain splitter | Markdown RAG Chunker | |
|---|---|---|
| Cost | Free | $0.80 / 1,000 chunks + $0.001 / run |
| Respects section boundaries | Prefers them, may cross | Always |
| Heading path per chunk | No | Yes (header_path) |
| Stable IDs for upserts | Build your own | Built in (chunk_id) |
| Skips unchanged pages | Build your own | document_hash to compare, or Skip if unchanged |
| Inputs | Text you pass in | Markdown, or a URL to HTML, PDF, DOCX, PPTX, XLSX, EPUB |
Alternative: Apify Website Content Crawler
Apify's Website Content Crawler produces similar Markdown. Use it when:
- You need scheduling, webhooks, or Apify integrations
- You already run Actors and want crawling and chunking on one platform and one bill
- You prefer Crawlee-based extraction logic
Input: start URL, max pages, include/exclude patterns. Output: Markdown per page. Pass each page to Markdown RAG Chunker so crawling and chunking run on the same Apify schedule, or chain it with the LangChain Apify content pipeline for summarization.
Full Python Example
End to end: Firecrawl crawl → header-aware chunks → Chroma → Claude answer with sources.
import hashlib
import os
from concurrent.futures import ThreadPoolExecutor
import chromadb
from anthropic import Anthropic
from apify_client import ApifyClient
from chromadb.utils import embedding_functions
from firecrawl import Firecrawl, ScrapeOptions
# 1. Crawl
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
job = firecrawl.crawl(
"https://docs.yourproduct.com",
limit=100,
max_discovery_depth=2,
scrape_options=ScrapeOptions(formats=["markdown"]),
)
# 2. Chunk only new or changed pages by headings (unchanged pages cost $0)
apify = ApifyClient(os.environ["APIFY_TOKEN"])
ef = embedding_functions.OpenAIEmbeddingFunction(
api_key=os.environ["OPENAI_API_KEY"], model_name="text-embedding-3-small"
)
coll = chromadb.PersistentClient(path="./chroma").get_or_create_collection(
"docs", embedding_function=ef
)
def page_hash(markdown): # the same value the chunker returns as document_hash
return hashlib.sha256(markdown.encode("utf-8")).hexdigest()[:16]
indexed = {m["source"]: m["document_hash"] for m in coll.get(include=["metadatas"])["metadatas"]}
todo = [d for d in job.data
if d.markdown and indexed.get(d.metadata.source_url) != page_hash(d.markdown)]
def chunk_page(doc):
run = apify.actor("codepoetry/markdown-rag-chunker").call(run_input={
"markdown_text": doc.markdown,
"source_id": doc.metadata.source_url,
"max_chunks": 600,
})
if run is None or run.status != "SUCCEEDED":
return []
return list(apify.dataset(run.default_dataset_id).iterate_items())
with ThreadPoolExecutor(max_workers=4) as pool: # parallel runs: same cost, less waiting
for items in pool.map(chunk_page, todo):
if not items:
continue
# 3. Upsert by chunk_id, then drop this page's chunks from older versions
coll.upsert(
ids=[c["chunk_id"] for c in items],
documents=[c["embedding_text"] for c in items],
metadatas=[{"source": c["source"], "section": c["header_path"],
"document_hash": c["document_hash"]} for c in items],
)
coll.delete(where={"$and": [
{"source": items[0]["source"]},
{"document_hash": {"$ne": items[0]["document_hash"]}},
]})
# 4. Retrieve and ask Claude
question = "How do I authenticate?"
hits = coll.query(query_texts=[question], n_results=5)
context = "\n\n".join(
f"[{m['source']} | {m['section']}]\n{d}"
for d, m in zip(hits["documents"][0], hits["metadatas"][0])
)
claude = Anthropic() # reads ANTHROPIC_API_KEY
msg = claude.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
messages=[{
"role": "user",
"content": f"Answer from these documentation excerpts and cite the source URL.\n\n{context}\n\nQuestion: {question}",
}],
)
print("".join(b.text for b in msg.content if b.type == "text"))
Install the latest stable versions (October 2026), which this example targets:
pip install firecrawl-py==4.53.2 apify-client==3.3.0 chromadb==1.5.9 anthropic==1.15.0 openai==3.35.1
To use Qdrant with hybrid (keyword + semantic) search instead of Chroma, see the Firecrawl + Qdrant tutorial and its companion repository on GitHub.
Comparison: Docs Crawling Tools
| Attribute | Firecrawl | Apify Website Content Crawler | Manual wget/curl |
|---|---|---|---|
| Markdown output | Native, clean | Native | Requires converter |
| Boilerplate removal | Automatic | Automatic | Manual |
| Rate limiting / proxies | Built in | Via Apify Proxy | Self-manage |
| Versioned docs | Path regexes | Include/exclude patterns | Manual URL list |
| Change detection | changeTracking format | Scheduled runs + dataset diff | Manual |
| API | REST, Python/JS SDK | Apify API, Python/JS client | N/A |
| Cost | 1 credit per basic page | Pay per usage | Free (time cost) |
| Best for | Fast RAG setup | Scheduled pipelines on Apify | Full control |
Fastest to a first result: Firecrawl, one call and Markdown out. Best for a pipeline that runs on a schedule: Apify, with Website Content Crawler and Markdown RAG Chunker on the same schedule.
Incremental Updates
Docs change over time. To keep your RAG index fresh without re-embedding everything:
- Firecrawl change tracking: add
"changeTracking"to the scrape formats (alongside"markdown"). Each page then reports achangeStatusofnew,same,changedorremovedcompared with your previous scrape, so you re-embed onlynewandchangedpages. Basic change tracking costs no extra credits (docs). - Sitemap lastmod: if the doc site publishes
<lastmod>dates, crawl only URLs whose date changed. - Content hashing: hash each page's Markdown and compare it with the
document_hashalready stored, as the full example above does. Unchanged pages never reach Markdown RAG Chunker or the embedding model, so they cost $0. When a page does change, sections you didn't edit keep theirchunk_id, so upserting by that ID rewrites only the changed chunks; then delete the old chunks whosesourcematches and whosedocument_hashdiffers. On an Apify schedule with no local state, the chunker's Skip if unchanged option does the same check for the $0.001 run fee. - Schedule: run the crawl daily for changelogs and weekly for API reference, then push updates to the vector DB.
Use Cases
- RAG over Stripe docs: crawl the API reference, chunk, embed. Claude answers billing and API questions with sources.
- Internal engineering wiki: crawl a public export, or chunk exported PDFs and Word files directly with Markdown RAG Chunker, which accepts document URLs.
- API reference for a Claude coding agent: index the reference docs so the agent retrieves parameters and examples while it writes code.
- Competitor feature tracking: crawl competitor changelogs with change tracking on, and review only what changed.
Crawl 20–50 pages first, chunk them, and ask Claude ten real questions before scaling up. If answers cite the wrong section, fix chunking before anything else.
Try Markdown RAG Chunker | Try Firecrawl | Firecrawl RAG Pipeline
Markdown is semantically dense: far fewer tokens than HTML, with the structure (headings, code blocks) preserved. HTML adds navigation, footers and tag overhead. Headings also give you natural chunk boundaries for retrieval.
Firecrawl can send custom headers (for example a session cookie) with its scrape options, but it cannot complete an interactive login for you. For login flows, use an Apify Actor that signs in first, or export the docs and chunk the exported files directly.
Split on headings, then cap sections at about 512 tokens with a small overlap. Use around 256 tokens for short factual lookups and up to 1,024 for long analytical answers. Test retrieval on your own questions before tuning further.
Not for documentation sites. Sending a whole doc set on every question costs far more and still dilutes the relevant passage. Retrieval with small, focused chunks keeps each request cheap and puts the right section in front of Claude.
Firecrawl is faster to set up: one REST call returns Markdown. Apify is better for scheduled pipelines: Website Content Crawler and Markdown RAG Chunker run on one schedule, with webhooks and storage included. Both output Markdown.
Yes. Add the changeTracking format to a Firecrawl crawl to get a changeStatus (new, same, changed or removed) per page, or compare each page's Markdown hash with the stored document_hash so unchanged pages are never re-chunked or re-embedded.
