Scraping Technical Documentation to Markdown for Claude Projects (2026)

Contains affiliate links. Disclosure.

Technical documentation is best consumed by Claude as clean markdown — no HTML tags, nav bars, or code-language headers. Firecrawl's crawl endpoint turns entire doc sites into LLM-ready markdown in minutes. Try Firecrawl for docs.

Why LLMs Need Docs in Markdown

Claude and other LLMs perform better on structured text than raw HTML. Markdown preserves:

• Hierarchy — Headings define chunk boundaries for retrieval
• Code blocks — Syntax without DOM noise
• Semantic density — ~3x fewer tokens than HTML for the same content
• No boilerplate — Navigation, footers, and ads add noise and cost

Firecrawl strips boilerplate and returns markdown by default. For RAG pipelines, this means higher retrieval quality and lower embedding costs. Our Firecrawl RAG guide covers full pipeline design.

Firecrawl Crawl for Documentation Sites

Use the crawl_url endpoint with max_depth: 3, onlyMainContent: true, and path filters to scope documentation:

from firecrawl import Firecrawl
import time

app = Firecrawl(api_key="fc-YOUR-API-KEY")
job = app.crawl_url(
    "https://docs.stripe.com",
    limit=500,
    max_depth=3,
    include_paths=["/docs/**"],
    scrape_options={"formats": ["markdown"], "onlyMainContent": True}
)
job_id = job.get("id")
while True:
    status = app.check_crawl_status(job_id)
    if status.get("status") in ("completed", "failed"):
        break
    time.sleep(10)
data = app.get_crawl_data(job_id) if status.get("status") == "completed" else []

max_depth: 3 limits link traversal so you don't crawl unrelated marketing pages. include_paths restricts to the docs subdomain. For versioned docs, target the specific version URL.

Handling Versioned Documentation

Many doc sites (Stripe, React, FastAPI) use path-based versioning: /docs/v3/, /docs/v2/, etc.

Strategy: Crawl the exact version URL and exclude sibling versions:

job = app.crawl_url(
    "https://docs.example.com/api/v3.2/",
    limit=200,
    include_paths=["/api/v3.2/**"],
    exclude_paths=["/api/v2/**", "/api/v4/**"],
    scrape_options={"formats": ["markdown"], "onlyMainContent": True}
)

Filter by path prefix so Claude receives only the version you need. Re-crawl when docs update; schedule daily or weekly depending on release cadence.

Cleaning Output

Firecrawl's onlyMainContent removes nav and footers. For extra control:

• Remove code language headers — Some docs render "python" labels as separate lines. Post-process with regex: `re.sub(r'^[\w]*\n', '```\n', text)`
• Strip metadata blocks — YAML frontmatter in docs can be stripped before chunking
• Normalize whitespace — \n\n\n → \n\n to avoid empty chunks

For most technical docs, Firecrawl's default output is sufficient. See the Firecrawl API tutorial for advanced options.

Chunking Strategy for Claude

Claude supports 200K context, but RAG works best with smaller, focused chunks:

Strategy	Chunk size	Overlap	Best for
Token-aware (tiktoken)	~512 tokens	50 tokens	General docs
Sentence boundary	~800 chars	100 chars	Prose-heavy
Heading-split	By H2/H3	—	API reference

Use RecursiveCharacterTextSplitter with chunk_size=1200, chunk_overlap=150 as a baseline. Preserve source_url and section (heading path) in metadata for retrieval filtering.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200,
    chunk_overlap=150,
    separators=["\n## ", "\n### ", "\n\n", "\n", " "]
)
chunks = splitter.split_text(markdown)

Alternative: Apify Website Content Crawler

Apify's Website Content Crawler produces similar markdown. Use it when:

• You need scheduling, webhooks, or Apify integrations
• You already run Actors and want one platform
• You prefer Crawlee-based extraction logic

Input: start URL, max pages, include/exclude patterns. Output: markdown per page. Chain with LangChain Apify content pipeline for summarization or RAG.

Full Python Example

End-to-end: Firecrawl → markdown files → Claude API with retrieval:

from firecrawl import Firecrawl
from langchain_text_splitters import RecursiveCharacterTextSplitter
from anthropic import Anthropic
import chromadb
from chromadb.utils import embedding_functions
import time

app = Firecrawl(api_key="fc-YOUR-API-KEY")
job = app.crawl_url(
    "https://docs.yourproduct.com",
    limit=100,
    max_depth=2,
    scrape_options={"formats": ["markdown"], "onlyMainContent": True}
)
job_id = job.get("id")
while True:
    status = app.check_crawl_status(job_id)
    if status.get("status") in ("completed", "failed"):
        break
    time.sleep(5)
data = app.get_crawl_data(job_id).get("data", []) if status.get("status") == "completed" else []

splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=150)
ef = embedding_functions.OpenAIEmbeddingFunction(api_key="sk-...")
client = chromadb.Client()
coll = client.create_collection("docs", embedding_function=ef)

for page in data:
    md = page.get("markdown", "")
    url = page.get("metadata", {}).get("sourceURL", "")
    chunks = splitter.split_text(md)
    for i, c in enumerate(chunks):
        coll.add(ids=[f"{url}#{i}"], documents=[c], metadatas=[{"source": url}])

# Query
results = coll.query(query_texts=["How do I authenticate?"], n_results=5)
context = "\n\n".join(results["documents"][0])
anthropic = Anthropic()
msg = anthropic.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Based on:\n{context}\n\nAnswer: How do I authenticate?"}]
)
print(msg.content[0].text)

Comparison: Docs Crawling Tools

Attribute	Firecrawl	Apify Website Content Crawler	Manual wget/curl
Markdown output	Native, clean	Native	Requires converter
Boilerplate removal	Automatic	Automatic	Manual
Rate limiting / proxies	Built-in	Via Apify proxy	Self-manage
Versioned docs	Path filters	Path filters	Manual URL list
API	REST, Python/JS SDK	Apify API	N/A
Cost	Per credit (~1/page)	Compute time	Free (time cost)
Best for	Fast RAG setup	Existing Apify users	Full control

Winner for speed: Firecrawl — one API call, markdown out. Winner for platform integration: Apify if you use Actors and scheduling.

Incremental Updates

Docs change over time. To keep your RAG index fresh without full recrawls:

• Sitemap lastmod — If the doc site publishes a sitemap with <lastmod>, crawl only URLs with changed dates.
• Content hashing — Store a hash (e.g. MD5 of markdown) per URL. On recrawl, re-embed only pages where the hash changed.
• Schedule — Run Firecrawl or Apify on a cadence (daily for changelogs, weekly for API reference). Use webhooks to push updates to your vector DB.

Firecrawl doesn't natively support incremental crawl; you implement the diff logic. Apify's scheduler + dataset diff is easier for large doc sets.

Use Cases

• RAG over Stripe docs — Crawl /docs/api, chunk, embed. Claude answers billing/API questions with sources.
• Internal engineering wiki — Crawl Confluence or Notion export; index for internal Q&A.
• API reference for Claude coding agent — Crawl OpenAPI or reference docs; agent retrieves params and examples at code-time.
• Competitor feature tracking — Crawl competitor changelogs/docs; chunk and compare over time.

Start with a small crawl

Crawl 20–50 pages first to validate markdown quality and chunk behavior. Scale up once retrieval results look correct.

Try Firecrawl | Firecrawl RAG Pipeline

Frequently Asked Questions

Markdown is semantically dense: fewer tokens, preserved structure (headings, code blocks). HTML adds nav, footers, and tag overhead. Claude processes markdown more efficiently.

Firecrawl doesn't support authenticated sessions. Use Apify with a custom Actor that logs in first, or export docs to public URLs. For internal wikis, consider API export or SSO bypass for crawlers.

512–1200 tokens with 50–150 token overlap. Test retrieval quality on your queries. Smaller chunks = more precise retrieval; larger = more context per chunk.

Firecrawl: faster setup, REST API. Apify: scheduling, webhooks, Actor ecosystem. Both output markdown. Choose by integration needs.

Match doc update frequency. API references: weekly. Changelogs: daily. Internal wikis: daily or on push.

Firecrawl doesn't support incremental crawl. Use sitemap lastmod if available, or hash content to detect changes. Apify can schedule and diff output datasets.