Firecrawl Python SDK: Complete Developer Guide 2026

Contains affiliate links. Disclosure.

The Firecrawl Python SDK (firecrawl-py) provides a simple interface: scrape(), crawl(), and map_url(). Install with pip install firecrawl-py, initialize with your API key, and start scraping. The SDK returns snake_case by default and supports async via FirecrawlAsync.

Get started with Firecrawl →

Install and Authenticate

pip install firecrawl-py

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR-API-KEY")
# Or: app = Firecrawl()  # reads FIRECRAWL_API_KEY from env

Store keys in environment variables. Never hardcode in source control.

Single-Page Scrape

doc = app.scrape("https://example.com/docs", formats=["markdown", "links"])
print(doc.get("markdown", "")[:300])
print(doc.get("links", [])[:5])

Formats: markdown, html, links, images, screenshots, extract. Use only_main_content=True (default) to exclude nav and footer.

Structured Extraction with Schema

Use the extract format with a JSON schema for structured output:

from pydantic import BaseModel

class CompanyProfile(BaseModel):
    name: str
    industry: str | None = None
    headquarters: str | None = None
    website: str

doc = app.scrape(
    "https://example.com/about",
    formats=["extract"],
    extract={"schema": CompanyProfile.model_json_schema()}
)
print(doc.get("extract"))

Validate extracted fields before writing to storage.

Map URL Discovery

Discover URLs on a domain before crawling:

mapped = app.map_url("https://example.com", limit=5000, search="docs")
links = mapped.get("links", [])
docs_urls = [l["url"] for l in links if "/docs/" in l["url"]]
print(f"Found {len(docs_urls)} docs URLs")

map_url costs 1 credit per request regardless of link count (up to 100,000).

Crawl Workflow

Crawl is async. Start, then poll for completion:

job = app.crawl_url(
    "https://example.com/docs",
    limit=100,
    scrape_options={"formats": ["markdown"]}
)
job_id = job.get("id")

# Poll until done
status = app.check_crawl_status(job_id)
while status.get("status") not in ("completed", "failed"):
    time.sleep(5)
    status = app.check_crawl_status(job_id)

if status.get("status") == "completed":
    data = app.get_crawl_data(job_id)
    for page in data.get("data", []):
        print(page.get("markdown", "")[:200])

Use include_paths and exclude_paths to scope the crawl.

Async Variant

For concurrent requests, use the async client:

from firecrawl import FirecrawlAsync
import asyncio

async def main():
    app = FirecrawlAsync(api_key="fc-YOUR-API-KEY")
    tasks = [
        app.scrape("https://example.com/page1"),
        app.scrape("https://example.com/page2"),
    ]
    results = await asyncio.gather(*tasks)
    return results

asyncio.run(main())

Respect rate limits. Free tier: 2 concurrent; Hobby: 5; Standard: 50.

Error Handling

try:
    doc = app.scrape(url)
    if not doc:
        raise ValueError("Empty response")
except Exception as e:
    # Handle 429, 402, 500
    if "rate limit" in str(e).lower():
        time.sleep(60)  # Backoff and retry

Add retries with exponential backoff. Keep source_url, run_id, and timestamp on every record for debugging.

Production Checklist

• Use environment variables for API keys
• Add retry logic with capped attempts
• Log request ID, domain, endpoint
• Validate response structure before downstream writes
• Monitor cost per valid record, not just request success

Apify Free planTry Apify free: $5 of usage every monthPermanent Free plan, no credit card. Runs any Store Actor; credits reset monthly.Claim the $5 credit

Next step

See the Firecrawl LangChain integration for RAG pipelines with document loaders.

Frequently Asked Questions

Run pip install firecrawl-py. Initialize with Firecrawl(api_key='...') or set FIRECRAWL_API_KEY in env.

Yes. Use formats=['extract'] with a JSON schema (e.g. from Pydantic) and validate outputs before production.

Yes. Use FirecrawlAsync for concurrent scraping. Respect plan limits: 2–50 concurrent depending on tier.

Track field completeness, failure rate by domain, retry volume, and cost per valid record.