Firecrawl Batch Scraping: Process Thousands of URLs 2026

Contains affiliate links. Disclosure.

Firecrawl handles batch workloads via the /crawl endpoint (multi-page from a start URL) or batched /scrape calls. Use /map first to scope URLs. Chunk submissions, track per-URL status, cap retries, and write only validated outputs.

Start batch scraping with Firecrawl →

When Batch Mode Fits

• 1,000+ URL ingestion runs per cycle
• Recurring docs, product, or news refresh jobs
• Queue-based extraction feeding analytics or RAG

For a fixed URL list, batch /scrape gives precise control. For site discovery, use /crawl with limits and path filters.

Batch Architecture Pattern

1. Build URL queue — Curated, deduplicated, canonical
2. Submit in chunks — 100–250 URLs per batch to start
3. Track status — pending / success / failed per URL
4. Retry failures — Capped attempts, exponential backoff
5. Write validated outputs — Reject malformed records

Using /crawl for Batch

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR-API-KEY")
job = app.crawl_url(
    "https://example.com/docs",
    limit=1000,
    include_paths=["/docs/*"],
    exclude_paths=["/docs/archive/*"],
    scrape_options={"formats": ["markdown"]}
)
job_id = job.get("id")

# Poll until completed
while True:
    status = app.check_crawl_status(job_id)
    s = status.get("status")
    if s in ("completed", "failed"):
        break
    time.sleep(10)

data = app.get_crawl_data(job_id)
for page in data.get("data", []):
    process(page.get("markdown"), page.get("metadata", {}))

Batched /scrape for Known URL Lists

import asyncio
from firecrawl import FirecrawlAsync

app = FirecrawlAsync(api_key="fc-YOUR-API-KEY")
urls = ["https://example.com/p1", "https://example.com/p2", ...]  # Your list

async def scrape_batch(batch):
    tasks = [app.scrape(url, formats=["markdown"]) for url in batch]
    return await asyncio.gather(*tasks, return_exceptions=True)

# Process in chunks of 50 (respect concurrency limits)
chunk_size = 50
for i in range(0, len(urls), chunk_size):
    batch = urls[i:i + chunk_size]
    results = asyncio.run(scrape_batch(batch))
    for url, r in zip(batch, results):
        if isinstance(r, Exception):
            log_failure(url, r)
        else:
            validate_and_store(r, url)

Concurrency: Free=2, Hobby=5, Standard=50. Stay under your plan limit.

Map Before Crawl

Run /map first to avoid wasted credits:

mapped = app.map_url("https://example.com", limit=5000, search="docs")
urls = [l["url"] for l in mapped.get("links", []) if "/docs/" in l["url"]]
# Feed urls into batched scrape or use as include_paths for crawl

Map costs 1 credit per request; crawl costs 1 per page.

Failure Handling

• Retry only transient failures (5xx, timeouts)
• Max retries: 3–5 per URL
• Move persistent failures to manual review
• Emit per-domain failure metrics for anti-bot hotspots

Cost Controls

• Filter URLs before extraction (canonical-only, allowed paths)
• Skip unchanged URLs with hash/signature checks
• Avoid extra formats (e.g. screenshots) unless needed
• Track credits per valid record, not per submitted URL

Next step

Combine batch Firecrawl with Make.com for scheduled ingestion pipelines.

Frequently Asked Questions

Yes. Use /crawl for site-scale or batched /scrape with controlled chunking, per-URL status, and capped retries.

Monitor valid-output rate, failure rate by domain, retry volume, and credits per valid record.

Canonicalize URLs, use content hashes for change detection, and deduplicate before submit.

Start with 100–250 URLs. Increase only when error rates stay low. Stay under concurrency limits.