Firecrawl + Anthropic Claude: Build an AI Content Analyzer

Crawl a competitor's blog with Firecrawl, pipe every article into Claude, and get back a ranked list of content gaps — all in under 100 lines of Python. This tutorial builds that pipeline end-to-end: extracting structured data from live URLs, batching it through the Anthropic SDK, and generating a competitive content report you can act on immediately.

Contains affiliate links. Disclosure.

You will build a script that:

1. Crawls a competitor blog and collects every article URL
2. Scrapes each article's title, headings, and body text
3. Sends the aggregated content to Claude for topic and tone analysis
4. Identifies content gaps relative to your own topic list
5. Outputs a Markdown report with prioritised recommendations

Prerequisites: Python 3.11+, a Firecrawl API key, and an Anthropic API key.

---

Why Firecrawl + Claude?

Most content gap tools work from keyword databases alone. That gives you search volume but tells you nothing about the quality, structure, or tone of what your competitors have already published.

Combining Firecrawl's managed crawling with Claude's language understanding flips that model:

Capability	Firecrawl	Claude
Discover URLs across a domain	✅ /map endpoint	—
Convert JS-heavy pages to clean Markdown	✅ /scrape endpoint	—
Analyse topics, tone, and structure	—	✅ Sonnet / Haiku
Identify content gaps from a list	—	✅ Structured output
Generate prioritised recommendations	—	✅ Chain-of-thought
Best for	Extracting live web content at scale	Reasoning over unstructured text

Firecrawl removes every fragile Selenium session and BeautifulSoup selector you would otherwise write. Claude removes every regex heuristic you would otherwise use to classify text. Together, the stack is about a dozen pip install dependencies.

---

Step 1 — Install Dependencies

pip install firecrawl-py anthropic rich

firecrawl-py is the official Firecrawl Python SDK. rich is optional but produces readable terminal output.

---

Step 2 — Map the Competitor Blog

Before scraping content, use Firecrawl's /map endpoint to discover all article URLs on the target domain. This avoids writing a custom link-following crawler.

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])

def discover_blog_urls(domain: str, limit: int = 50) -> list[str]:
    """Return up to `limit` article URLs from a competitor blog."""
    result = app.map_url(
        url=domain,
        params={"limit": limit, "includeSubdomains": False},
    )
    # map_url returns a MapResponse with a .links attribute
    return result.links or []

urls = discover_blog_urls("https://blog.example.com", limit=30)
print(f"Discovered {len(urls)} URLs")

map_url parses the sitemap and internal links without spinning up a full Chromium session for each page — dramatically faster than recursive crawling.

---

Step 3 — Scrape Article Content

With URLs in hand, scrape each page and extract clean Markdown. The /scrape endpoint handles JavaScript rendering, cookie banners, and navigation clutter automatically.

from firecrawl import ScrapeOptions

def scrape_articles(urls: list[str]) -> list[dict]:
    """Scrape each URL and return structured article data."""
    articles = []
    for url in urls:
        try:
            result = app.scrape_url(
                url,
                options=ScrapeOptions(formats=["markdown"]),
            )
            if result.markdown:
                articles.append({
                    "url": url,
                    "title": result.metadata.get("title", ""),
                    "content": result.markdown,
                })
        except Exception as exc:
            print(f"Skipping {url}: {exc}")
    return articles

articles = scrape_articles(urls[:20])  # limit for demo
print(f"Scraped {len(articles)} articles")

Keep formats=["markdown"] rather than requesting HTML — Claude processes Markdown token-efficiently, and the Markdown conversion strips boilerplate that would inflate your prompt cost.

---

Step 4 — Analyse Content with Claude

Send the scraped articles to Claude and request a structured analysis. Claude Sonnet 4.6 handles long-context inputs well, but if you are processing 50+ articles, batch them into groups of 10 and merge the results.

import anthropic
import json

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

ANALYSIS_PROMPT = """
You are a content strategist. Analyse the following competitor blog articles and return JSON with:
- topics: list of distinct topics covered (strings)
- tone: overall tone description (string, e.g. "technical tutorial", "thought leadership")
- avg_length: estimated average word count per article (integer)
- top_headlines: list of the 5 most common H2 patterns (strings)
- content_gaps: list of topics NOT covered that a technical audience would expect (strings)

Return only valid JSON with those exact keys. No prose before or after.

ARTICLES:
{articles}
"""

def analyse_content(articles: list[dict]) -> dict:
    """Send articles to Claude and get structured content analysis."""
    # Truncate each article to ~800 words to stay within context limits
    corpus = "\n\n---\n\n".join(
        f"URL: {a['url']}\nTitle: {a['title']}\n\n{a['content'][:3000]}"
        for a in articles
    )

    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        messages=[
            {
                "role": "user",
                "content": ANALYSIS_PROMPT.format(articles=corpus),
            }
        ],
    )

    raw = message.content[0].text
    return json.loads(raw)

analysis = analyse_content(articles)
print("Topics found:", len(analysis["topics"]))
print("Tone:", analysis["tone"])

The prompt requests JSON directly. This avoids parsing prose and makes the next step (gap detection) deterministic. For production use, add a retry loop and validate the parsed JSON against a schema before proceeding to the gap-analysis step.

---

Step 5 — Identify Content Gaps

Compare the competitor's topic coverage against your own published content. Provide Claude with both lists and ask for a prioritised gap analysis.

YOUR_TOPICS = [
    "web scraping with Python",
    "Playwright automation",
    "proxy rotation",
    "Apify actors",
    "LLM data pipelines",
]

GAP_PROMPT = """
You are a content strategist. I will give you two lists:
1. COMPETITOR TOPICS: what a competitor blog covers
2. MY TOPICS: what I have already published

Return JSON with:
- gaps: list of topics the competitor covers that I do not (strings)
- opportunities: list of gaps ranked by likely search demand (top 5, strings)
- recommendations: list of 5 specific article titles I should write next (strings)

Return only valid JSON with those exact keys.

COMPETITOR TOPICS:
{competitor_topics}

MY TOPICS:
{my_topics}
"""

def identify_gaps(analysis: dict, my_topics: list[str]) -> dict:
    """Ask Claude to compare topic coverage and rank opportunities."""
    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": GAP_PROMPT.format(
                    competitor_topics="\n".join(f"- {t}" for t in analysis["topics"]),
                    my_topics="\n".join(f"- {t}" for t in my_topics),
                ),
            }
        ],
    )
    return json.loads(message.content[0].text)

gaps = identify_gaps(analysis, YOUR_TOPICS)
print("Content gaps:", gaps["gaps"][:5])
print("Top opportunities:", gaps["opportunities"])

---

Step 6 — Generate the Competitive Content Report

Write the final report to Markdown. This file is ready to drop into a content planning board.

from datetime import date

def generate_report(
    competitor_url: str,
    analysis: dict,
    gaps: dict,
    articles: list[dict],
) -> str:
    """Assemble a Markdown competitive content report."""
    today = date.today().isoformat()
    lines = [
        f"# Competitive Content Report: {competitor_url}",
        f"_Generated {today}_",
        "",
        "## Competitor Overview",
        f"- **Tone:** {analysis['tone']}",
        f"- **Average article length:** {analysis['avg_length']} words",
        f"- **Distinct topics covered:** {len(analysis['topics'])}",
        "",
        "## Topics Covered by Competitor",
        *[f"- {t}" for t in analysis["topics"]],
        "",
        "## Content Gaps (Topics Competitor Has, You Don't)",
        *[f"- {g}" for g in gaps["gaps"]],
        "",
        "## Top 5 Opportunities",
        *[f"{i+1}. {o}" for i, o in enumerate(gaps["opportunities"])],
        "",
        "## Recommended Articles to Write",
        *[f"{i+1}. {r}" for i, r in enumerate(gaps["recommendations"])],
        "",
        "## Scraped Article Index",
        *[f"- [{a['title']}]({a['url']})" for a in articles],
    ]
    return "\n".join(lines)

report = generate_report(
    "https://blog.example.com",
    analysis,
    gaps,
    articles,
)

with open("content_report.md", "w") as f:
    f.write(report)

print("Report written to content_report.md")

---

Putting It All Together

Here is the complete script in runnable form:

#!/usr/bin/env python3
"""
Competitive content analyzer: Firecrawl + Claude
Usage: FIRECRAWL_API_KEY=... ANTHROPIC_API_KEY=... python analyzer.py
"""

import json
import os
from datetime import date

import anthropic
from firecrawl import FirecrawlApp, ScrapeOptions

# ── Config ──────────────────────────────────────────────────────────────────
COMPETITOR_URL = "https://blog.example.com"
MAX_ARTICLES = 20
MY_TOPICS = [
    "web scraping with Python",
    "proxy rotation",
    "LLM data pipelines",
    "Apify actors",
]

# ── Clients ──────────────────────────────────────────────────────────────────
fc = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])
claude = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])


def discover(url: str, limit: int) -> list[str]:
    return (fc.map_url(url, params={"limit": limit}).links or [])


def scrape(urls: list[str]) -> list[dict]:
    out = []
    for u in urls:
        try:
            r = fc.scrape_url(u, options=ScrapeOptions(formats=["markdown"]))
            if r.markdown:
                out.append({"url": u, "title": r.metadata.get("title", ""), "content": r.markdown})
        except Exception as e:
            print(f"  skip {u}: {e}")
    return out


def ask(prompt: str) -> dict:
    msg = claude.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(msg.content[0].text)


def analyse(articles: list[dict]) -> dict:
    corpus = "\n\n---\n\n".join(
        f"URL: {a['url']}\nTitle: {a['title']}\n\n{a['content'][:3000]}" for a in articles
    )
    return ask(
        "Analyse these blog articles and return JSON with keys: topics (list), tone (str), "
        "avg_length (int), top_headlines (list of 5), content_gaps (list).\n\nARTICLES:\n" + corpus
    )


def find_gaps(analysis: dict, mine: list[str]) -> dict:
    return ask(
        "Compare these topic lists and return JSON with keys: gaps (list), opportunities (top 5 list), "
        "recommendations (5 article titles).\n\n"
        "COMPETITOR:\n" + "\n".join(f"- {t}" for t in analysis["topics"]) + "\n\n"
        "MINE:\n" + "\n".join(f"- {t}" for t in mine)
    )


def report(competitor: str, analysis: dict, gaps: dict, articles: list[dict]) -> str:
    return "\n".join([
        f"# Competitive Content Report: {competitor}",
        f"_Generated {date.today().isoformat()}_\n",
        f"**Tone:** {analysis['tone']} | **Avg length:** {analysis['avg_length']} words\n",
        "## Content Gaps",
        *[f"- {g}" for g in gaps["gaps"]], "",
        "## Top Opportunities",
        *[f"{i+1}. {o}" for i, o in enumerate(gaps["opportunities"])], "",
        "## Recommended Articles",
        *[f"{i+1}. {r}" for i, r in enumerate(gaps["recommendations"])],
    ])


if __name__ == "__main__":
    print("1/4  Discovering URLs …")
    urls = discover(COMPETITOR_URL, MAX_ARTICLES)

    print(f"2/4  Scraping {len(urls)} articles …")
    articles = scrape(urls)

    print("3/4  Analysing with Claude …")
    analysis = analyse(articles)
    gaps = find_gaps(analysis, MY_TOPICS)

    print("4/4  Writing report …")
    md = report(COMPETITOR_URL, analysis, gaps, articles)
    with open("content_report.md", "w") as f:
        f.write(md)
    print("Done → content_report.md")

---

Cost Estimate

Before running this at scale, understand what you are paying for:

Item	Unit cost	30-article run
Firecrawl /map (1 call)	~$0.001	$0.001
Firecrawl /scrape per page	~$0.003	~$0.09
Claude Sonnet 4.6 input tokens	$3/MTok	~$0.10
Claude Sonnet 4.6 output tokens	$15/MTok	~$0.06
Total		~$0.25

Thirty articles analysed for 30 cents. Run this monthly against 5 competitors: under $2. At this price point, automating it as a scheduled Apify Actor makes sense — you get persistent storage, scheduled runs, and a REST API to trigger analysis from your content workflow.

---

Scaling to Production

For one-off analysis, the script above is sufficient. For recurring competitive monitoring:

• Schedule with Apify: Package the script as a Python Actor on Apify and set a weekly schedule. Results land in a dataset you can query via API.
• Store structured output: Write the JSON analysis to a database instead of a Markdown file. Query it to track which competitor topics are growing over time.
• Expand Claude's context: Use Claude's 200K-token context window to process entire blogs in a single call rather than batching. This improves topic coherence across articles.
• Add RAG retrieval: Embed the scraped articles and your own content, then use vector similarity to surface the most directly competing pieces. See our RAG pipeline guide for the architecture.

For larger-scale crawling needs — thousands of pages, authenticated sessions, or custom extraction logic — Apify provides a managed infrastructure layer on top of Crawlee with built-in proxy rotation. Read the Firecrawl vs Crawlee comparison to understand which tool fits which scale.

---

Frequently Asked Questions

Yes. Firecrawl converts web pages to clean Markdown, and Claude processes that Markdown for analysis, classification, or extraction. The combination is straightforward: use the Firecrawl Python SDK to scrape URLs, then pass the resulting Markdown text as a user message to Claude via the Anthropic SDK. No special integration layer is required.

The practical approach is: (1) discover competitor URLs with Firecrawl's /map endpoint, (2) scrape each article to clean Markdown, (3) send the aggregated content to an LLM like Claude with a structured prompt requesting JSON output, and (4) compare the identified topics against your own content list to surface gaps. The tutorial above provides complete Python code for each step.

Content gap analysis identifies topics that your competitors rank for or write about that you have not yet covered. Traditionally it relies on keyword tools like Ahrefs or SEMrush. The AI approach shown here adds a qualitative layer: Claude can identify topic clusters, structural patterns, and tone differences that keyword databases miss — for example, noting that a competitor publishes step-by-step tutorials while you only publish conceptual overviews.

Analysing 30 articles costs roughly $0.15 to $0.30 in API fees using Claude Sonnet 4.6 at standard pricing. Claude Haiku 4.5 reduces this by about 10x at the cost of slightly less nuanced analysis. Check the current Anthropic pricing page for exact token rates before running at scale.

Yes. Firecrawl provisions headless Chromium on its backend and waits for JavaScript hydration before capturing the DOM. You do not need to configure browser automation yourself — the /scrape endpoint returns fully rendered Markdown from React SPAs and other JS-heavy sites.