How to Build a Web-Browsing AI Assistant with Firecrawl and Claude (2026)
A web-browsing AI assistant: the user asks a question, Claude decides what to search and scrape, Firecrawl fetches content, and Claude synthesizes the answer. Build it with the Anthropic API, Firecrawl Python SDK, and a tool-calling loop. Get Firecrawl for Claude.
Architecture
- User asks — e.g. "What are the latest pricing changes at OpenAI?"
- Claude decides — Chooses to call
scrape_urlon openai.com/pricing orsearch_webfor URLs first - Firecrawl executes — Returns markdown (or search results)
- Claude synthesizes — Answers with source attribution
- Loop — Repeat until Claude has enough context or produces a final answer
No MCP required — a Python script with tool definitions and a message loop suffices. For Claude Desktop integration, use the Firecrawl MCP server.
Tool Definitions
Claude's tool_use supports function-style tools. Define two: scrape_url and search_web.
from anthropic import Anthropic
TOOLS = [
{
"name": "scrape_url",
"description": "Scrape a URL and return its content as markdown. Use when you know the exact URL to fetch.",
"input_schema": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The full URL to scrape (e.g. https://openai.com/pricing)"
}
},
"required": ["url"]
}
},
{
"name": "search_web",
"description": "Search the web for URLs matching a query. Use when you need to find relevant URLs before scraping.",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query (e.g. 'OpenAI pricing 2026')"
}
},
"required": ["query"]
}
}
]
Python Implementation
from anthropic import Anthropic
from firecrawl import Firecrawl
import json
client = Anthropic()
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
def scrape_url(url: str) -> str:
try:
result = firecrawl.scrape(url, formats=["markdown"])
md = result.get("markdown", "")
return md[:15000] if md else "[No content extracted]"
except Exception as e:
return f"[Error: {str(e)}]"
def search_web(query: str) -> str:
try:
result = firecrawl.search(query, limit=5)
links = result.get("data", [])
return json.dumps([l.get("url") for l in links]) if links else "[]"
except Exception as e:
return f"[Error: {str(e)}]"
def run_tool(name: str, args: dict) -> str:
if name == "scrape_url":
return scrape_url(args.get("url", ""))
if name == "search_web":
return search_web(args.get("query", ""))
return "[Unknown tool]"
def ask_assistant(question: str, max_turns: int = 5) -> str:
messages = [{"role": "user", "content": question}]
for _ in range(max_turns):
resp = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=4096,
tools=TOOLS,
messages=messages
)
for block in resp.content:
if block.type == "text":
if block.text:
return block.text
if block.type == "tool_use":
result = run_tool(block.name, block.input)
messages.append({
"role": "assistant",
"content": [{"type": "tool_use", "id": block.id, "name": block.name, "input": block.input}]
})
messages.append({
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": block.id, "content": result}]
})
messages.append({"role": "assistant", "content": resp.content})
return "[Max turns reached]"
Example Interaction
User: "What are the latest pricing changes at OpenAI?"
Claude (internal): Calls scrape_url with https://openai.com/pricing.
Firecrawl: Returns markdown of the pricing page.
Claude (output): "Based on the current OpenAI pricing page, here are the latest details: [structured summary with source attribution]. Source: https://openai.com/pricing"
With search_web, Claude can first find URLs before scraping — e.g. "OpenAI pricing 2026" → list of URLs → scrape the top result.
Adding Search
Firecrawl's /search endpoint returns URLs for a query. Use it when Claude doesn't know the exact URL:
# Firecrawl search (check SDK for exact method)
def search_web(query: str) -> str:
result = firecrawl.search(query, limit=5)
# Returns list of { url, title?, description? }
items = result.get("data", [])
return "\n".join(f"- {i.get('url')}" for i in items)
Claude can then call scrape_url on one or more of those URLs. For search availability, check the Firecrawl API tutorial — the /search endpoint may vary by plan.
Error Handling
| Scenario | Handling |
|---|---|
| Timeout | Retry once with exponential backoff. If fail, return "[Timeout]" so Claude can try another URL. |
| Blocked / 403 | Return "[Blocked]". Claude may try search for alternative sources. |
| Empty response | Return "[No content]". Avoid infinite retries. |
| Rate limit (429) | Back off, retry. Consider queuing requests. |
def scrape_url_safe(url: str) -> str:
for attempt in range(2):
try:
result = firecrawl.scrape(url, formats=["markdown"])
md = result.get("markdown", "")
return md[:15000] if md else "[No content extracted]"
except Exception as e:
if "429" in str(e) and attempt < 1:
time.sleep(2)
continue
return f"[Error: {str(e)}]"
return "[Failed after retries]"
Comparison: Firecrawl-Backed Claude vs Built-in Web Search
| Attribute | Firecrawl-backed Claude agent | Claude built-in web search (Anthropic) |
|---|---|---|
| Control | Full control over scrape logic | Black box |
| URL targeting | Scrape any URL, any depth | Search-only, limited URLs |
| Cost | Claude API + Firecrawl credits | Claude API (bundled or separate) |
| Customization | Add tools (map, crawl, extract) | None |
| Offline / air-gapped | Can self-host Firecrawl | No |
| Best for | Research agents, custom pipelines | Quick answers, general use |
Use Firecrawl when you need specific URLs, versioned docs, or custom extraction. Use built-in search when you want the simplest setup for general queries.
Extending with More Tools
You can add crawl_url and map_url tools so Claude can discover and batch-scrape:
{
"name": "crawl_url",
"description": "Crawl a site from a start URL. Returns markdown for discovered pages. Use for full doc sites.",
"input_schema": {
"type": "object",
"properties": {"url": {"type": "string"}, "limit": {"type": "integer", "default": 50}},
"required": ["url"]
}
}
Wire to Firecrawl's crawl endpoint. Claude can then say "Crawl docs.example.com" and get structured content for many pages in one call — useful for building RAG on the fly.
Cost
- Claude: Per token (input + output). ~$3/1M input, ~$15/1M output for Sonnet.
- Firecrawl: 1 credit per scrape. Free tier: 500 credits; Hobby: 3,000 credits/mo.
- Typical research session: 3–5 scrapes + 2–4 Claude exchanges ≈ 5–10 Firecrawl credits, ~2K–5K Claude tokens.
Add search_web only if Claude needs URL discovery. Many research flows work with the user providing URLs or Claude inferring them (e.g. openai.com/pricing).
No. A Python script with Claude tool_use and Firecrawl SDK is enough. MCP is useful when you want Claude Desktop to have web access without writing code.
Firecrawl offers search in some plans. Check the API docs. If not available, use a separate search API (e.g. SerpAPI, Brave) and pass URLs to scrape_url.
Set max_tool_uses in the messages.create call if your Anthropic SDK supports it. Otherwise, cap iterations in your loop (e.g. max 3 tool rounds).
Yes. Add tool definitions for crawl_url and map_url. Wire to Firecrawl crawl and map. Claude can then discover and batch-scrape entire doc sites.
Apify MCP gives Claude access to 2,000+ Actors (scrapers, search). Firecrawl MCP gives scrape, search, crawl, extract. Choose by ecosystem: Apify for Actor variety, Firecrawl for markdown-first workflows.
Firecrawl handles some anti-bot. For hard targets, use Bright Data Scraping Browser or an Apify Actor with residential proxies. See our Bright Data Scraping Browser guide.