Firecrawl + OpenAI: Build an AI Research Assistant 2026

Contains affiliate links. Disclosure.

Combine Firecrawl (web scrape) and OpenAI (GPT-4) to build a research assistant: scrape URLs, extract markdown, send to the model with a prompt, and return structured findings. Pipeline: user query → Firecrawl scrape → chunk if needed → GPT-4 with system + user messages → structured response.

Get Firecrawl →

Minimal Python Flow

from firecrawl import Firecrawl
from openai import OpenAI

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
openai_client = OpenAI()

def research(query: str, urls: list[str]) -> str:
    # 1. Scrape
    contents = []
    for url in urls:
        doc = firecrawl.scrape(url, formats=["markdown"])
        md = doc.get("markdown", "")
        contents.append(f"--- Source: {url} ---\n{md[:8000]}")  # Truncate per page

    # 2. Build prompt
    context = "\n\n".join(contents)
    messages = [
        {"role": "system", "content": "You are a research assistant. Summarize findings and cite sources by URL."},
        {"role": "user", "content": f"Research question: {query}\n\nSources:\n{context}"}
    ]

    # 3. Call GPT-4
    resp = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=0.3
    )
    return resp.choices[0].message.content

Example Usage

result = research(
    "What are the main features of Firecrawl?",
    ["https://docs.firecrawl.dev", "https://firecrawl.dev"]
)
print(result)

Structured Output

Use a JSON schema for deterministic output:

from openai import OpenAI

resp = openai_client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=messages,
    response_format=ResearchOutput  # Pydantic model
)

Cost and Limits

• Firecrawl: 1 credit per page
• OpenAI: Per-token pricing. Truncate or chunk to control cost.
• Keep source URLs and timestamps for citations and auditability.

Apify Free planTry Apify free: $5 of usage every monthPermanent Free plan, no credit card. Runs any Store Actor; credits reset monthly.Claim the $5 credit

Next step

Scale research with RAG: index many pages, retrieve only relevant chunks.

Frequently Asked Questions

Yes. Scrape with Firecrawl, pass markdown to GPT-4. Use chunking for long content.

Scrape URLs → extract markdown → send to LLM with a research prompt → return findings with citations.

A tool that gathers web content, analyzes it with an LLM, and returns structured research with source citations.

Chunk content per page. Summarize each, or use RAG to retrieve only relevant sections.