Build an AI Research Agent: Automated Web Research with LangGraph and Apify (2026)

March 19, 2026 · 7 min read

Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

An AI research agent automates the full loop: given a research question, it searches the web via Apify, fetches and reads pages, extracts key findings, and synthesizes a structured report. This guide walks you through building one with LangGraph and the Apify Python client.

Start building with Apify →

What the Agent Does

Given a research question (e.g. "What are the main risks of AI agents in production as of 2026?"), the agent:

Searches — Calls Apify Google Search Scraper to get relevant URLs.
Fetches — Uses Apify Website Content Crawler on each URL to extract markdown.
Analyzes — An LLM (Claude Sonnet 4.6 or a comparable GPT model) reads each page and extracts key findings.
Reports — Synthesizes all findings into a structured markdown report with sources cited.

If insufficient data is found after analysis, a conditional edge can route back to search with refined queries.

Architecture: LangGraph StateGraph

┌─────────┐    ┌──────────────┐    ┌─────────┐    ┌───────┐
│ START   │───▶│ search       │───▶│ fetch   │───▶│analyze│───▶ report ──▶ END
└─────────┘    │ (Apify SERP) │    │ (Apify  │    │ (LLM) │
               └──────────────┘    │ Crawler)│    └───┬───┘
                    ▲               └─────────┘        │
                    │                    │             │
                    └──────────────────┴─────────────┘
                         (re-search if insufficient)

Prerequisites

Python 3.10+
Apify account (sign up free)
OpenAI or Anthropic API key
APIFY_API_TOKEN in environment

pip install langgraph langchain-anthropic langchain-openai apify-client

Complete Python Code

from typing import TypedDict
from langgraph.graph import StateGraph, END
from langchain_anthropic import ChatAnthropic
from langchain_openai import ChatOpenAI
from apify_client import ApifyClient
import os

# --- State ---
class ResearchState(TypedDict):
    research_question: str
    urls: list[str]
    documents: list[dict]
    analysis: list[dict]
    report: str
    iteration: int

# --- Node 1: Search ---
def search_node(state: ResearchState) -> dict:
    """Call Apify Google Search Scraper. Return top URLs."""
    client = ApifyClient(os.environ["APIFY_API_TOKEN"])
    query = state["research_question"][:200]  # truncate for SERP
    run_input = {
        "queries": query,
        "maxPagesPerQuery": 1,
        "resultsPerPage": 10,
    }
    run = client.actor("apify/google-search-scraper").call(run_input=run_input)
    items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
    urls = []
    for item in items:
        if "organicResults" in item:
            for r in item["organicResults"][:5]:
                if r.get("url"):
                    urls.append(r["url"])
    return {"urls": urls[:10]}  # top 10

# --- Node 2: Fetch pages ---
def fetch_pages_node(state: ResearchState) -> dict:
    """Crawl each URL with Website Content Crawler. Return markdown."""
    client = ApifyClient(os.environ["APIFY_API_TOKEN"])
    urls = state["urls"]
    if not urls:
        return {"documents": []}
    start_urls = [{"url": u} for u in urls]
    run_input = {
        "startUrls": start_urls,
        "maxCrawlPages": len(urls),
        "maxCrawlDepth": 0,
        "crawlerType": "cheerio",
    }
    run = client.actor("apify/website-content-crawler").call(run_input=run_input)
    items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
    docs = [{"url": i.get("url", ""), "markdown": i.get("markdown", "")} for i in items]
    return {"documents": docs}

# --- Node 3: Analyze ---
def analyze_node(state: ResearchState) -> dict:
    """LLM reads each doc, extracts key findings."""
    llm = ChatAnthropic(model="claude-sonnet-4-6", api_key=os.environ.get("ANTHROPIC_API_KEY"))
    # Or: llm = ChatOpenAI(model="gpt-4o", api_key=os.environ.get("OPENAI_API_KEY"))
    docs = state["documents"]
    question = state["research_question"]
    analyses = []
    for d in docs[:5]:  # limit to 5 pages
        if not d.get("markdown"):
            continue
        prompt = f"""Given this web page content, extract key findings relevant to: "{question}".
Cite the source URL: {d.get('url', 'unknown')}

Content:
{d['markdown'][:8000]}

Return: bullet points of findings, each with [Source: URL]."""
        resp = llm.invoke(prompt)
        analyses.append({"url": d["url"], "findings": resp.content})
    return {"analysis": analyses}

# --- Node 4: Report ---
def report_node(state: ResearchState) -> dict:
    """Synthesize final report."""
    llm = ChatAnthropic(model="claude-sonnet-4-6", api_key=os.environ.get("ANTHROPIC_API_KEY"))
    analyses = state["analysis"]
    question = state["research_question"]
    combined = "\n\n".join([f"## Source: {a['url']}\n{a['findings']}" for a in analyses])
    prompt = f"""Synthesize a structured markdown report for this research question:
"{question}"

Input findings (with sources):
{combined}

Output: Executive summary, main sections, conclusions, and a Sources section listing all URLs."""
    report = llm.invoke(prompt)
    return {"report": report.content}

# --- Graph ---
def should_continue(state: ResearchState) -> str:
    """Optionally re-search if insufficient content."""
    if len(state.get("documents", [])) < 2 and state.get("iteration", 0) < 1:
        return "search"
    return "report"

graph_builder = StateGraph(ResearchState)
graph_builder.add_node("search", search_node)
graph_builder.add_node("fetch_pages", fetch_pages_node)
graph_builder.add_node("analyze", analyze_node)
graph_builder.add_node("report", report_node)

graph_builder.set_entry_point("search")
graph_builder.add_edge("search", "fetch_pages")
graph_builder.add_edge("fetch_pages", "analyze")
graph_builder.add_conditional_edges("analyze", should_continue, {"search": "search", "report": "report"})
graph_builder.add_edge("report", END)

graph = graph_builder.compile()

# --- Run ---
if __name__ == "__main__":
    result = graph.invoke({
        "research_question": "What are the main risks of AI agents in production as of 2026?",
        "urls": [],
        "documents": [],
        "analysis": [],
        "report": "",
        "iteration": 0,
    })
    print(result["report"])

Output Format

The agent produces a markdown report with:

Executive summary
Main sections (findings grouped by theme)
Conclusions
Sources section (all URLs cited)

Example excerpt:

## Executive Summary
AI agents in production face three primary risks in 2026: ...

## 1. Hallucination and Data Drift
- [Source: https://example.com/...]
- ...

## Sources
- https://example.com/...

Handling Loops: Re-search if Insufficient Data

The conditional edge should_continue checks whether fewer than 2 documents were fetched. If so and iteration < 1, it routes back to search with refined behavior (you can add query refinement logic in the search node). Otherwise it proceeds to report.

Cost Per Research Run

Component	Estimate
Apify Google Search Scraper	~$0.002 per SERP page
Apify Website Content Crawler	~$0.001–0.01 per page (depends on size)
Claude Sonnet 4.6	~$3/1M input, ~$15/1M output
GPT-4o	~$2.50/1M input, ~$10/1M output

A typical run (10 URLs, 5 pages analyzed, 2K tokens out): ~$0.05–0.15 in Apify credits + ~$0.03–0.10 in LLM costs. Scale up by increasing URLs or pages; watch Apify compute units and token usage.

Comparison: Approaches to AI Research

Approach	Pros	Cons
LangGraph + Apify (this)	Full control, loops, structured output	Requires Python and Apify/LangGraph setup
Claude + Apify MCP	Interactive, no code	Manual per-query; not batch
LangChain Apify pipeline	Simpler, linear	No graph, no re-search loops
Make.com + Apify	No-code, scheduled	Less flexible logic

For automated, scheduled research with re-search logic, LangGraph + Apify is the strongest option. See the LangChain Apify content pipeline for a simpler linear scrape→summarize flow, or Claude real-time web access with Apify for interactive use.

Apify Free planTry Apify free: $5 of usage every monthPermanent Free plan, no credit card. Runs any Store Actor; credits reset monthly.Claim the $5 credit

Next step

Clone the code, set APIFY_API_TOKEN and ANTHROPIC_API_KEY (or OPENAI_API_KEY), then run with your own research question. Try Apify free →

Frequently Asked Questions

apify/google-search-scraper for SERP URLs and apify/website-content-crawler for page content. Both are official Apify Actors with pay-per-usage pricing.

Yes. Replace ChatAnthropic with ChatOpenAI(model='gpt-4o') and set OPENAI_API_KEY. The prompts work with either provider.

Limit to 5–10 URLs per run. Use Apify's built-in concurrency controls. Add delays between crawls if hitting target site rate limits.

Yes. The should_continue conditional edge routes back to search when documents < 2 and iteration < 1. Extend logic for refined queries.

LangChain pipeline is linear: scrape → summarize → publish. This LangGraph agent adds search, multi-step analysis, re-search loops, and report synthesis.

LangGraph uses standard Python. Call the Apify client (apify-client) inside any node. No official LangGraph–Apify integration; the client works directly.

What the Agent Does​

Architecture: LangGraph StateGraph​

Prerequisites​

Complete Python Code​

Output Format​

Handling Loops: Re-search if Insufficient Data​

Cost Per Research Run​

Comparison: Approaches to AI Research​

Common mistakes and fixes