Agentic Extraction: Integrating LLMs with Live DOM via MCP (2026)

Models like GPT-5 and Claude Sonnet 4.6 do not see the live web the way your scraper does. Built-in browsing tools are limited: they hit WAFs (web application firewalls) more often than a tuned stack, and dumping huge DOM trees into the context window burns tokens fast.

A practical pattern is to keep the model for reasoning and route fetching and parsing to infrastructure you control.

This post shows how to wire that up with the Model Context Protocol (MCP) and OpenAI function calling, so ChatGPT- or API-driven flows can trigger real crawls (including past many Cloudflare setups) without pretending the model is the browser.

Contains affiliate links. Disclosure.

What you gain by outsourcing the browser

Need	Typical in-model browsing	With Apify (e.g. via MCP)
Getting past hard WAFs	Often blocked on Cloudflare-heavy sites	Residential proxies and headless runs you configure
Output shape	Mostly narrative / markdown	JSON, CSV, or files as the Actor defines
Parsing	Easy to lose structure on big pages	XPath/CSS selectors in the Actor layer
Scale	One URL at a time in practice	Queues and parallel runs on Apify

Option 1: Model Context Protocol (Claude Desktop)

MCP is a standard way for an LLM client to call external tools. Claude Desktop ships with MCP support, so you can point it at Apify and let the model start Actors with JSON config.

Watch the clock: Long runs (boot, render, big queues) can exceed what feels comfortable in one MCP round-trip. If a job takes more than about a minute, prefer async runs and polling so the client is not stuck mid-call.

Config steps

1. Ensure Claude Desktop is installed locally.
2. Generate an Apify API token.
3. Inject the server mapping directly into the config file (~/Library/Application Support/Claude/claude_desktop_config.json on macOS).

{
  "mcpServers": {
    "apify-execution-layer": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server"],
      "env": {
        "APIFY_TOKEN": "YOUR_PRODUCTION_TOKEN_HERE"
      }
    }
  }
}

After restart, Claude can see Apify’s tool surface (30,000+ Actors) and call them from chat when it fits the task.

Option 2: OpenAI function calling (ChatGPT / API)

The consumer ChatGPT app does not speak MCP the way Claude Desktop does. For OpenAI, you implement tools (function calling) in your own code and call Apify from there.

Watch payload size: If an Actor returns a huge JSON blob (think thousands of products), stuffing all of it into messages will hit context limits (ContextWindowExceededError or truncation). Write results to storage, pass URLs, or summarize / slice before appending to the thread.

Defining the tool

Describe the Actor inputs with a JSON Schema so the model knows what to pass.

import openai
import requests
import json

APIFY_TOKEN = "YOUR_PRODUCTION_TOKEN"

defined_tools = [{
    "type": "function",
    "function": {
        "name": "trigger_apify_maps_extraction",
        "description": "Execute a distributed scraping sequence across Google Maps for business intelligence.",
        "parameters": {
            "type": "object",
            "properties": {
                "target_query": {
                    "type": "string",
                    "description": "The exact geographic search syntax, e.g. 'Enterprise Software in SF'"
                },
                "maximum_yield": {
                    "type": "integer",
                    "description": "Hard limit on records returned to prevent buffer overruns.",
                    "default": 10
                }
            },
            "required": ["target_query"]
        }
    }
}]

def execute_remote_actor(query: str, max_results: int = 10) -> list:
    """Calls Apify run-sync endpoint; use async + dataset for large jobs."""
    run_url = "https://api.apify.com/v2/acts/compass~crawler-google-places/run-sync-get-dataset-items?fpr=use-apify"
    response = requests.post(
        run_url,
        headers={"Authorization": f"Bearer {APIFY_TOKEN}"},
        json={"searchStringsArray": [query], "maxCrawledPlaces": max_results}
    )
    return response.json()

client = openai.OpenAI()
conversation_stream = [{"role": "user", "content": "Correlate data on 5 high-end cafes in Chicago."}]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=conversation_stream,
    tools=defined_tools,
    tool_choice="auto"
)

if response.choices[0].finish_reason == "tool_calls":
    tool_invocation = response.choices[0].message.tool_calls[0]
    inferred_args = json.loads(tool_invocation.function.arguments)
    
    payload = execute_remote_actor(inferred_args["target_query"], inferred_args.get("maximum_yield", 5))
    
    conversation_stream.append(response.choices[0].message)
    conversation_stream.append({
        "role": "tool",
        "tool_call_id": tool_invocation.id,
        "content": json.dumps(payload[:5]) 
    })
    
    final_synthesis = client.chat.completions.create(
        model="gpt-4o", messages=conversation_stream
    )
    print(final_synthesis.choices[0].message.content)

Which path to use

Claude Desktop + MCP is hard to beat for quick experiments: you describe the scrape in chat and Apify runs in the cloud without you wiring a full app.

OpenAI function calling fits when you own the backend: you choose when to run an Actor, how much JSON goes back into the model, and how to handle errors and billing—better for production agents and scheduled jobs.

Frequently Asked Questions

In-model browsing often comes from datacenter IPs that sites treat as bots. Apify Actors can run with residential proxies and headless browsers so the request looks more like a normal user; the model then works off the extracted data instead of fighting the firewall itself.

The chat API has a fixed context budget. If you paste a huge JSON result (for example a full catalog export) into the message list, the request can fail or get truncated. Return a short slice, a summary, or a link to stored data instead of the whole payload.

Not in the same first-class way today. Anthropic’s desktop client runs MCP locally; for ChatGPT you typically use the API with function calling, or put a small proxy in front that translates between your app and OpenAI’s API.