MCP Servers for Web Scraping: Complete Integration Guide (2026)

Contains affiliate links. Disclosure.

MCP (Model Context Protocol) is an open standard, originated by Anthropic in November 2024 and now community-governed, that lets AI agents like Claude call web scraping tools on demand without prompt stuffing or manual API calls. Think of it as a USB-C port for AI: one connector between an AI app and external tools or data. Instead of pasting scraped data into the chat, you ask Claude to scrape a URL, run an Apify Actor, or extract product data. The agent invokes the MCP tool, waits for the result, and uses it in its reply. This guide covers available MCP scraping servers (Apify, Firecrawl, Bright Data, Jina), setup steps, comparison tables, building a custom server, and security considerations. If you want to follow along, you can try Claude free for a week and connect the Apify MCP server to it.

Why MCP for Web Scraping?

AI agents have limited context. Giving Claude direct access to live web data via MCP tools solves:

• No manual copy-paste: Claude scrapes, receives data, and answers in one turn
• On-demand freshness: prices, inventory, and search results are current
• Consistent interface: same MCP protocol across Claude Desktop, Cursor, Windsurf, VS Code

MCP servers expose tools (e.g., scrape_url, run_actor), plus optional resources and prompts. The host sends a tool call, the server runs the scraper and returns the result. The agent never sees raw API docs. It uses tools by their description and parameters. MCP is supported by Claude (Desktop and Code), ChatGPT, VS Code, Cursor, and other clients, so the same server works across hosts.

Available MCP Scraping Servers

Server	Supported Operations	Auth	Cost	Best For
Apify MCP	Run Actors, search Store, get dataset	APIFY_TOKEN	Apify pricing	Full Actor ecosystem, custom scrapers
Firecrawl MCP	Scrape, crawl, map, extract	FIRECRAWL_API_KEY	Credit-based	Clean markdown, LLM pipelines
Bright Data MCP	SERP, unblocked scrape, datasets	Bright Data API	Per usage	Anti-bot, SERP, pre-built data
Jina Reader MCP	URL → reader-friendly markdown	Jina API key	Free tier + paid	Simple URL-to-text for LLMs

Apify MCP: run any Apify Actor (Google Search, Website Content Crawler, Amazon, etc.). Claude picks the Actor from the Store or from your allowed list. See Apify Claude Desktop MCP.

Firecrawl MCP: scrape a URL to markdown, crawl a domain, extract with schema. Output is clean and LLM-friendly. See Firecrawl MCP server setup.

Bright Data MCP: access SERP API, Scraping Browser, and pre-built datasets. Best when targets require unblocking or you need SERP data.

Jina Reader MCP: single responsibility, turning a URL into readable markdown. Minimal setup, good for quick content extraction.

Setting Up Apify MCP for Claude Desktop

Apify ships two ways to connect: the hosted server at https://mcp.apify.com (OAuth, auto-updating, recommended) and a local server you run with npx. The hosted route is the simplest because it handles auth and stays current without you reinstalling anything.

Option A: hosted server (recommended)

Point your client at the hosted endpoint and authenticate with OAuth. With over 30,000 Actors in the Store, exposing every tool at once bloats the model's context, so scope the connection to the Actors you actually need using the tools parameter:

https://mcp.apify.com?fpr=use-apify&tools=apify/website-content-crawler,apify/google-search-scraper

Option B: local server with npx

1. Prerequisites: Node.js 18+, an MCP-capable Claude plan, Apify account
2. Get token: Apify Console → Settings → Integrations → API tokens
3. Edit MCP config: usually ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows)

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--actors", "apify/website-content-crawler"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxx"
      }
    }
  }
}

Pass --actors with one or two Actors rather than loading the full catalog. Each exposed Actor adds tool metadata to the context window, so a short list keeps responses fast and cheap.

4. Restart Claude Desktop: MCP servers load on startup
5. Verify: ask Claude to "run the Website Content Crawler for https://example.com"

For more detail, see Apify Claude Desktop MCP.

Setting Up Firecrawl MCP

1. Get API key: Firecrawl → Dashboard → API key
2. Add to MCP config:

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-xxxxxxxx"
      }
    }
  }
}

3. Restart: Claude or Cursor picks up the new server
4. Verify: ask to "scrape https://example.com to markdown"

See Firecrawl MCP server setup for Cursor and VS Code.

Building a Custom MCP Server for Scraping

Use @modelcontextprotocol/sdk to expose a crawl_url tool:

import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { chromium } from 'playwright';

const server = new Server({ name: 'scraper-mcp', version: '1.0.0' });

server.tool('crawl_url', 'Fetch a URL and return its main content as text', {
  url: { type: 'string', description: 'URL to scrape' },
}, async ({ url }) => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  const content = await page.locator('body').innerText();
  await browser.close();
  return { content: [{ type: 'text', text: content.slice(0, 10000) }] };
});

async function main() {
  const transport = new StdioServerTransport();
  await server.connect(transport);
}
main();

Run with node server.js and add to MCP config:

{
  "mcpServers": {
    "custom-scraper": {
      "command": "node",
      "args": ["/path/to/server.js"]
    }
  }
}

For production, add error handling, timeouts, and proxy support. See MCP servers explained for the full architecture.

When to Use Each Server

Use Case	Recommended MCP
Run Apify Actors (Google, Amazon, custom)	Apify MCP
Quick URL → markdown for RAG/LLMs	Firecrawl MCP or Jina Reader MCP
Anti-bot, SERP, blocked sites	Bright Data MCP
Custom logic, internal tools	Build your own
Hybrid: markdown + Actor data	Apify + Firecrawl side by side

Apify MCP: full Actor ecosystem. One MCP, many data sources. Best when you need product prices, job listings, social data, or custom crawls.

Firecrawl MCP: clean markdown output. Best for RAG, summarization, and content extraction. Simpler than Apify when you just want the page text.

Bright Data MCP: when Apify or Firecrawl hit blocks. Use for SERP, marketplaces, and targets with heavy anti-bot.

Remote MCP Deployment for Teams

For team use, run an MCP server as a remote HTTP/SSE service instead of stdio. Team members point their Claude/Cursor config to the same URL. Benefits:

• Shared API keys (server-side only)
• Centralized rate limiting and logging
• No need for each user to run npx locally

Apify and Firecrawl MCP can run in HTTP transport mode. Check their docs for remote deployment. For custom servers, use StreamableHTTPServerTransport from the MCP SDK.

Security Considerations

• API key management: store APIFY_TOKEN, FIRECRAWL_API_KEY in env vars. Never commit to git.
• Allowlist Actors: Apify MCP supports --actors to restrict which Actors can run. Use for production.
• Rate limits: MCP tool calls can trigger expensive scrapes. Set limits in your API provider dashboard.
• Input validation: custom servers should validate URLs (block file://, internal IPs) before crawling.

Hybrid Workflows: MCP + Automation

MCP gives Claude on-demand scraping. Combine with automation for scheduled pipelines: use Make.com or n8n to trigger Apify Actors on a schedule, then have Claude query the dataset via MCP when you need analysis. Or use Firecrawl MCP for ad-hoc URL scraping while Apify handles the recurring product price crawl. That separation (scheduled ingestion versus interactive query) keeps context windows small and costs predictable.

Cursor and VS Code Integration

MCP works in Cursor and VS Code (with MCP support). Add the same server config to Cursor Settings → Features → MCP Servers. Developers can ask Cursor to "scrape this URL and summarize" or "run the Amazon Product Scraper for this ASIN" without leaving the editor. Useful for research, debugging scrapers, or quick data pulls during development.

Troubleshooting MCP Scraping Tools

Tools not listed: restart the MCP host. Verify the server starts (run npx -y @apify/actors-mcp-server manually). Check config path: Claude uses claude_desktop_config.json, while Cursor uses its own MCP settings.

Timeout on scrape: increase timeout in tool implementation or reduce input. For Apify, limit maxItems or URLs. Consider an async pattern: trigger run, return run ID, add a separate "get run result" tool that polls.

Rate limit errors: MCP tools share your API quota. If Claude makes many rapid calls, you may hit Firecrawl or Apify limits. Add client-side throttling or use a queue for expensive operations. For a conceptual overview of MCP and how tools map to AI agent capabilities, see MCP servers explained.

Pair MCP servers for coverage

Use Apify MCP for Actor-based scraping and Firecrawl MCP for quick markdown. Claude can choose the right tool per request.

Apify MCP | Bright Data MCP

Frequently Asked Questions

Model Context Protocol lets AI agents call external tools. MCP scraping servers expose scrape/crawl/run_actor tools. Claude invokes them, gets live data, and uses it in replies, with no manual API calls or copy-paste.

Apify MCP: run any Actor, including Google, Amazon, and custom ones. Firecrawl MCP: scrape URL to markdown, crawl, extract. Use Apify for full data pipelines and Firecrawl for clean text extraction and RAG.

Build an MCP server with @modelcontextprotocol/sdk. Implement a tool (e.g., crawl_url) that runs Playwright or your scraper. Add the server to Claude's MCP config with command and args.

Yes. Add Apify, Firecrawl, and Bright Data to the same mcpServers block. Claude sees all tools and picks the appropriate one per request.

Use the env block with variable names (APIFY_TOKEN) and set the actual keys in your system environment or a local .env file. Never hardcode keys in the config file.