Claude Code Web Research Setup: CLAUDE.md, MCP and Hooks (2026)

Contains affiliate links. Disclosure.

Quick Answer

Four files turn a folder into a Claude Code research workspace: a CLAUDE.md with sourcing rules, a project-scope .mcp.json that connects the Apify MCP server with only three tools (search-actors, fetch-actor-details, call-actor), a .claude/settings.json that wires a PreToolUse hook to refuse any Actor run without a spend cap of $1 or less, and a /research skill that runs the whole loop. Every file is below, ready to copy.

Most Claude Code setup guides are about writing code. Mine is mostly about reading the web: pulling search results, pages Claude's built-in fetch can't render, and datasets I want to keep. That job has two failure modes code work does not have. Claude can state things it never sourced, and a scraper can quietly spend money.

This setup addresses both. It is a folder you can recreate in five minutes, and it works the same for a solo researcher as for a team that commits it to a repo.

Let Claude set it up for you

If you would rather not copy files by hand, paste this into Claude Code from an empty folder (or the repo you want to add it to). It asks what it needs, checks the Apify connection, runs one capped test, and only then writes the files. The rest of this post explains every file it creates.

Set up my web-research workspaceInteractive setupPaste into Claude Code

Claude interviews you, connects Apify with three tools, tests one capped run, then writes CLAUDE.md, .mcp.json, the spend-cap hook and a /research skill.

1. Asks your research topic, budget cap and output folder, one question at a time
2. Adds the Apify MCP server at project scope and checks it is connected
3. Runs one small capped test and shows the cost
4. Writes the four files after you confirm

Set up this folder as a web-research workspace for Claude Code. Work step by step and confirm with me before you write any file or start anything that costs money.

1. Ask me these one at a time, waiting for each answer:
   a) What I mostly research (so you can tailor CLAUDE.md).
   b) The maximum spend per Apify run in USD (suggest 1).
   c) Where research output should go (suggest research/).

2. Check whether an MCP server named "apify" is connected (run `claude mcp list` or tell me to run /mcp). If not, propose adding it at project scope with:
   claude mcp add --transport http apify "https://mcp.apify.com?tools=search-actors,fetch-actor-details,call-actor" --scope project
   Then tell me to run /mcp in this session to sign in to Apify with OAuth. Wait until I confirm.

3. Run one small test: use search-actors to find apify/rag-web-browser, then call-actor with input {"query": "<a question from my topic>", "maxResults": 3} and callOptions.maxTotalChargeUsd set to my cap. Read the results with get-dataset-items, then tell me what the run cost (check it with get-actor-run) before going further.

4. Show me, then write after I approve:
   - CLAUDE.md with research rules: cite a URL for every factual claim, try WebSearch/WebFetch before Apify, save raw results and notes under my output folder, never exceed my cap.
   - .claude/hooks/cap-apify-spend.sh: a PreToolUse hook that denies mcp__apify__call-actor when tool_input.callOptions.maxTotalChargeUsd is missing or above my cap, and returns "ask" for Actors not on a short pay-per-event list. Make it executable and check that jq is installed.
   - .claude/settings.json: allow mcp__apify__search-actors, mcp__apify__fetch-actor-details, mcp__apify__get-actor-run, mcp__apify__get-dataset-items and WebSearch; put mcp__apify__call-actor under "ask"; register the hook under hooks.PreToolUse with matcher "mcp__apify__call-actor" and command "${CLAUDE_PROJECT_DIR}/.claude/hooks/cap-apify-spend.sh".
   - .claude/skills/research/SKILL.md with disable-model-invocation: true, so I run it as /research <question>.

5. Test the hook by piping two sample JSON inputs into the script (one with no cap, one with a cap under the limit) and show me the output. Finish with a summary of every file you created.

Claude asks before each change it makes. Read each step before you approve it.

The file tree

This is the whole workspace. Nothing here is specific to one topic. The research rules live in CLAUDE.md; everything under .claude/ is reusable.

research-workspace/
├── CLAUDE.md                      # research rules Claude reads every session
├── .mcp.json                      # Apify MCP server, three tools only
├── .claude/
│   ├── settings.json              # permissions + hook wiring
│   ├── hooks/
│   │   └── cap-apify-spend.sh     # refuses runs without a $1 cap
│   └── skills/
│       └── research/
│           └── SKILL.md           # the /research command
└── research/
    ├── raw/                       # dataset dumps, one JSON file per run
    └── notes/                     # Markdown write-ups with sources

CLAUDE.md: the research rules

CLAUDE.md is loaded at the start of every session, so it is where rules belong that should hold no matter what you ask. Anthropic's guidance is to keep each one under 200 lines, because longer files cost context and get followed less reliably. Mine is about 25.

# Research workspace

This folder is for web research, not software. Output goes to research/.

## Sourcing
- Every factual claim in a note gets a source URL next to it. No URL, no claim.
- If two sources disagree, say so and give both. Do not average them.
- Mark anything you could not verify as UNVERIFIED rather than dropping or guessing it.
- Prices, dates and version numbers must come from the vendor's own page.

## Tool order (cheapest first)
1. WebSearch and WebFetch for a handful of normal pages. They are free.
2. Apify only when WebFetch fails (JavaScript-heavy page, blocked, login wall
   you are allowed past) or when you need more than ~20 pages or a full SERP.
3. Before any Apify run, use fetch-actor-details to read the input schema
   and pricing, then set callOptions.maxTotalChargeUsd to 1 or less.
   A hook rejects runs without it.

## Preferred Actors
- apify/rag-web-browser: search + fetch top results as Markdown
- apify/google-search-scraper: raw Google results at scale
- apify/web-fetch: one URL to Markdown with JS rendering

## Files
- Save every Apify dataset to research/raw/YYYY-MM-DD-<topic>.json.
- Write findings to research/notes/<topic>.md: summary first, then
  findings with sources, then open questions.

The tool order section does the most work. Claude Code already has free WebSearch and WebFetch tools, and for a few static pages they are enough. Telling Claude to escalate to Apify only when those fail keeps paid runs for the jobs that need them: rendered pages, blocked sites, and volume.

.mcp.json: Apify with three tools

A project-scope .mcp.json sits in the folder root, so anyone who clones the workspace gets the same server. Claude Code asks you to approve a project-scoped server the first time you open the folder.

{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=search-actors,fetch-actor-details,call-actor"
    }
  }
}

You can create the same entry from the terminal with claude mcp add --transport http apify "https://mcp.apify.com?tools=search-actors,fetch-actor-details,call-actor" --scope project.

Three details matter here:

• "type": "http" is required. Claude Code treats an entry with a url but no type as a broken stdio server and skips it.
• No token in the file. The hosted server supports OAuth, so you sign in once with /mcp inside Claude Code and nothing secret gets committed.
• The tools parameter replaces the defaults. When you pass it, the server loads only what you list. get-dataset-items, get-actor-run, get-key-value-store-record and abort-actor-run are added automatically whenever call-actor is present, so you do not list them. The full tool table is in the Apify MCP server README.

Why only three tools

The usual reason given for trimming MCP tools is context, and it is weaker than it used to be: Claude Code's tool search is on by default and loads only tool names until Claude needs a definition. The reason that still holds is control.

By default the Apify server also exposes two Actor tools directly, apify--rag-web-browser and apify--web-fetch. Each is a paid run that does not go through call-actor. Leaving them out means every paid run in this workspace passes through one tool, and one tool is easy to guard with a single hook. Claude can still run those two Actors, just through call-actor. We also measured it: what each MCP server costs in context, with tool search on and off.

For the Claude Desktop version of tool scoping, including the older --actors flag, see the Apify MCP setup for Claude Desktop.

settings.json: permissions and the hook

.claude/settings.json does two things: it pre-approves the free tools so you are not prompted for every search, and it registers the spend-cap hook.

{
  "permissions": {
    "allow": [
      "mcp__apify__search-actors",
      "mcp__apify__fetch-actor-details",
      "mcp__apify__get-actor-run",
      "mcp__apify__get-dataset-items",
      "WebSearch"
    ],
    "ask": ["mcp__apify__call-actor"]
  },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "mcp__apify__call-actor",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/cap-apify-spend.sh"
          }
        ]
      }
    ]
  }
}

MCP tools are named mcp__<server>__<tool>, so mcp__apify__call-actor is the Apify server's run tool. The matcher contains only letters, underscores and hyphens, which Claude Code compares as an exact string, so the hook fires on that one tool and nothing else.

The ask rule means Claude Code still asks you before every run, even when the hook lets it through. That is the setting I keep for interactive research. If you later run this workspace unattended, remove the ask line; the hook keeps enforcing the cap on its own.

The spend-cap hook

The Apify server's call-actor tool accepts a callOptions.maxTotalChargeUsd field that caps what a pay-per-event Actor can bill for one run. The hook makes that field mandatory.

#!/bin/bash
# .claude/hooks/cap-apify-spend.sh
# Runs before every Apify call-actor. Denies any run without a spend cap of $1 or less,
# and asks you first when the Actor is not on the pay-per-event list (where the cap is ignored).
input=$(cat)
actor=$(jq -r '.tool_input.actor // ""' <<<"$input")
cap=$(jq -r '.tool_input.callOptions.maxTotalChargeUsd // "none"' <<<"$input")

# Actors billed per event, so maxTotalChargeUsd actually caps the bill.
capped_actors="apify/rag-web-browser apify/google-search-scraper apify/web-fetch"

if [ "$cap" = "none" ] || ! jq -e '.tool_input.callOptions.maxTotalChargeUsd <= 1' <<<"$input" >/dev/null; then
  jq -n --arg cap "$cap" '{
    hookSpecificOutput: {
      hookEventName: "PreToolUse",
      permissionDecision: "deny",
      permissionDecisionReason: "Spend cap missing or above $1 (got: \($cap)). Retry with callOptions.maxTotalChargeUsd set to 1 or less."
    }
  }'
  exit 0
fi

if [[ " $capped_actors " != *" $actor "* ]]; then
  jq -n --arg actor "$actor" '{
    hookSpecificOutput: {
      hookEventName: "PreToolUse",
      permissionDecision: "ask",
      permissionDecisionReason: "\($actor) is not on the pay-per-event list, so the $1 cap may not apply. Approve only if you accept the cost."
    }
  }'
  exit 0
fi

exit 0  # capped, known Actor: normal permission flow applies

Make it executable with chmod +x .claude/hooks/cap-apify-spend.sh. It needs jq.

How it behaves:

Claude's call	Hook result
No maxTotalChargeUsd	Denied. Claude reads the reason and retries with a cap.
Cap above $1	Denied, same retry.
Cap of $1 or less, Actor on the list	Passes to the normal permission flow.
Cap of $1 or less, Actor not on the list	Ask: you approve or reject it yourself.

The second check exists because the cap does not apply to every Actor. Apify's own tool description says maxTotalChargeUsd applies to pay-per-event Actors only and is ignored otherwise. apify/website-content-crawler, for example, is billed by platform usage rather than per event, so a $1 cap on it does nothing. When you add an Actor to the list, check its pricing first: fetch-actor-details returns it, and so does the Actor's Store page.

You can test the hook without starting Claude by piping a sample input into it:

echo '{"tool_name":"mcp__apify__call-actor","tool_input":{"actor":"apify/rag-web-browser","input":{"query":"test"}}}' \
  | .claude/hooks/cap-apify-spend.sh

That input has no cap, so the script prints a deny decision.

Two limits to know. The cap bounds what one run can bill, not how much work the Actor does, so still set the Actor's own limits (maxResults and similar) in the input. And it caps each run, not your day; ten capped runs can still cost ten dollars.

The /research skill

The skill packages the loop I run most: question in, sourced note out. I set disable-model-invocation: true so it only runs when I type /research, never on its own. How skills work in general is covered in building a Claude Skill for web scraping; this is just the file.

---
name: research
description: Research a question on the live web and write a sourced note.
argument-hint: "[question]"
disable-model-invocation: true
---

Research this question: $ARGUMENTS

1. Restate the question and list 3-5 sub-questions you need answered.
2. For each sub-question, try WebSearch and WebFetch first.
3. If you need a full SERP, a JavaScript-rendered page, or more than ~20 pages,
   use the Apify server: fetch-actor-details, then call-actor with
   callOptions.maxTotalChargeUsd set to 1 or less, then get-dataset-items.
   Save the items to research/raw/.
4. Write research/notes/<short-topic>.md:
   - a 3-sentence answer first
   - findings, each with its source URL
   - disagreements between sources
   - what you could not verify
5. Tell me what the Apify runs cost in total (from get-actor-run).

First run

1. Open the folder in Claude Code and approve the apify server when asked.
2. Run /mcp, pick apify, and sign in to Apify in the browser. A free Apify account includes monthly credits, which covers a lot of capped research runs.
3. Run /context and check that CLAUDE.md is listed under memory files.
4. Try /research which European cities added short-term rental caps this year.

The first time Claude reaches for call-actor without a cap, you will see the hook deny it and Claude retry with one. That retry is the setup working.

What one capped run returns and costs

This is the call the setup prompt uses for its test, apify/rag-web-browser with {"query": "European cities short-term rental caps 2026", "maxResults": 3}. Below is real output from a test run on 2026-10-21, trimmed to the fields that matter (the full items also carry the page as Markdown):

[
  {
    "searchResult": {
      "title": "Europe's war on short-term rentals is reshaping travel",
      "description": "Apr 9, 2026 — Amsterdam is cutting its annual rental cap to just 15 nights in core neighbourhoods. Paris has reduced its cap to 90 days, ..."
    },
    "metadata": { "url": "https://whiteskyhospitality.com/europes-war-on-short-term-rentals-is-reshaping-travel/" },
    "crawl": { "httpStatusCode": 200, "requestStatus": "handled" }
  },
  {
    "searchResult": {
      "title": "Proposed EU rules would curb Airbnb, short-term rental ...",
      "description": "Sep 4, 2026 — The EU is seeking to harmonise rules across Europe after authorities in cities including Paris, Barcelona and Venice cracked down on ..."
    },
    "crawl": { "httpStatusCode": 500, "httpStatusMessage": "Request blocked - received 401 status code.", "requestStatus": "failed" }
  },
  {
    "searchResult": {
      "title": "Europe Short Term Rentals in 2026",
      "description": "Dec 29, 2025 — No. Europe is not banning short term rentals in 2026. Instead, the EU is enforcing standardized registration, data sharing, and verific..."
    },
    "metadata": { "url": "https://internationalreal.estate/insights/europe-short-term-rentals-in-2026" },
    "crawl": { "httpStatusCode": 200, "requestStatus": "handled" }
  }
]

The run took about 24 seconds and billed $0.0085: one search event, two fetch events and the Actor-start events. The blocked page was not billed as a fetch. Per-event prices drop on higher Apify plans, so a free-plan account pays slightly more, around a cent for the same run. At that rate the $1 cap is far above what a normal research question costs, which is the point: it only bites when something goes wrong, like Claude asking for 500 results.

Two things in that output are why the CLAUDE.md rules exist. One of three sources failed to load, and Claude should say so in the note rather than quietly answering from two. And the search snippets disagree in tone ("war on short-term rentals" vs "not banning"), which is exactly the kind of conflict the rules tell Claude to report instead of smooth over.

What this setup does not do

• It is not a budget. It caps runs, not totals. Check spend in the Apify Console under Billing.
• It does not check the sources themselves. The rules make Claude cite URLs; whether a source is any good is still your call.
• It does not run on a schedule. It is built for interactive sessions. For recurring jobs, Claude Code's headless mode with cron or a Claude Cowork scheduled task is the better fit.
• It is not the only way in. If you would rather not touch config files at all, Claude Cowork with Apify gets you live web data with no setup beyond a connector.

Add an Actor to the capped listPaste into Claude Code

Checks an Actor's pricing model before adding it to the hook.

Use fetch-actor-details on <username/actor-name> and tell me its pricing model. If it is pay-per-event, add it to capped_actors in .claude/hooks/cap-apify-spend.sh and to the Preferred Actors list in CLAUDE.md. If it is not, explain how its cost is billed and what input limits I should set instead.

FAQ

Frequently Asked Questions

Both go in the project root. CLAUDE.md can also live at .claude/CLAUDE.md. A project-scope .mcp.json is shared with everyone who clones the folder, and Claude Code asks each person to approve its servers the first time.

Add a tools parameter to the server URL, for example https://mcp.apify.com?tools=search-actors,fetch-actor-details,call-actor. Only the listed tools load. The dataset and run tools are added automatically whenever call-actor is present.

Yes. A PreToolUse hook on mcp__apify__call-actor can deny any call that lacks callOptions.maxTotalChargeUsd or sets it too high. The cap only applies to pay-per-event Actors, so Actors billed another way need their own input limits.

No. The hosted server at mcp.apify.com supports OAuth, so you sign in once with /mcp inside Claude Code. That keeps credentials out of a file you might commit.

Next steps

Copy the four files, run /research on something you actually need this week, and see where it falls short. That will tell you what to put in CLAUDE.md next. For the reasoning behind giving Claude a scraper at all, read Claude real-time web access with Apify. For research prompts you can paste straight in, see Claude prompts that use live web data.

Create a free Apify account for your research runs →