Skip to main content
use-apify.com

AI agents: guides & tutorials

Autonomous agents browse, extract, and act without brittle selectors. Orchestration, guardrails, and Apify-backed tools for dependable scraping.

47 articlesPage 1 of 5

View all tags

AI agents handle web tasks autonomously, browsing and extracting without selectors that break on every redesign. These guides cover orchestration, guardrails, and Apify-backed tools for dependable agent scraping.

The hard part is reliability: agents need bounded actions, retries, and observable runs to be trusted in production. Apify supplies the scraping tools they invoke. Below you will find practical patterns for building robust AI agents.

Related topics

Proxies10 min read

Proxies for AI Agents (2026): Web Access at Machine Speed

· 10 min read
Yassine El Haddad
Software Developer & Automation Specialist

An AI agent that browses the web hits every limit a human browser does, but faster: one IP making thousands of requests looks like an attack, not a user. Proxies give each agent request a different, better-class IP address, which is what keeps agent web access alive after the first hundred fetches. For agent builders, the proxy is not an optimization. It is the difference between a demo and something that runs all week.

Agents changed the shape of web traffic. A single agent session can fan out across dozens of domains in a minute, retry aggressively on failure (a behavior LLMs are prone to), and re-fetch pages it has already seen because nobody wired up a cache. Sites answer that pattern with rate limits, datacenter-IP blocks and bot challenges. The fixes are boring and known: pool classes, rotation, sessions, caching, backoff. This page maps those onto agent architectures, names the providers that fit, and ends with a setup prompt you can hand to your own agent.

Why agents get blocked faster than scripts​

A traditional scraper is engineered restraint: polite delays, configured retries, one target at a time. An agent is improvising. Four behaviors in practice:

  • Burst fan-out. Research tasks hit many domains concurrently, all from your server's single IP.
  • Retry storms. A 403 reads as "try again" to an agent, not "back off". Ten retries in ten seconds is a self-inflicted rate limit.
  • No memory of fetches. Without caching, the same URL is fetched fresh in every context window.
  • Datacenter origin. Most agents run on cloud hosts, whose IP ranges are pre-flagged by every serious anti-bot vendor.

The result: agents meet blocks that a carefully written crawler would never trigger. The mitigation stack is the crawler's stack, adapted.

Where the proxy sits in an agent stack​

Agents reach the web through tools: HTTP fetch tools, browser automation, and increasingly MCP servers that wrap both. The proxy can enter at three layers:

  1. Environment-level. HTTP_PROXY/HTTPS_PROXY environment variables, picked up by most HTTP clients automatically. Cheapest to set up, applies to everything, including SDK calls you forgot about.
  2. Tool-level. Pass the proxy URL into each browser or fetch tool's config (Playwright proxy option, requests proxies dict). Precise control: one tool proxied, others direct.
  3. Service-level. The agent calls a scraping API or MCP server that applies its own proxy pool server-side. No proxy management in your process at all.

For agent platforms, layer 3 is increasingly the default: Apify runs Actors through its own proxy groups, and scraping APIs like ScraperAPI apply their pools before your request ever leaves. Layer 1 and 2 matter when you control the runtime and want the cost control of your own pool.

Choosing the proxy class for agent workloads​

WorkloadClassWhy
Fetching public APIs, docs, feedsdatacentercheap ($0.30-$0.60/GB), fast, no ASN checks to beat
Research across many websitesresidential rotationfresh IP per request, survives soft blocks
Logged-in flows, cart states, watchlistsresidential sticky or ISPone identity across the task's steps
Social and mobile-first targetsmobilecarrier IPs behind CGNAT, hardest to block
Geo-specific checksresidential with country/city targetingthe agent sees what a local user sees

Two rules of thumb. First, match the session to the agent's identity, not the request: a research agent that browses like one user should hold one sticky IP for a task; a monitoring agent that fans out should rotate per request. Second, cache: an agent that refetches because the context window forgot is paying twice for the same bytes. A content-addressed cache with a TTL (an hour for research, a day for monitoring) cuts proxy spend more than any provider switch.

Providers that fit agent workloads​

Verified November 2026; all of these appear in our other guides with full pricing:

  • DataImpulse: $1/GB residential, $0.50/GB datacenter, pay-as-you-go, traffic never expires. Fits agents precisely because agent traffic is bursty and unpredictable: no subscription to waste. A threads parameter caps concurrency so a runaway agent loop cannot eat the balance.
  • Floppydata: one balance for residential, mobile and datacenter at $1/GB (30-day validity), plus a free Cloud Browser aimed at AI agents: a managed Chromium over CDP, billing only proxy traffic. Good when the agent needs a real browser, not just an HTTP client.
  • Evomi: residential from $0.49/GB, and an MCP server that lets Claude scrape, crawl and manage proxy credentials directly. The closest to proxies-as-an-agent-tool.
  • Bright Data: the enterprise option, with its own MCP server and web data APIs for AI pipelines; our Bright Data for AI walkthrough covers the product set.
  • Apify: if the agent's jobs can run as Actors, the platform's proxy groups (residential, datacenter, SERP) come with scheduling, storage and retries included.

For the MCP angle specifically, see Bright Data's MCP server and the best MCP servers roundup. For driving scraping APIs from Claude specifically, our Claude API with Bright Data proxies guide has working code.

The cost problem agents create​

Agents burn proxy traffic in ways scripts do not: refetching without caches, fetching pages when an API endpoint would do, rendering pages headlessly for data that sits in the HTML. Three controls, in order of impact:

  1. Fetch budget. Hard cap on requests or GB per task. DataImpulse's traffic-limit setting and Evomi's PAYG balance (no expiry) both work as natural circuit breakers.
  2. Cache layer. Even a naive disk cache keyed on URL saves 30-50% of agent traffic in practice, because agents revisit the same sources.
  3. Class routing. Route doc and API fetches through datacenter at $0.50/GB, and reserve residential for the protected targets. Routing everything through residential is the most common waste.

Setup prompt: give your agent a proxy layer​

The prompt below sets up a proxy-aware fetch layer with an interview first. It works in Claude Code, Cowork, or Claude Desktop with a shell tool.

What proxies do not solve​

A proxy changes the IP. It does not change the client. Anti-bot systems score the TLS fingerprint, the HTTP/2 settings, the mouse-less interaction pattern, and the request cadence, none of which a proxy touches. Agents that get blocked through clean residential pools are usually being fingerprinted, not IP-blocked, and the fix is browser-grade tooling and human pacing, covered in our anti-detection reference. For the engineering analysis of how the WAF inspection stack is ordered, see bypass architecture for deep-packet WAFs.

Claude37 min read

60 Claude Prompts That Use Live Web Data, Sorted by Role

· 37 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Each prompt below tells Claude which Apify Actor to run, what to pull, and what to hand back. Paste it into Claude with the Apify connector switched on. Every prompt makes Claude check the Actor's inputs and price first and wait for your OK before it spends anything. Each one links to a full guide where we ran the workflow for real on 2026-10-21.

Generic prompt lists ask Claude to "research competitors" and get an answer from training data. These prompts name a real Actor, real input fields and the traps our test runs hit, so Claude works from today's web, not last year's.

Claude18 min read

Price Your Airbnb Against Nearby Comps Every Week with Claude

· 18 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Set up a weekly Claude Cowork scheduled task that runs the Airbnb Scraper API (codepoetry) on Apify for about 50 comparable listings near yours. Claude prices the next four weekends against them, works out each comp's calendar-implied occupancy, pulls the amenities guests praise from their reviews, and writes a one-page pricing memo. A full weekly run costs about $1.27 in Apify usage on the free plan, or about $0.79 on Starter.

"How should I price my Airbnb?" has one honest answer: the same way the listings guests compare you with are priced, on the same dates. That comp set changes every week. New listings open, neighbours drop their rates for a quiet weekend, and a festival fills the calendar three streets over. This post sets up a Monday-morning memo that tracks it for you, with no code.

Claude21 min read

Voice of Customer Analysis with Claude: Find Feature Gaps (2026)

· 21 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Connect Apify's MCP server to Claude, then pull reviews of your product and three competitors from G2, the App Store and Reddit, plus transcripts of YouTube review videos. Claude codes every review against one theme list, counts how often and how badly each theme hurts, and returns a ranked feature-gap table with verbatim quotes and source links for your PRD. A run of about 1,200 items costs roughly $5 in Apify credits.

Most product teams read reviews the same way: someone skims the latest 1-star posts before planning, remembers the loudest three, and those become "what customers want". That is anecdote, not analysis. The fix is not more reading. It is coding every review against the same theme list, for you and your competitors, so you can see where they are praised for something you are criticised for.

Claude16 min read

Buying Signals with Claude: Hiring, Funding and Tech Changes

· 16 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Give Claude your list of target accounts. Once a month it pulls three kinds of public evidence through Apify: new job ads (LinkedIn), funding news (Google News) and the technologies on each company's website. It throws out anything it cannot tie to the right company, scores what is left, and logs each real signal as a note and follow-up task in HubSpot, with the source link. For 50 accounts the scraping costs roughly $1 to $2 a month.

A lead list tells you who could buy. A buying signal tells you who has a reason to buy now: they just posted three RevOps roles, closed a round, or swapped their analytics tool. Sales teams pay intent-data vendors for this. For a list of 20 to 200 named accounts, most of the same evidence is public, and the hard part is not collecting it but reading it carefully enough to tell a real signal from a name collision. That reading is the part Claude does well.

This post covers the signal sweep on accounts you already know. If you still need to build the account list, start with the AI lead generation playbook.

Claude16 min read

Claude for Recruiting: Build a Candidate Shortlist from Job Boards and Team Pages

· 16 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Paste a role spec into Claude Cowork. Claude uses Apify scrapers to pull job ads for the same role in your market, turns the hiring companies into a target list, reads those companies' public team pages, and scores the people it finds against your must-haves. Every score cites the line it came from. You review the list and decide who to contact; Claude only drafts outreach for the people you approve. Scraping for one role costs well under a dollar. The legal limits (GDPR, LinkedIn's and GitHub's terms) shape the workflow more than the tooling does, so they get their own section.

Sourcing tools sell you a database. This workflow builds a small, fresh one per role from pages that companies chose to publish, and it keeps a person in charge of every decision about a human being. That second part is not decoration: in the EU, the same scoring step that saves you an afternoon is the part regulators care most about.

Claude14 min read

Claude Code for Non-Coders: Your First Day, Install to Undo

· 14 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Install Claude Code with one command, sign in with a paid Claude plan (Pro, Max, Team or Enterprise) or a Console account, and switch to Manual mode with Shift+Tab for your first day so Claude asks before every change. Undo with /rewind (or Esc twice), but know its blind spot: it restores files Claude edited, not files it moved, renamed or deleted with terminal commands. Ask for a log and an undo file before any bulk move.

Most "Claude Code for beginners" guides jump straight to building an app. This one stops earlier, at the part that decides whether you trust the tool: what the permission prompts are asking you, which answer to pick, and how to get back to where you were when something goes wrong. You finish with three small jobs done on your own files, and you know exactly which of them you could undo.

Claude12 min read

MCP Context Cost in Claude Code: 12 Servers Measured

· 12 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

With Claude Code's default tool search, 12 popular MCP servers added only 5,848 input tokens to a session in our test. With tool search turned off, the same 12 servers added 63,186 tokens, about 6% of a 1M-token window and nearly a third of a 200K one. The biggest single server was GitHub's remote server: 46 tools, about 19.8k tokens when loaded. The startup cost that remains with tool search on is mostly server instructions, not tool definitions.

Every Claude Code setup guide warns that MCP servers eat your context window. Almost none of them give numbers, and the advice was written before tool search became the default. So we measured it on 21 October 2026: 12 servers, one at a time and all together, with tool search on, off and in auto mode.

Claude13 min read

Claude Code SEO Content Briefs from the Live Google Top 10

· 13 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Give Claude Code a keyword and one Apify Actor run, the Google Search Scraper with its Full website content add-on, returns the live top 10 with the full text of every ranking page, plus People Also Ask and related searches. Claude then pulls out each page's H2s and word count and writes a brief in your template: what every ranking page covers, which questions the SERP asks, and what the top pages miss. Our test run for one keyword took about two minutes and cost $0.048.

Most "AI content brief" tools work from the same raw material: the current top results for a keyword. You can get that raw material yourself, hand it to Claude Code, and keep the output in your own template and your own repo. This post shows the whole loop, including a real run for the keyword "content brief template" and what it turned up.

Claude15 min read

Claude Job Search: A Daily Digest of New Jobs Ranked Against Your CV

· 15 min read
Yassine El Haddad
Software Developer & Automation Specialist
Quick Answer

Create a Claude Cowork scheduled task that runs every morning. It calls two Apify job scrapers (LinkedIn and Indeed) for postings from the last 24 hours, scores each one against the CV you pasted into the task, and gives you the top 5 with tailoring notes. The notes can only reuse lines from your CV, so Claude cannot invent experience for you. At the job volumes below, the scraping stays inside Apify's free $5 monthly credit.

Job alerts from the boards themselves match on keywords, so you get 40 emails a week and skim them all. The job that matters is the reverse: read every new posting, compare it to what you have actually done, and surface the few worth an application today. That is reading work, which is what Claude is good at, provided something fetches the postings for it.

Guides on this site

Frequently asked questions

Frequently Asked Questions

An AI agent can autonomously research a list of companies (scraping their sites, LinkedIn, and news), compile findings into a structured report, and flag action items — without step-by-step human instruction. Other real uses: monitoring competitor pricing daily and drafting a summary email, finding and qualifying leads from directories, or crawling job boards and filtering openings by criteria. The agent decides what to fetch, processes results, and acts on them.

The practical path: give the agent a tool that calls an Apify actor (via the Apify MCP server or REST API), define the goal and output format clearly, and set stop conditions. Use LangGraph or CrewAI for orchestration if you need multi-step workflows. Start with a narrow, testable task — e.g. "given a list of 20 company domains, return the CEO name and headquarters city" — before scaling to open-ended research.

Autonomous agents can be 5–20x more expensive than a single-pass LLM extraction because they loop: planning, fetching, reading, re-planning. For bulk data collection, deterministic scrapers are cheaper. Agents shine on tasks where the path is unknown — researching an unfamiliar company, navigating a dynamic site, or resolving ambiguous queries. Limit iteration counts and cache fetched pages to control costs.

LangGraph is popular for stateful, multi-step workflows with clear control flow. CrewAI suits collaborative multi-agent setups. For simpler pipelines, LangChain's agent executor is faster to set up. Apify integrates with all three via the MCP server or REST API, so agents can trigger real scraping jobs rather than simulating browsing with an LLM. Choose the framework your team can debug, not the one with the most features.