Skip to main content
use-apify.com

Web scraping: guides & tutorials

Extract structured data from websites with code: crawling, parsing, and anti-bot handling for engineers building datasets and automations on Apify.

125 articlesPage 1 of 13

View all tags

Web scraping turns public web pages into structured datasets you can analyze, monitor, or feed into AI. These guides cover the full workflow: choosing between HTTP requests and headless browsers, parsing HTML with CSS or XPath selectors, handling pagination and infinite scroll, and getting past rate limits and bot detection without breaking sites or laws.

Whether you write your own scraper in Python or JavaScript or run a ready-made actor from the Apify Store, the patterns are the same. Start with a small, well-behaved crawl, add proxies and retries as targets get stricter, and export clean JSON or CSV your pipeline can trust. The tutorials below take you from a first script to production crawls running on a schedule.

Related topics

Proxies9 min read

Best Datacenter Proxies 2026: From $0.30/GB or $0.02/IP

· 9 min read
Yassine El Haddad
Software Developer & Automation Specialist

Datacenter proxies are server IPs: the fastest and cheapest proxy class, and the easiest for a target to detect, because bot filters index commercial ASN ranges on sight. Buy them for work that does not check ASN class, APIs, open data, staging, anything unprotected, and the November 2026 floor is $0.45/GB (Evomi), $0.50/GB (DataImpulse), $0.49/IP (Proxy-Seller IPv4) or $0.02/IP (Proxy-Seller IPv6).

Quick Answer

Datacenter pricing splits into two models (verified November 2026). Per GB, for bursty scraping: Evomi at $0.45/GB pay-as-you-go ($0.30/GB at 1 TB, unlimited concurrency) and DataImpulse at $0.50/GB flat ($0.45 at 1 TB, traffic never expires). Per IP, for steady throughput: Proxy-Seller IPv4 from $0.49/IP and IPv6 from $0.02/IP, Floppydata static from $0.79/IP, IPRoyal dedicated from $1.39/IP with unlimited bandwidth. Enterprise: Bright Data from $0.60/GB or $0.90/IP shared; Oxylabs from $0.44/GB shared or $1.20/IP dedicated across 188 countries.

Datacenter proxy comparison (November 2026)​

ProviderModelEntryBest rateNotes
Evomiper GB$0.45$0.30 at 1 TB ($300/mo)unlimited concurrency, 181 countries
DataImpulseper GB$0.50$0.45 at 1 TB ($450)traffic never expires, 20M IPs, sub-100 ms
Bright Dataper GB or per IP$0.60/GB · $1.40 shared/IP$0.40/GB · $0.90 shared/IP at 1,0001.3M+ IPs, 98 countries, 100 GB/mo per IP
Oxylabsper GB or per IP$0.60/GB shared · $1 shared/IP$0.40/GB · $1.20 dedicated/IP at volume188 countries, ~1.9M dedicated IPs
Proxy-Sellerper IP$0.49 IPv4 · $0.02 IPv6volume discounts up to 30%AUTOMATE15 for 15% off, 40+ countries IPv4
Floppydataper GB or per IP$1/GB rotating · $0.79 static/IP$1.50/GB non-expiringsame balance covers residential and mobile
IPRoyalper IP$1.39 (90-day)$1.57 (30d) · $1.48 (60d)unlimited bandwidth, 100 GB/mo per IP, 10K+ concurrent connections

NetNut, which we covered in earlier datacenter comparisons, was seized by the FBI in July 2026 and its parent paused the service; it is no longer recommendable and does not appear here.

Per-GB providers: bursty workloads​

Evomi: $0.45/GB, cheapest at volume​

Evomi bills datacenter per GB with unlimited concurrency: $0.45/GB pay-as-you-go, $0.40 on 200 GB, $0.35 on 500 GB, $0.30 on 1,000 GB ($300/month). Country and ISP targeting are included across 181 countries. The floor rate of $0.30/GB is the cheapest datacenter bandwidth on this page. The catch is structural: every proxy request bills a minimum of 512 bytes, which only matters at very high volumes of tiny requests.

DataImpulse: $0.50/GB flat, never expires​

DataImpulse prices datacenter at $0.50/GB ($5 for 10 GB intro) down to $0.45/GB at the 1 TB tier ($450), from a 20M-IP pool in 195+ locations with sub-100 ms latency and a 99.9% advertised uptime. Purchased traffic never expires, sticky sessions hold up to 30 minutes, and IP-whitelist authentication is available. For uneven calendars, non-expiring datacenter GBs at a flat rate beat a monthly commitment.

Bright Data and Oxylabs: enterprise per-GB​

Bright Data sells datacenter per GB at $0.60/GB pay-as-you-go, dropping to $0.51 (1 TB, $499/mo), $0.45 (2 TB) and $0.42 (5 TB), alongside per-IP shared plans from $1.40/IP at 10 to $0.90 at 1,000. Oxylabs prices shared datacenter per GB from $0.60 down to $0.40 at volume (dedicated per-IP from $2.00 to $1.20 across 188 countries). Both make sense when datacenter rides on a broader enterprise contract.

Per-IP providers: steady throughput​

When a job pushes consistent traffic through a known set of addresses, per-IP pricing with unlimited bandwidth beats per-GB, and the crossover arrives fast: at roughly 30 GB/month through one address, a $0.79 static IP has already beaten $0.50/GB metered traffic.

Proxy-Seller: IPv4 from $0.49/IP, IPv6 from $0.02/IP​

Proxy-Seller lists dedicated IPv4 from $0.49/IP across 40+ countries, with volume discounts up to 30% and the AUTOMATE15 code stacking another 15%. IPv6 addresses at $0.02/IP are the cheapest proxy product on the market, period, and fit IPv6-native targets (Google, Facebook, Reddit). Full tier ladders price out in the on-site configurator, and dead addresses are replaced or refunded within 24 hours.

Floppydata: static from $0.79/IP​

Floppydata sells static datacenter IPs from $0.79/IP/month in bundles of 20, 50 or 100 with unlimited bandwidth, and its rotating balance ($1/GB) also covers datacenter exits when you want rotation instead of fixed addresses. Twenty static IPs run about $16/month, the cheapest small bundle here.

IPRoyal: dedicated with headroom​

IPRoyal dedicated datacenter runs $1.57/IP (30 days), $1.48 (60), $1.39 (90, the popular tier), with unlimited traffic, ~10 ms latency, ~2 Gbps throughput and 10,000+ concurrent connections per address per its published specs, plus a 100 GB/month fair-use allowance pooled per IP. The most expensive entry here, and the only one publishing throughput specs, which matters for load-heavy jobs.

When datacenter is the wrong purchase​

Datacenter fails fast on Amazon, LinkedIn, Instagram, Google SERP (without CAPTCHA handling) and most Cloudflare-fronted sites; the class decision guide covers the escalation path to residential, ISP and mobile. The ten-request test settles it: send ten requests through a datacenter IP; 403s or challenges mean buy a different class, not more datacenter IPs.

Proxies10 min read

Best Proxy Services 2026: The Seven Providers We Recommend

· 10 min read
Yassine El Haddad
Software Developer & Automation Specialist

The best proxy service depends on one question, which class of IP and which billing model your workload needs. Across the providers we cover on this site, seven earn a recommendation in 2026: Floppydata, DataImpulse, Evomi, Proxy-Seller, Bright Data, IPRoyal and Oxylabs. Every price below was verified against the provider's own pricing page in November 2026, and this hub links to the specialist roundup for each proxy class.

Quick Answer

Pick by workload, not by brand. Cheapest flat-rate residential: DataImpulse, $1/GB, traffic never expires. Cheapest steady-volume residential: Evomi, $0.49/GB on its 100 GB monthly plan. Most flexible budget product: Floppydata, one balance for residential, mobile and datacenter at $1/GB. All five proxy classes on one invoice: Proxy-Seller, IPv6 from $0.02/IP. Enterprise scale and hardest targets: Bright Data, 400M+ IPs. Non-expiring subscription value: IPRoyal. SERP and e-commerce APIs at scale: Oxylabs.

The seven, compared​

ProviderResidential entryPoolProxy typesBillingBest for
Floppydata$1/GB (30-day validity; $1.50 non-expiring)148 countriesresidential, mobile, datacenter, static ISP, static DC, scraper, browserper GB / per IPmobile at the residential rate, one balance for everything
DataImpulse$1/GB, never expires90M+residential, mobile, datacenter, premiumper GBflat-rate pay-as-you-go, uneven crawl calendars
Evomi$0.49/GB (100 GB plan) or $0.99 PAYG54M+residential, mobile, datacenter, ISP, Scraper API, browserper GB / per IPlowest steady-volume residential price, managed scraping
Proxy-Seller$1.99 trial; $2.20/GB at 100 GB47M+IPv4, IPv6, ISP, residential, mobileper IP / per GBevery proxy class, per-IP pricing, IPv6
Bright Data$8/GB list, $4 with RESIGB50 promo400M+residential, datacenter, ISP, mobile, unlocker, SERPper GB / per IPhardest targets, compliance, enterprise
IPRoyal$7.00/GB at 1 GB, $1.75 custom tiers64M+residential, datacenter, ISP, mobileper GB / per IPnon-expiring traffic, per-IP datacenter and ISP
Oxylabs$6/GB Starter, $2.50 at Enterprise175M+residential, mobile, ISP, datacenter, SERP/e-com APIsper GB / per IPSERP monitoring, e-commerce scraping APIs

The order matches our residential proxies roundup: Floppydata first for the one-balance product, DataImpulse and Evomi on price, Proxy-Seller on breadth, then the enterprise pair and IPRoyal on value. On any single dimension the order shifts, which is what the rest of this page is for.

Floppydata: one balance, three IP types​

Floppydata sells gigabytes that work as residential, mobile or datacenter exits, switched in the proxy username, all at $1/GB (30-day validity; $1.50/GB never expires). Mobile traffic at the residential rate is the structural saving: most providers charge two to four times more for carrier IPs. Static ISP addresses run from $1.79/IP/month and static datacenter from $0.79/IP/month, both with unlimited bandwidth. The weak spots: no published residential pool size, 148-country coverage (narrower than the 195-country providers), and a 19 GB minimum order.

Best for: teams mixing mobile, residential and datacenter work on one budget. Full detail: Floppydata pricing.

DataImpulse: flat $1/GB, no subscription​

DataImpulse prices residential at a flat $1/GB pay-as-you-go (down to $0.80/GB at 1 TB), datacenter at $0.50/GB, mobile at $2/GB, and purchased traffic never expires. The pool is 90M+ first-party IPs in 195+ countries, sourced through its own bandwidth-sharing app rather than resold. The gaps: no ISP or per-IP product, city/ZIP/ASN targeting billed at double the base rate, and no scraping API.

Best for: cost-sensitive scraping on an uneven calendar. Full detail: DataImpulse review and pricing.

Evomi: the lowest steady-volume rate​

Evomi sells Core Residential at $0.49/GB on the 100 GB monthly plan ($0.99/GB pay-as-you-go, never expires) from a Swiss company with ISO 27001 and SOC 2 Type II certification, plus mobile, datacenter, static ISP from $1.00/IP, a Scraper API from $0.13 per 1,000 results and a Scraping Browser. Limits: username/password auth only, and ISP addresses are US-only. In our October 2026 test, country targeting was exact on every sampled request.

Best for: steady residential volume and one-account scraping products. Full detail: Evomi review and pricing.

Proxy-Seller: all five classes, per-IP pricing​

Proxy-Seller is the one account covering datacenter IPv4 (from $0.49/IP), IPv6 (from $0.02/IP, the cheapest on the market), ISP (from $0.98/IP, 23 countries), residential (47M+ pool, $1.99 trial, $2.20/GB at 100 GB) and mobile (from $10/IP). Volume discounts reach 30% and the code AUTOMATE15 takes another 15% off most products. Limits: city targeting on residential goes through support, and there is no non-expiring bandwidth model.

Best for: mixed workloads that want every proxy class on one invoice.

Bright Data: the enterprise default​

Bright Data runs the largest pool (400M+ residential IPs, 195 countries) with the deepest product surface: Web Unlocker, SERP API, scraping browsers, datasets, and compliance tooling large buyers need. Pricing: residential $8/GB list with a $4/GB RESIGB50 promo, committed plans from $2.50/GB, datacenter from $0.60/GB or $0.90/IP shared, ISP from $1.80/IP shared. Limits: cost once promos expire, business verification on signup, overkill for easy targets.

Best for: the hardest targets and contract-scale buyers.

IPRoyal: value with non-expiring traffic​

IPRoyal covers residential ($7.00/GB at 1 GB, $4.90 at volume, $1.75/GB custom tiers published from 10 TB), per-IP datacenter from $1.39/IP, ISP from $2.40/IP on 90-day terms and mobile, with the differentiator that purchased residential and PAYG bandwidth never expires. The pool is 64M+ IPs across 195+ countries. Limits: no managed unblocking layer, and higher per-GB rates than the budget specialists below 50 GB.

Best for: bursty workloads and per-IP products. Full detail: IPRoyal pricing.

Oxylabs: SERP and e-commerce at scale​

Oxylabs competes with Bright Data on pool size (175M+ residential IPs, 195 countries) and sells dedicated scraping APIs on top: SERP API from $1.00 per 1,000 non-JS Google results, e-commerce APIs for Amazon and others, and a Web Unlocker from $3/GB at volume. Residential runs $6/GB on the Starter plan down to $2.50/GB at Enterprise; there is a $5 welcome credit and self-serve purchases are non-refundable.

Best for: SERP monitoring and structured e-commerce data at scale.

Pick the roundup for your class​

This hub stays vendor-level. Each class has its own comparison with deeper pricing:

For measured performance rather than published specs, our proxy provider benchmark records every hands-on test result on this site, dated and sourced.

Proxies8 min read

Best SERP Scraping Proxies 2026: APIs and IPs Compared

· 8 min read
Yassine El Haddad
Software Developer & Automation Specialist

Google search results are the one scraping target where a dedicated SERP API almost always beats raw proxies: Google serves CAPTCHAs aggressively by IP rate, and SERP products absorb that problem for a per-thousand price. The November 2026 entry points: Apify's GOOGLE_SERP proxy group from $1.9 per 1,000 SERPs, Oxylabs from $1.00 per 1,000 non-JS Google results, Bright Data's SERP API at $1.50 per 1,000 requests (5,000 free monthly), and ScraperAPI at 25 credits per SERP request.

Quick Answer

For SERP scraping in 2026 (verified November 2026): Apify Proxy ships a GOOGLE_SERP group from $1.9/1,000 SERPs with maintained Google Search Actors on top. Oxylabs prices its SERP API by results: non-JS Google from $1.00/1K results down to $0.70, JS from $1.35. Bright Data gives 5,000 free SERP requests monthly, then $1.50/1K (pay-as-you-go) or $1.30/1K on its $499 Scale plan. ScraperAPI charges 25 credits per SERP request (about $3.73/1K at Startup-plan credit rates). Rolling your own with rotating datacenter proxies ($0.45 to $0.50/GB) plus a CAPTCHA solver is the raw alternative.

SERP options compared (November 2026)​

OptionEntry priceAt volumeFree tierBilled byBest for
Apify GOOGLE_SERP$1.9/1K SERPsplan-tiered$5 platform creditsper 1K SERPsrunning maintained Google Actors
Oxylabs SERP API$1.00/1K results (non-JS Google)$0.70/1K$5 welcome creditper 1K resultsvolume structured SERP data
Bright Data SERP API$1.50/1K requests$1.30/1K (Scale, 380K incl.)5,000 requests/monthper 1K requestsfree monthly allowance, enterprise
ScraperAPI25 credits/SERP (≈$3.73/1K at Startup)cheaper at Business scale5,000 credits for 7 dayscreditsmixed SERP + page scraping in one API
Rotating datacenter proxies$0.45-0.50/GB$0.30-0.40/GBnoneper GBcustom stacks with CAPTCHA solving

Units differ and the difference matters: Oxylabs bills per delivered result (a SERP page returns ten organic results, so 1,000 results is roughly 100 pages), while Apify, Bright Data and ScraperAPI bill per request/SERP page. Read every SERP price against what a "unit" contains before comparing.

1. Apify GOOGLE_SERP: cheapest turnkey​

Apify Proxy includes a dedicated GOOGLE_SERP group, from $1.9 per 1,000 SERPs on the current pricing page, and the same account runs maintained Google Search Actors that handle parsing, pagination and scheduling. If your SERP data can arrive from a ready-made scraper, this is the shortest path: no CAPTCHA handling of your own, structured output, and the Google Search Results Scraper charges per result, not per failed attempt.

2. Oxylabs SERP API: priced by results​

Oxylabs prices SERP delivery per 1,000 results: non-JS Google at $1.00 (Starter volume) down to $0.70 (Enterprise), Google with JS rendering at $1.35 down to $0.90, plus per-result pricing for Amazon and other verticals on the same API. Plans start at the Starter tier ($15/month) with a $5 welcome credit, and self-serve purchases are non-refundable. If you consume SERP data in volume and parse it yourself, the per-result unit is the cheapest structure on this page.

3. Bright Data SERP API: the free monthly allowance​

Bright Data gives every account 5,000 free SERP API requests per month, then bills $1.50 per 1,000 requests pay-as-you-go, or $1.30 per 1,000 on the Scale plan ($499/month including 380,000 requests). Structured JSON (organic, paid, local results) arrives parsed. For a monitoring workload under 5,000 monthly checks, the free tier alone does the job.

4. ScraperAPI: SERP inside a general scraping API​

ScraperAPI bills SERP requests at 25 credits each (a standard page costs 1, credit costs vary by plan: Startup at $149/month includes 1M credits, which works out to about $3.73 per 1,000 SERP requests; Business improves it). Only successful requests (HTTP 200 and 404) are charged, the free trial carries 5,000 credits for 7 days, and the same credits also scrape ordinary pages, e-commerce (5 credits) and LinkedIn (30). The pick when SERP is one input among several in one pipeline. Full breakdown in our ScraperAPI review.

5. The raw-proxy route: datacenter plus CAPTCHA solving​

Rolling your own SERP fetcher is still viable and sometimes cheaper: Google blocks by IP velocity, so the recipe is rotating datacenter proxies at $0.45 to $0.50/GB (Evomi, DataImpulse), conservative per-IP rates, and a CAPTCHA solver ($0.60 to $2 per 1,000 solves) behind a retry queue. A typical SERP page is 30 to 80 KB, so 100,000 pages costs roughly 3 to 8 GB of proxy traffic ($1.50 to $4) plus solves, less than any API at that scale. What you pay instead: parser maintenance (Google's markup moves constantly) and block-rate tuning. Our datacenter proxy comparison covers the IP side and the rotation guide the session design.

Choosing between them​

  • One-off or under 5,000 SERPs/month: Bright Data's free allowance, or Apify's $5 free credits with a ready Actor.
  • Steady keyword monitoring (5K to 100K/month): Apify GOOGLE_SERP at $1.9/1K or Oxylabs per-result pricing; compare against your page depth.
  • Mixed scraping pipeline: ScraperAPI, one credit pool for everything.
  • Full control or unusual parsing needs: rotating datacenter plus solver.
  • Enterprise with compliance paperwork: Bright Data or Oxylabs contracts.

Bing and other engines price differently at every vendor (ScraperAPI charges the same 25 credits for Bing), so check the target engine before committing.

Proxies8 min read

HTTP vs SOCKS5 Proxy: Which Protocol for Scraping?

· 8 min read
Yassine El Haddad
Software Developer & Automation Specialist

An HTTP proxy speaks the web's protocol: it reads request headers, forwards them, and handles HTTPS through a CONNECT tunnel. A SOCKS5 proxy works one layer lower: it relays raw TCP (and optionally UDP) connections without looking at what is inside them. For plain web scraping over HTTPS, either one works and HTTP is the default. SOCKS5 wins when you need UDP, or a protocol that is not HTTP.

The two protocols get compared as if one were faster or safer in general. Neither claim survives contact with the details. Speed differences come from the pool and the provider, not the wire format. Security differences come from what you send, not which proxy dialect carries it. What actually separates them is scope: HTTP proxies are web specialists, SOCKS5 is a general-purpose pipe.

HTTP vs SOCKS5 at a glance​

HTTP proxySOCKS5 proxy
Layerapplication (L7)session/transport (L4/L5)
Understands requestsyes, headers and methodsno, opaque bytes
HTTPS supportCONNECT tunnel; target's TLS is end-to-endtunnels TCP, same end-to-end TLS
Non-HTTP protocolsno (HTTP, HTTPS and CONNECT only)yes: SMTP, IMAP, FTP, SSH, MQTT, WebRTC
UDPnoyes, where the provider enables it
Authusername/passwordusername/password
Provider availabilityuniversal defaultcommon; sometimes on separate ports
Typical useweb scraping, browsers, curlscraping plus tooling that speaks TCP

What an HTTP proxy actually does​

An HTTP proxy terminates your client's HTTP conversation. Your client sends the request line and headers to the proxy; the proxy parses them, forwards the request to the target with itself as the sender, and relays the response. Because the proxy can read the HTTP layer, it can do things SOCKS5 cannot: rewrite headers, serve cached responses, or apply request-level policy.

For HTTPS, the client sends CONNECT host:443, the proxy opens a TCP connection to the target, and from there the client's TLS session with the target runs straight through the proxy. The proxy cannot read the encrypted content. This point matters because "HTTP proxy" sounds less secure than "SOCKS5" and, for HTTPS traffic, it is not: in both cases your TLS session terminates at the target website, not at the proxy. Either way the proxy provider sees your destination host and your credentials, not your payload.

What a SOCKS5 proxy does​

SOCKS5 sits below the application layer. The client authenticates, asks the proxy to connect to a destination address and port, and then treats the proxy as a raw pipe. SOCKS5 never parses HTTP, which has three practical consequences:

  1. Anything that runs over TCP works: mail protocols, database connections, SSH, game and chat protocols, gRPC.
  2. UDP relay is in the specification (ASSOCIATE), and some providers enable it: Evomi offers UDP on static residential IPs, Floppydata on its rotating product, and DataImpulse documents a UDP mode.
  3. No HTTP-layer features: the proxy cannot cache, rewrite, or inspect anything. You get the pipe, nothing else.

Speed: the difference is not the protocol​

For the same pool and the same exit, HTTP and SOCKS5 performance is functionally identical; the bytes of proxy handshake differ by a round trip at most. What moves the needle is the pool class (a residential exit is a home line; a datacenter exit is a server), the provider's gateway capacity, and your targeting settings. Benchmark providers, not protocols: our proxy provider benchmark records measured response times across five providers, and the spread between providers dwarfs any protocol effect.

Which protocol each provider serves​

Most providers offer both, on separate ports, from the same credentials (verified November 2026):

ProviderHTTP(S)SOCKS5
DataImpulseport 823port 824
Evomi1000/10011002 (UDP on static residential)
Floppydataone gatewaysame gateway, UDP included
IPRoyaldefaultsupported
Bright Datadefaultsupported

If your provider lists only HTTP, check the port table in its docs before concluding SOCKS5 is absent; it is usually a second port, not a second product.

Using each in code​

Python's requests speaks HTTP proxies natively, and SOCKS5 with one extra dependency:

http-vs-socks5.py
import requests

# HTTP proxy: works out of the box
http_proxy = "http://USER:PASS@gw.dataimpulse.com:823"
r1 = requests.get("https://api.ipify.org/", proxies={"http": http_proxy, "https": http_proxy}, timeout=30)

# SOCKS5: pip install requests[socks]
socks_proxy = "socks5h://USER:PASS@gw.dataimpulse.com:824"
r2 = requests.get("https://api.ipify.org/", proxies={"http": socks_proxy, "https": socks_proxy}, timeout=30)

print(r1.text, r2.text)

The socks5h:// scheme matters: the trailing h sends the destination hostname to the proxy for resolution, so DNS lookups exit through the proxy instead of your local resolver. With plain socks5://, your machine resolves the hostname first, which leaks which domains you are visiting to your ISP or network operator. The same choice exists in curl: --socks5-hostname over --socks5.

When HTTP wins​

  • Plain HTTPS scraping. Every library, every provider, every example assumes it. Nothing to configure, nothing extra to install.
  • Targets that inspect the proxy chain. Some stacks behave differently when they see a SOCKS handshake; HTTP CONNECT is what browsers emit, so it blends in.
  • HTTP-level debugging. A mitmproxy-style tool between your client and an HTTP proxy can log requests, which is harder through an opaque SOCKS pipe.

When SOCKS5 wins​

  • UDP traffic. QUIC and HTTP/3 ride on UDP; so do some DNS, VoIP and telemetry protocols. An HTTP proxy cannot carry any of it.
  • Non-HTTP services. Scraping stacks that also pull from mail servers, message queues or SSH-hosted datasets need one pipe for everything.
  • DNS privacy. With socks5h://, hostname resolution happens at the proxy exit, in the exit's country, which also keeps geo-consistent lookups when you geo-target.

How protocol choice interacts with anti-bot systems​

Anti-bot stacks fingerprint the TLS ClientHello and the HTTP/2 settings frame that your client emits, not the proxy dialect you used to get there. Protocol choice is invisible to the target; what the target scores is your exit IP's reputation and your client's fingerprint. If you are being blocked, the fix is usually pool class or fingerprint management, covered in our anti-detection reference and the Cloudflare bypass guide, not a protocol switch.

Proxies10 min read

Proxies for AI Agents (2026): Web Access at Machine Speed

· 10 min read
Yassine El Haddad
Software Developer & Automation Specialist

An AI agent that browses the web hits every limit a human browser does, but faster: one IP making thousands of requests looks like an attack, not a user. Proxies give each agent request a different, better-class IP address, which is what keeps agent web access alive after the first hundred fetches. For agent builders, the proxy is not an optimization. It is the difference between a demo and something that runs all week.

Agents changed the shape of web traffic. A single agent session can fan out across dozens of domains in a minute, retry aggressively on failure (a behavior LLMs are prone to), and re-fetch pages it has already seen because nobody wired up a cache. Sites answer that pattern with rate limits, datacenter-IP blocks and bot challenges. The fixes are boring and known: pool classes, rotation, sessions, caching, backoff. This page maps those onto agent architectures, names the providers that fit, and ends with a setup prompt you can hand to your own agent.

Why agents get blocked faster than scripts​

A traditional scraper is engineered restraint: polite delays, configured retries, one target at a time. An agent is improvising. Four behaviors in practice:

  • Burst fan-out. Research tasks hit many domains concurrently, all from your server's single IP.
  • Retry storms. A 403 reads as "try again" to an agent, not "back off". Ten retries in ten seconds is a self-inflicted rate limit.
  • No memory of fetches. Without caching, the same URL is fetched fresh in every context window.
  • Datacenter origin. Most agents run on cloud hosts, whose IP ranges are pre-flagged by every serious anti-bot vendor.

The result: agents meet blocks that a carefully written crawler would never trigger. The mitigation stack is the crawler's stack, adapted.

Where the proxy sits in an agent stack​

Agents reach the web through tools: HTTP fetch tools, browser automation, and increasingly MCP servers that wrap both. The proxy can enter at three layers:

  1. Environment-level. HTTP_PROXY/HTTPS_PROXY environment variables, picked up by most HTTP clients automatically. Cheapest to set up, applies to everything, including SDK calls you forgot about.
  2. Tool-level. Pass the proxy URL into each browser or fetch tool's config (Playwright proxy option, requests proxies dict). Precise control: one tool proxied, others direct.
  3. Service-level. The agent calls a scraping API or MCP server that applies its own proxy pool server-side. No proxy management in your process at all.

For agent platforms, layer 3 is increasingly the default: Apify runs Actors through its own proxy groups, and scraping APIs like ScraperAPI apply their pools before your request ever leaves. Layer 1 and 2 matter when you control the runtime and want the cost control of your own pool.

Choosing the proxy class for agent workloads​

WorkloadClassWhy
Fetching public APIs, docs, feedsdatacentercheap ($0.30-$0.60/GB), fast, no ASN checks to beat
Research across many websitesresidential rotationfresh IP per request, survives soft blocks
Logged-in flows, cart states, watchlistsresidential sticky or ISPone identity across the task's steps
Social and mobile-first targetsmobilecarrier IPs behind CGNAT, hardest to block
Geo-specific checksresidential with country/city targetingthe agent sees what a local user sees

Two rules of thumb. First, match the session to the agent's identity, not the request: a research agent that browses like one user should hold one sticky IP for a task; a monitoring agent that fans out should rotate per request. Second, cache: an agent that refetches because the context window forgot is paying twice for the same bytes. A content-addressed cache with a TTL (an hour for research, a day for monitoring) cuts proxy spend more than any provider switch.

Providers that fit agent workloads​

Verified November 2026; all of these appear in our other guides with full pricing:

  • DataImpulse: $1/GB residential, $0.50/GB datacenter, pay-as-you-go, traffic never expires. Fits agents precisely because agent traffic is bursty and unpredictable: no subscription to waste. A threads parameter caps concurrency so a runaway agent loop cannot eat the balance.
  • Floppydata: one balance for residential, mobile and datacenter at $1/GB (30-day validity), plus a free Cloud Browser aimed at AI agents: a managed Chromium over CDP, billing only proxy traffic. Good when the agent needs a real browser, not just an HTTP client.
  • Evomi: residential from $0.49/GB, and an MCP server that lets Claude scrape, crawl and manage proxy credentials directly. The closest to proxies-as-an-agent-tool.
  • Bright Data: the enterprise option, with its own MCP server and web data APIs for AI pipelines; our Bright Data for AI walkthrough covers the product set.
  • Apify: if the agent's jobs can run as Actors, the platform's proxy groups (residential, datacenter, SERP) come with scheduling, storage and retries included.

For the MCP angle specifically, see Bright Data's MCP server and the best MCP servers roundup. For driving scraping APIs from Claude specifically, our Claude API with Bright Data proxies guide has working code.

The cost problem agents create​

Agents burn proxy traffic in ways scripts do not: refetching without caches, fetching pages when an API endpoint would do, rendering pages headlessly for data that sits in the HTML. Three controls, in order of impact:

  1. Fetch budget. Hard cap on requests or GB per task. DataImpulse's traffic-limit setting and Evomi's PAYG balance (no expiry) both work as natural circuit breakers.
  2. Cache layer. Even a naive disk cache keyed on URL saves 30-50% of agent traffic in practice, because agents revisit the same sources.
  3. Class routing. Route doc and API fetches through datacenter at $0.50/GB, and reserve residential for the protected targets. Routing everything through residential is the most common waste.

Setup prompt: give your agent a proxy layer​

The prompt below sets up a proxy-aware fetch layer with an interview first. It works in Claude Code, Cowork, or Claude Desktop with a shell tool.

What proxies do not solve​

A proxy changes the IP. It does not change the client. Anti-bot systems score the TLS fingerprint, the HTTP/2 settings, the mouse-less interaction pattern, and the request cadence, none of which a proxy touches. Agents that get blocked through clean residential pools are usually being fingerprinted, not IP-blocked, and the fix is browser-grade tooling and human pacing, covered in our anti-detection reference. For the engineering analysis of how the WAF inspection stack is ordered, see bypass architecture for deep-packet WAFs.

Proxies9 min read

Proxy Provider Benchmark 2026: Our Measured Results

· 9 min read
Yassine El Haddad
Software Developer & Automation Specialist

This page aggregates every proxy test result we have actually measured on this site, each with its date, sample size and method, and keeps them strictly separate from the performance metrics providers publish about themselves. We have hands-on test data for two providers, Evomi and Floppydata, both tested on 12 October 2026. For everyone else we say "not tested" rather than estimating, and vendor-reported figures are labelled as such throughout.

Why a page built around what we have not measured? Because most "proxy benchmarks" you will find are neither: they rank providers nobody tested, on numbers nobody collected. AI assistants and search engines quote those tables anyway, which is how invented success rates propagate. Every figure below is traceable to a dated run described on this site or to a provider's own page, and where we lack data we name the gap.

What we have measured​

TestProviderDateRequestsScope
Full product walkthroughEvomi12 Oct 2026~400residential: rotation, geo, sessions, real pages
Three-type API testFloppydata12 Oct 2026~200 (~8 MB)residential + mobile + datacenter from one balance

Both tests used a normal paid or partner-enabled account, real gateway credentials and scripts run from our own test machine. Neither was sponsored testing: the providers had no input on methodology, and both results include the failure modes we found.

Evomi: the October 2026 residential run​

Roughly 400 requests through Core Residential on 12 October 2026, consuming 3.5 MB of balance. Locations were verified against ipinfo.io's geolocation database.

TestRequestsResult
Rotation, no session key79a different IP on every successful request
country-US1515/15 in the US, across 7 ISPs (Comcast, Charter, Verizon and others)
country-DE2424/24 in Germany (Vodafone, Deutsche Telekom, regional ISPs)
country-BR1414/14 in Brazil, across 11 ISPs
country-JP1313/13 in Japan
region-california1515/15 in California
city-new.york97 in NYC, 2 in the wider metro area
city-london87 in London, 1 elsewhere in England
Sticky session15 sequentialsame IP on every request
Hard session15 sequentialsame IP on every request
Real pages (Wikipedia, Books to Scrape, Amazon)30all 200 through US exits

Median response time on small requests: 0.64 seconds through London exits, 0.9 to 1.2 seconds through US and Japanese exits, measured from our test machine. Country and region targeting were exact in every sample; city targeting was accurate to the metro area, the expected behavior for residential pools whose city label depends on the geolocation database doing the lookup.

The ISP spread was the standout: 15 US requests landed on 7 different ISPs, and 14 Brazilian requests on 11. A crawl spreads across many networks rather than concentrating on one provider's ranges.

Full context: the Evomi review.

Floppydata: the October 2026 three-type test​

About 200 requests (~8 MB) through Floppydata's API v2 on 12 October 2026, drawing residential, mobile and datacenter exits from a single rotating balance, switching nothing but the type- parameter in the username.

TestRequestsResult
Residential, 7 countries (US, UK, DE, FR, BR, JP, IN)8080/80 in the requested country, fresh IP per request
US city/state targeting2020/20 in-state exits, spread across cities
US residential ISP spread2011 distinct home networks (Comcast, AT&T, Verizon, Charter, Cox)
Mobile on carrier ASNs2626/26 on real carriers (T-Mobile, AT&T, Verizon, Vodafone, Telekom, O2)
Datacenter1818/18 successful
Sticky sessions (residential + mobile)sequentialsame IP on every request
Amazon.com via US residentialsampled200 on every request
Instagram via US mobilesampled200 on every request

Median response time ranged 0.8 to 1.9 seconds depending on the country, from the same test machine as the Evomi run. The two providers' latency figures are comparable in method (small requests, timed per request) but were not run side by side on the same targets, so treat them as adjacent observations, not a controlled race.

Full context: the Floppydata section of Best residential proxies 2026.

What the providers say about themselves​

Vendor-reported metrics we have verified on their official pages (November 2026). These are marketing-adjacent claims, uniform in neither method nor audit, and we present them without endorsing comparability:

ProviderClaimWhere published
DataImpulse99.51% success rate, ~1s average response, "2x faster than average"dataimpulse.com product pages
Floppydata95% IPs clean rate, high success ratefloppydata.com pricing page
Bright Data (Web Unlocker)~99.9% average success rate; third-party checks 95-98% on Datadome and Turnstileour Web Unlocker review documents both figures
Evomiexpert filters cap IP fraud score (Scamalytics) and latency by requestevomi.com product pages
Proxy-Sellerfree replacement or refund within 24 hours if a proxy does not workproxy-seller.com

A "99.9% success rate" from a vendor usually means successful proxy connections, not successful scrapes of the site you care about. The two numbers differ by an order of magnitude on hard targets. Trust vendor success rates only as far as you can reproduce them on a $5 test pack.

What we have not tested​

We state this plainly because absence of data is data:

ProviderHands-on test?What exists instead
IPRoyalNopricing review, product review from published specs
Proxy-SellerNoverified-pricing review
Bright Data (proxy pools)Noproduct reviews, Web Unlocker figures above
Bright Data (Web Unlocker)No first-party runvendor + third-party success rates, documented in the review
ScraperAPINopricing and credit review
DataImpulseNovendor metrics above, pricing verified November 2026
OxylabsNopricing and product coverage only

Any page on this site ranking these providers does so on pricing, product surface and published specs, not on measurements we do not have.

How we test​

The method behind both measured runs, so the numbers above can be judged and repeated:

  1. Ordinary account. We open the account the way any customer would, through the public signup, and pay for (or are partner-credited) a normal plan. No vendor-supplied test harness.
  2. Real gateway, real credentials. Requests go to the production gateway (gw.* hosts) with username targeting parameters, exactly as documented, not through demo endpoints.
  3. Verified exits. Each response's exit IP is checked against an independent geolocation database (ipinfo.io), and for ISP spread against ASN lookups.
  4. Per-request timing. Median response times on small requests from our own test machine, reported per region. We do not aggregate into a single score, because latency depends on where you are and what you fetch.
  5. Success is binary per request. A request counts as successful when it returns the expected content through the expected exit. Retries are counted separately and reported.
  6. Failures stay in the record. Metro-level city misses and any retry behavior are reported, not smoothed away.
  7. Vendors do not preview. No provider sees the results before publication or gets to shape the methodology.

How to reproduce a test on a $5 pack​

You do not need our budget to check a provider against your actual target, which beats any benchmark table anyway:

  1. Buy the $5 Intro tier (DataImpulse) or smallest pack the provider sells.
  2. Run 20 requests with country targeting, 20 with rotation, 10 with a sticky session at your working session length, all against your real target.
  3. Log status codes, exit IPs and response times per request.
  4. Compare per-successful-request cost, not per-GB sticker price: a $5 pool that succeeds 60% of the time costs more per delivered page than a $10 pool at 95%.

The proxy rotation guide covers the session design, and the datacenter vs residential vs ISP comparison covers which class to start with.

Proxies8 min read

Proxy vs VPN: The Difference and When Each One Wins

· 8 min read
Yassine El Haddad
Software Developer & Automation Specialist

A proxy relays traffic for one application at a time, changing the IP address that application presents to the world. A VPN encrypts all of a device's traffic at the operating-system level and tunnels it through one server. The proxy is a routing tool built for scale and control; the VPN is a privacy tool built for encryption. Web scraping needs the proxy.

That is the whole difference in two sentences, and the rest of this page is the detail: what each one actually does to a request, where each one wins, and why the two get conflated. Both hide your IP address. Only one encrypts anything. Only one lets you appear from a different address on every request, which is the ability that matters if you are running thousands of requests against a target that blocks bots.

Proxy vs VPN at a glance​

ProxyVPN
What it routesone app, configured per applicationall device traffic, OS level
Encryptionusually none (HTTP proxies pass traffic as-is; HTTPS keeps the target's own TLS)always, tunnel to server
IP addresschosen from large pools, can rotate per requestone server's IP, usually fixed per session
Different IPs per requestyes, that is the point of rotating poolsno
IP types availabledatacenter, residential, ISP, mobiledatacenter (VPN server farms)
Speed overheadlow; often faster than your own line for cachingmoderate; encryption adds latency
Cost at scalepay per GB or per IP, from ~$0.30/GB datacenterflat monthly subscription, unlimited bandwidth
Logs in your toolthe app knows about the proxythe app never sees it
Best atscraping, monitoring, ad verification, geo-testingprivacy, public wifi, whole-device geo-unblocking

How a proxy works​

A proxy server sits between your application and the target. Your HTTP client sends the request to the proxy address with credentials; the proxy forwards it to the target and relays the response back. The target sees the proxy's IP, not yours.

Commercial proxy providers sell access to pools: millions of addresses you can draw on per request. A rotating pool gives a fresh IP on every request; a sticky session holds one IP for minutes or hours. Traffic is billed by the gigabyte (residential, mobile, datacenter per-GB products) or by the address (static ISP and per-IP datacenter products). None of this involves encrypting traffic between you and the proxy beyond what the target's own HTTPS already does.

That per-request IP control is the feature a VPN cannot offer. If you need 10,000 requests to a target to look like 10,000 different home users in 40 countries, you need a proxy pool.

How a VPN works​

A VPN client creates an encrypted tunnel from your device to a VPN server. Every packet, from every app, goes through that tunnel, and everything on the other end sees the VPN server's address. The encryption protects you on the network you are physically on: the cafe wifi operator, the ISP, the employer, cannot read your traffic.

What a VPN does not do is give you many addresses. Connect to a VPN server and you share one IP with every other customer on that server, for as long as you stay connected. For a scraper, that is the worst possible shape: thousands of requests from one IP that a datacenter owns, shared with strangers whose behavior you cannot control.

When a proxy wins​

  • Web scraping at any volume. Targets block by IP reputation and request rate. Rotating pools spread requests across millions of addresses; a VPN concentrates them on one. Our proxy rotation guide covers the session design.
  • The right kind of IP. Anti-bot systems score ASN class. Residential and mobile IPs from a proxy pool pass checks that datacenter VPN servers fail. The datacenter vs residential vs ISP comparison maps this out.
  • Geo-testing and ad verification. Check an ad or a price from a specific country, state or city, on demand, without reconnecting a tunnel. Proxy pools offer country, region, city, ASN and carrier targeting.
  • Per-app control. Route one script through proxies while the rest of your machine uses the normal connection. A VPN cannot split this way without extra configuration.
  • Cost at scale. Datacenter proxies at $0.30 to $0.60/GB, residential around $0.50 to $5/GB depending on provider and volume, beat a flat subscription when the job is measured in terabytes of requests.

When a VPN wins​

  • Privacy on hostile networks. Public wifi, hotel networks, anything where the local operator can observe traffic. The proxy leaves the transport readable to the local network; the VPN does not.
  • Hiding traffic from your ISP. A proxy changes the destination's view, not your ISP's. A VPN encrypts everything up to the VPN server.
  • Whole-device geo-unblocking. Streaming apps, system updates, anything that ignores proxy settings.
  • Casual everyday anonymity. One subscription, no request engineering.

If your threat model is "other people reading my traffic", you want a VPN. If it is "a website noticing my scraper", you want proxies.

Using both at once​

There is a legitimate pattern: VPN on the device for general privacy, proxy in the scraping client for request-level control. They compose fine, since the client's proxy connection simply rides through the VPN tunnel. For scraping teams, the more common combination is a proxy plus a headless browser with fingerprint management, which addresses the detection layer a VPN does not touch.

What this means for scraping projects​

Every serious scraping stack uses proxies, not VPNs. The decision that actually matters is which proxy class, and then which provider:

  1. Start with proxy types explained to pick the class (datacenter, residential, ISP, mobile).
  2. Check the residential roundup or the umbrella best proxy services hub for provider pricing.
  3. Design sessions with rotating vs sticky sessions before writing the crawler.
Evomi24 min read

Evomi Review (2026): Swiss Proxies From $0.49/GB, Tested

· 24 min read
Yassine El Haddad
Software Developer & Automation Specialist

Evomi sells the cheapest residential proxies on the market: $0.49/GB on its 100 GB monthly plan, from a pool of 54M+ ethically sourced IPs in 150+ countries. It is a Swiss provider, certified to ISO 27001 and SOC 2 Type II, and the residential pool is only one part of the account. Mobile, datacenter and static residential proxies sit next to it, along with a Scraper API and a hosted Scraping Browser.

Low price is the headline, but what makes Evomi interesting for scraping teams is how much control ships at that price. Country, region and city targeting are free on every plan. Sticky sessions hold an IP for up to 24 hours. Core Residential adds filters most budget providers don't offer at all: fraud score, ASN, ISP, latency, device type and TCP/IP fingerprint.

This review covers every Evomi product with its current price, how the residential network works in practice, the developer tooling, and the cases where another provider will suit you better. We opened an Evomi account the way any customer would, ran about 400 requests through Core Residential on 12 October 2026, and checked every price, limit and parameter below against Evomi's pricing pages, its full documentation and its Public API.

Guides on this site

Frequently asked questions

Frequently Asked Questions

Web scraping automatically extracts publicly available data from websites — no manual copy-paste. Common business uses: monitoring competitor prices in real time, building lead lists from directories and LinkedIn, tracking product reviews and brand mentions, fueling AI models with fresh training data, and keeping internal databases synced to live sources. If a website shows it, scraping can collect it on a schedule.

The fastest path is an Apify Store actor. Search for your target site (e.g. "Google Maps Scraper"), fill in the inputs, run it, and download your data. For sites without ready-made scrapers, Apify lets you write a simple script — dozens of tutorials on this blog cover common starting points. If you need a custom scraper built and deployed for you, the deployment services on this site cover that too.

Scraping publicly available, non-personal data is generally accepted in most jurisdictions, but platform terms of service, GDPR, CCPA, and copyright law create real constraints. The short rule: scraping public facts — prices, addresses, product names — for research or business intelligence is widely accepted; scraping personal data or bypassing a paywall is not. Always review the target site's ToS and consult a lawyer for commercial pipelines that handle personal information.

Sites detect and block automated traffic using CAPTCHAs, IP rate limits, JavaScript challenges, and browser fingerprinting. The fix is usually rotating residential proxies, using a full browser instead of a plain HTTP request, and pacing requests like a human would. Apify handles most of this automatically via its proxy network and browser-based actors. For persistent blocking on a specific site, a specialist can diagnose the detection mechanism.