use-apify.com
Anti Bot: guides & tutorials
How sites fingerprint bots—TLS, headers, behavior, rate limits—and what scrapers do next: residential IPs, real browsers, backoff, and Apify Actor design.
3 articles
View all tags
Anti-bot systems fingerprint automated traffic through TLS signatures, headers, behavior, and rate limits, then block or challenge anything that looks non-human. These guides explain how detection works and what scrapers do in response.
The durable answer is residential IPs, real browsers, consistent fingerprints, and request pacing that mimics people, plus backoff when challenges appear. Apify actor design folds these tactics in. Below you will find breakdowns of detection methods and practical anti-blocking techniques.

Modern data extraction pipelines frequently collide with sophisticated Web Application Firewalls (WAFs) like Cloudflare, Datadome, Akamai, and PerimeterX. Good residential IPs alone are often not enough; these systems validate TLS handshakes, inspect JavaScript environments, and use signals like Canvas/WebGL to separate real browsers from automation.
The Bright Data Web Unlocker wraps that work behind a proxy-style endpoint. Instead of maintaining stealth Puppeteer patches yourself, you send normal HTTP requests to Bright Data, which handles CAPTCHAs, browser-like fingerprints, and returns the unblocked HTML when it succeeds.

A 403 from a big consumer site is rarely “you hit the rate limit.” Products like Cloudflare Turnstile, DataDome, and Akamai Bot Manager stack checks: IP reputation, TLS fingerprint, headers, JavaScript probes, then behavior. Fail early in that stack and you may never reach the HTML you wanted.
This post is the mechanics analysis: each defense layer in the order a request meets it, what actually fails, and what people do to pass it, without pretending any of it stays stable forever. For the hands-on reference (Cloudflare bypass recipes, tool comparisons, testing your setup), see web scraping anti-detection; for the worked Cloudflare walkthrough, how to bypass Cloudflare.