Complete Guide to Web Scraping APIs: REST, GraphQL, and Headless Browsers (2026)

Contains affiliate links. Disclosure.

Web scraping APIs fall into three categories: public APIs (official, stable, no scraping needed), unofficial or reverse-engineered APIs (hidden endpoints used by the frontend), and scraping APIs (services like Apify, Bright Data, Firecrawl that handle scraping for you). This guide covers finding hidden APIs, REST and GraphQL scraping patterns, authentication, rate limiting, pagination—and when to use a scraping API service instead of building your own. Try Apify API · Firecrawl API

Three Categories of APIs for Data Extraction

Category	Description	When to Use
Public APIs	Official, documented, stable. No scraping required.	First choice when available (Twitter API, GitHub API, etc.)
Unofficial / reverse‑engineered	Hidden endpoints used by the site's frontend.	When no public API exists; requires inspection
Scraping APIs	Managed services (Apify, Firecrawl, Bright Data) that fetch and parse pages.	When rendering, anti-bot, or scale matter

Always prefer public APIs when they provide the data you need. Use unofficial APIs when the frontend calls them and you can replicate the requests. Use scraping APIs when you need full HTML rendering, anti-bot bypass, or turnkey extraction.

Finding Hidden APIs

Most modern sites load data via XHR or fetch. The frontend calls internal APIs; you can too.

Chrome DevTools Network Tab

1. Open DevTools (F12) → Network.
2. Filter by XHR or Fetch.
3. Reload or interact with the page. Inspect requests: URL, method, headers, body.
4. Copy as cURL or use the request in your script. Look for authentication headers (Authorization, cookies, API keys).

Fiddler, mitmproxy, Charles

For mobile apps or complex flows, use a proxy to capture traffic:

• Fiddler (Windows): HTTP(S) proxy with decryption.
• mitmproxy (cross-platform): Scriptable, ideal for automation.
• Charles (macOS, Windows): Similar capture and replay.

Decrypt HTTPS by installing the proxy's CA certificate. Route mobile app traffic through the proxy to see API calls.

REST API Scraping

Authentication Patterns

Pattern	Example	Implementation
API key	Authorization: Bearer sk-xxx or ?api_key=xxx	Store in env; add to every request
OAuth 2.0	Token exchange, refresh	Use requests-oauthlib or authlib; handle refresh
Session cookies	Cookie from browser login	Extract from DevTools; reuse until expiry

Cookie extraction: Log in manually in Chrome → Application → Cookies → copy needed values. Use in a Cookie header or requests.Session().

OAuth PKCE flow: For public clients (no secret), use PKCE: generate code_verifier, code_challenge, exchange authorization_code for access_token. Refresh before expiry.

Pagination Patterns

Style	How it works	Example
Offset	?page=2&limit=50	Sequential pages
Cursor	?cursor=abc123	Opaque token for next page
Link header	Link: <...?page=2>; rel="next"	Parse header for next URL

# Python: offset-based pagination
import requests

def fetch_all_pages(base_url, auth_header):
    results = []
    page = 1
    while True:
        r = requests.get(base_url, params={"page": page, "limit": 50},
                         headers={"Authorization": auth_header})
        data = r.json()
        if not data.get("items"):
            break
        results.extend(data["items"])
        if not data.get("has_more"):
            break
        page += 1
    return results

Rate Limit Handling

• Exponential backoff: Double wait after each 429: sleep = min(2 ** retry_count, 60)
• Token bucket: Allow N requests per second; refill over time.
• Per-endpoint limits: Some APIs have different limits per route; track per key.

Always respect Retry-After when the API returns it. Use Apify Proxy or Bright Data to distribute load when limits are IP-based.

GraphQL Scraping

GraphQL uses a single endpoint. You send queries that specify exactly which fields you want.

Introspection

Many GraphQL APIs support introspection. Query __schema to discover types and fields:

query Introspection {
  __schema {
    types { name }
    queryType { name }
  }
}

Use Altair or Insomnia to explore. Disable introspection in production is common; if so, reverse-engineer from network traffic.

Batching Queries

GraphQL supports batching: send multiple operations in one HTTP request. Reduces round-trips.

# Example: batched GraphQL with requests
import requests

queries = [
    {"query": "query { product(id: 1) { name price } }"},
    {"query": "query { product(id: 2) { name price } }"},
]
response = requests.post(
    "https://api.example.com/graphql",
    json=queries,
    headers={"Authorization": f"Bearer {token}"}
)

Not all servers support batching; check docs or test.

When to Use a Scraping API Service

Use Apify, Firecrawl, or Bright Data when:

• Rendering required: JavaScript-heavy SPAs; no usable API.
• Anti-bot: Site blocks direct HTTP; you need browser automation and proxies.
• Speed to production: Pre-built Actors (Apify) or endpoints (Firecrawl) get you live faster.
• Scale: Managed infrastructure handles concurrency, retries, storage.

Approach	Use Case	Pros	Cons
Direct API	Public or discovered endpoints	Fast, no parsing, low cost	Not always available
Scraping API service	Pages needing render or anti-bot	Managed, scalable	Per-page cost
Headless browser	Complex interaction, SPAs	Full control	Higher resource use

Code Examples

REST: Paginated API with Backoff

import requests
import time

def fetch_with_backoff(url, auth_header, max_pages=100):
    results = []
    next_url = url
    retries = 0
    while next_url and len(results) < max_pages * 50:
        try:
            r = requests.get(next_url, headers={"Authorization": auth_header})
            if r.status_code == 429:
                retries += 1
                wait = min(2 ** retries, 60)
                time.sleep(wait)
                continue
            r.raise_for_status()
            data = r.json()
            results.extend(data.get("data", []))
            next_url = data.get("next_page_url")
            retries = 0
        except requests.RequestException as e:
            retries += 1
            time.sleep(2 ** retries)
    return results

GraphQL: Batched Extraction

def batch_graphql(api_url, token, ids):
    query = """
    query GetProducts($ids: [ID!]!) {
      products(ids: $ids) {
        id name price availability
      }
    }
    """
    results = []
    batch_size = 10
    for i in range(0, len(ids), batch_size):
        batch = ids[i:i + batch_size]
        r = requests.post(api_url, json={"query": query, "variables": {"ids": batch}},
                          headers={"Authorization": f"Bearer {token}"})
        data = r.json()
        results.extend(data.get("data", {}).get("products", []))
    return results

Error Handling and Resilience

API scraping introduces failure modes beyond HTTP:

• Transient errors: 5xx, network timeouts — retry with backoff.
• Permanent errors: 404, 401 — log and skip; do not retry indefinitely.
• Rate limit (429): Respect Retry-After; implement exponential backoff.
• Auth expiry: OAuth tokens expire; implement refresh before expiry. Cookie-based auth may need periodic re-login.

Use idempotent request logic: if a request fails after partial processing, you should be able to retry safely. For scraped data, deduplicate by a stable key (URL, ID) before storing.

Storing and Exporting API Data

Once you have data from APIs or scraping services:

• Apify datasets: Push to Actor dataset; export to JSON, CSV, or integrations (Sheets, webhooks).
• Firecrawl: Returns data in the response; persist to your DB or S3.
• Bright Data datasets: Pre-collected; download via API or dashboard.

For large volumes, stream to a database (PostgreSQL, MongoDB) or object storage (S3, MinIO). Use batch inserts and connection pooling. See Apify integrations for built-in exporters.

Direct API vs Scraping API vs Headless

Factor	Direct API	Scraping API (Apify, Firecrawl)	Headless (Playwright)
Setup	Minimal	API key, input config	Infra, browser, proxies
Cost	Often free or low	Per page/credit	Compute + proxies
Latency	Low	Medium	Higher
Anti-bot	N/A	Handled by service	You configure proxies
Maintenance	API changes	Service handles updates	You maintain selectors
Best for	Structured data, known endpoints	Quick deploy, render, scale	Full custom control

Start with Network tab

Before writing scrapers, inspect the site in DevTools. If data comes from an API, use it. If not, evaluate Apify Actors or Firecrawl for managed scraping.

Frequently Asked Questions

REST uses multiple endpoints and URL params. GraphQL uses one endpoint with query payloads. Both can be scraped. GraphQL often requires less guesswork about which endpoints to hit once you know the schema.

Use Chrome DevTools Network tab, filter by XHR/Fetch. Reload or interact with the page. Inspect request URL, headers, body. Copy as cURL or replicate in code. For mobile apps, use mitmproxy or Fiddler.

When you need JavaScript rendering, anti-bot bypass, or fast deployment. Direct API access is better when endpoints exist and are accessible. Scraping APIs handle infra, proxies, and parsing.

Implement exponential backoff on 429. Respect Retry-After. Use distributed proxies (Apify, Bright Data) when limits are per-IP. Consider token bucket or per-endpoint tracking.

Yes. Use requests-oauthlib or authlib. Implement token refresh before expiry. For public clients, use OAuth PKCE. Store tokens securely (env vars, secrets manager).

Offset (page/limit), cursor (opaque token), or Link header. Check the response structure. Cursor-based is common for large datasets; offset can have performance issues at scale.