API Rate Limiting for Self-Hosted Scraping Services

Contains affiliate links. Disclosure.

Without rate limiting, a single misbehaving client can exhaust your server's bandwidth, IP pool, and memory in minutes. Internal scrapers and customer-facing APIs both need throttling — it is the first safeguard you should ship.

This guide covers the three most practical implementations for a self-hosted scraping service: fixed window, sliding window, and token bucket. You get drop-in examples for Express.js and FastAPI, plus a Redis-backed approach that survives horizontal scaling.

Why Rate Limit a Scraping API?

A scraping API is expensive to run. Each request may spin up a headless browser, consume a residential proxy credit, or parse several megabytes of HTML. Without limits, you face:

• Resource exhaustion — one heavy client starves all others.
• IP bans — burst traffic from a single user triggers the target site's bot detection, burning your proxy pool.
• Cost overruns — cloud bills spike without per-client caps.
• Abuse — competitors or scrapers hammer your service to enumerate data.

Rate limiting solves all four by enforcing a ceiling on how many requests a client can make in a given time window.

Rate Limiting Algorithms: Which to Choose

Algorithm	Best for	Trade-off
Fixed window	Simplest setup, low Redis overhead	Allows burst at window boundary
Sliding window	Smooth enforcement, fairness	Slightly more Redis reads
Token bucket	Burst-tolerant APIs with steady refill	More state to track per client
Leaky bucket	Strict queue-based rate shaping	Higher latency, complex to implement

For a scraping service, sliding window or token bucket are the right defaults. Fixed window is fine for internal services where strict fairness isn't critical.

What Is a Token Bucket?

A token bucket starts with a fixed capacity of tokens. Each request consumes one token. Tokens refill at a constant rate (e.g., 10 per second). If the bucket is empty, the request is rejected with a 429 Too Many Requests response.

This allows short bursts (a user can fire 20 requests instantly if the bucket is full) while enforcing a long-run average rate (e.g., 10 req/s). It's the algorithm used by AWS API Gateway and Stripe's API.

Bucket capacity: 20 tokens
Refill rate:     10 tokens/second
Client fires 25 requests in 1 second:
  → First 20 succeed (bucket drains)
  → Requests 21–25 get 429
  → After 500ms, 5 tokens refill → next 5 requests succeed

What Is Sliding Window Rate Limiting?

Instead of resetting the counter at a fixed boundary, a sliding window tracks requests in a rolling time period. If a client made 8 requests 45 seconds ago and the limit is 10 per minute, they can make only 2 more — not 10.

This prevents the "boundary burst" problem of fixed windows, where a client can make 10 requests at 11:59:59 and 10 more at 12:00:01.

Redis-Based Rate Limiting

For any horizontally scaled service (multiple Node or Python workers), you need a shared rate limit store. Redis is the standard choice: it's fast, supports atomic operations via Lua scripts, and has built-in key expiry.

Setting Up Redis on a Liquid Web VPS

Liquid Web managed VPS plans give you root access and SSD storage suitable for running Redis alongside your scraping API. Their fully managed tier handles OS patching and monitoring, which reduces operational overhead for small teams.

# Ubuntu 22.04 on Liquid Web VPS
sudo apt update && sudo apt install redis-server -y
sudo systemctl enable redis-server

# Bind Redis to localhost only (never expose to public)
sudo sed -i 's/^bind .*/bind 127.0.0.1/' /etc/redis/redis.conf
sudo systemctl restart redis-server

# Verify
redis-cli ping  # → PONG

Set a strong requirepass in /etc/redis/redis.conf before going to production.

Implementing Rate Limiting in Express.js

The express-rate-limit package handles in-process limiting. Pair it with rate-limit-redis for a distributed store.

npm install express-rate-limit rate-limit-redis ioredis

Sliding Window with Redis (Express.js)

import rateLimit from 'express-rate-limit';
import RedisStore from 'rate-limit-redis';
import Redis from 'ioredis';

const redis = new Redis({ host: '127.0.0.1', port: 6379 });

// Per-API-key rate limiter: 100 requests per 15 minutes
const scraperLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100,
  keyGenerator: (req) => req.headers['x-api-key'] ?? req.ip,
  store: new RedisStore({
    sendCommand: (...args) => redis.call(...args),
  }),
  handler: (req, res) => {
    res.status(429).json({
      error: 'rate_limit_exceeded',
      message: 'Too many requests. Retry after the window resets.',
      retryAfter: Math.ceil(res.getHeader('Retry-After')),
    });
  },
  standardHeaders: true,   // Return X-RateLimit-* headers
  legacyHeaders: false,
});

app.use('/api/scrape', scraperLimiter);

Token Bucket with Redis Lua Script (Express.js)

For true token bucket behavior, use a Lua script to atomically check and decrement tokens:

const TOKEN_BUCKET_SCRIPT = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refillRate = tonumber(ARGV[2])   -- tokens per second
local now = tonumber(ARGV[3])          -- timestamp in milliseconds

local bucket = redis.call('HMGET', key, 'tokens', 'lastRefill')
local tokens = tonumber(bucket[1]) or capacity
local lastRefill = tonumber(bucket[2]) or now

-- Refill tokens based on elapsed time
local elapsed = (now - lastRefill) / 1000
local newTokens = math.min(capacity, tokens + elapsed * refillRate)

if newTokens < 1 then
  redis.call('HSET', key, 'tokens', newTokens, 'lastRefill', now)
  redis.call('EXPIRE', key, 3600)
  return 0   -- rejected
end

redis.call('HSET', key, 'tokens', newTokens - 1, 'lastRefill', now)
redis.call('EXPIRE', key, 3600)
return 1     -- allowed
`;

async function tokenBucketAllow(apiKey, capacity = 20, refillRate = 10) {
  const result = await redis.eval(
    TOKEN_BUCKET_SCRIPT, 1,
    `tb:${apiKey}`, capacity, refillRate, Date.now()
  );
  return result === 1;
}

app.use('/api/scrape', async (req, res, next) => {
  const apiKey = req.headers['x-api-key'];
  if (!apiKey) return res.status(401).json({ error: 'missing_api_key' });

  const allowed = await tokenBucketAllow(apiKey);
  if (!allowed) {
    return res.status(429).json({
      error: 'rate_limit_exceeded',
      message: 'Token bucket empty. Requests refill at 10/second.',
    });
  }
  next();
});

Implementing Rate Limiting in FastAPI

For Python-based scraping APIs, slowapi (a FastAPI/Starlette port of Flask-Limiter) integrates cleanly with Redis.

pip install slowapi redis fastapi uvicorn

Sliding Window Rate Limiting (FastAPI)

from fastapi import FastAPI, Request, HTTPException
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
import redis

# Use API key as the rate limit key, fall back to IP
def get_api_key(request: Request) -> str:
    return request.headers.get("x-api-key") or get_remote_address(request)

limiter = Limiter(
    key_func=get_api_key,
    storage_uri="redis://127.0.0.1:6379",
    strategy="sliding-window",
)

app = FastAPI()
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)

@app.get("/api/scrape")
@limiter.limit("100/15minutes")
async def scrape(request: Request, url: str):
    # Your scraping logic here
    return {"url": url, "status": "queued"}

Per-Tier Rate Limits (FastAPI)

Different plan tiers need different ceilings. Map API keys to tiers in Redis or your database:

TIER_LIMITS = {
    "free":       "20/hour",
    "starter":   "200/hour",
    "pro":       "2000/hour",
    "enterprise": "20000/hour",
}

def get_limit_for_request(request: Request) -> str:
    api_key = request.headers.get("x-api-key", "")
    # Look up tier from Redis or DB cache
    tier = redis_client.get(f"tier:{api_key}") or b"free"
    return TIER_LIMITS.get(tier.decode(), "20/hour")

@app.get("/api/scrape/advanced")
@limiter.limit(get_limit_for_request)
async def scrape_advanced(request: Request, url: str):
    return {"url": url, "tier": "applied"}

Rate Limit Response Headers

Clients need to know when their limit resets. The standard set of headers is:

Header	Value	Description
X-RateLimit-Limit	100	Max requests per window
X-RateLimit-Remaining	42	Remaining requests in current window
X-RateLimit-Reset	1710000000	Unix timestamp when window resets
Retry-After	47	Seconds until client can retry (on 429)

express-rate-limit emits these automatically when standardHeaders: true is set. For FastAPI with slowapi, the _rate_limit_exceeded_handler sets Retry-After on 429 responses.

IP-Based vs API Key-Based Rate Limiting

Approach	Pros	Cons	Best for
IP-based	No auth required, stops anonymous abuse	IPv6 subnets can bypass, shared NAT punishes teams	Public-facing endpoints
API key-based	Per-client fairness, supports tiers	Requires auth layer, key rotation overhead	Authenticated APIs
Combined	Defense in depth	More Redis keys, more logic	APIs in live use

Use IP-based limiting as a coarse outer guard and API key-based limiting as the precise per-client enforcer. A reasonable combined setup:

// 1. Outer IP guard: 500 req/15min across all endpoints
app.use(ipLimiter);

// 2. Inner key guard: 100 req/15min per API key on scrape routes
app.use('/api/scrape', apiKeyLimiter);

Protecting Against Abuse Patterns

Rate limiting alone doesn't cover all attack vectors. Pair it with:

Burst detection — if a client hits 50% of their limit in the first 10 seconds of a window, flag them for review. This catches burst-then-pause attack patterns.

Concurrent request caps — use a Redis counter with INCR/DECR to limit simultaneous in-flight requests per API key:

async function acquireConcurrencySlot(apiKey, maxConcurrent = 5) {
  const key = `concurrency:${apiKey}`;
  const current = await redis.incr(key);
  await redis.expire(key, 60);  // safety expiry
  if (current > maxConcurrent) {
    await redis.decr(key);
    return false;
  }
  return true;
}

async function releaseConcurrencySlot(apiKey) {
  await redis.decr(`concurrency:${apiKey}`);
}

Cost-based limiting — not all requests are equal. A JavaScript-rendered page scrape costs 10x a static HTML fetch. Deduct weighted tokens per request type:

SCRAPE_COSTS = {
    "html":       1,
    "javascript": 10,
    "screenshot": 15,
    "pdf":        20,
}

cost = SCRAPE_COSTS.get(request.query_params.get("mode", "html"), 1)
if not await token_bucket_allow(api_key, cost=cost):
    raise HTTPException(status_code=429, detail="Token budget exceeded")

Deploying on a Liquid Web VPS

A self-hosted scraping API with rate limiting requires a server that can sustain persistent Redis connections and handle bursts. Liquid Web VPS plans starting at around $25/month include:

Pricing verified June 2026. Plans change frequently — check the vendor’s official pricing page for current rates.

• NVMe SSD storage (fast enough for Redis AOF persistence)
• 1 Gbps network port with no bandwidth caps on higher tiers
• Fully managed security updates if you choose managed hosting
• Root SSH access for custom Redis and application configuration

For a scraping API handling under 100 concurrent users, a 2 vCPU / 2 GB RAM VPS is sufficient. Enable Redis persistence (appendonly yes in redis.conf) to survive reboots without losing rate limit counters.

If your scraping service sends traffic through proxies, see the proxy rotation guide to understand how IP-based limits interact with your proxy pool.

Monitoring Your Rate Limiter

Track these Redis keys to understand usage patterns:

# Count active rate limit keys
redis-cli --scan --pattern "rl:*" | wc -l

# Inspect a specific client's bucket
redis-cli HGETALL "tb:client_api_key_here"

# Monitor real-time commands
redis-cli monitor | grep "rl:"

Add structured logging when limits are hit to build usage dashboards:

handler: (req, res, next, options) => {
  console.log(JSON.stringify({
    event: 'rate_limit_exceeded',
    apiKey: req.headers['x-api-key'],
    ip: req.ip,
    path: req.path,
    timestamp: new Date().toISOString(),
  }));
  res.status(options.statusCode).json({ error: options.message });
}

FAQ

How do I rate limit my scraping API? Use express-rate-limit with a Redis store (Node.js) or slowapi (Python/FastAPI). Key on the client's API key for per-client fairness. Set limits based on your server's scraping capacity and proxy credits — start conservative (100 req/15min) and raise based on observed usage.

What is a token bucket? A token bucket is a rate limiting algorithm that allows bursts up to a maximum capacity while enforcing a long-term average rate. Tokens are consumed per request and refill at a constant rate. When the bucket is empty, new requests are rejected until tokens refill. It's ideal for scraping APIs where clients occasionally need burst capacity.

How do I prevent API abuse? Layer three controls: (1) API key authentication to identify clients, (2) Redis-backed rate limiting to enforce per-key ceilings, and (3) concurrency limits to cap simultaneous in-flight requests. Monitor for burst patterns and add IP-level guards as a fallback for unauthenticated paths.

What's the difference between sliding window and fixed window rate limiting? A fixed window resets the counter at regular intervals (e.g., every minute on the minute). A sliding window tracks requests in a rolling period relative to each request. Sliding window prevents the "boundary burst" problem where clients double their rate by bursting at the end of one window and the start of the next.

Should I use IP-based or API key-based rate limiting? Use both. IP-based limiting as a coarse outer guard for all endpoints (stops anonymous abuse), and API key-based limiting as precise per-client enforcement on authenticated routes. Combined limits provide defense in depth without requiring changes to your auth layer.

How many requests per minute is a reasonable scraping API limit? It depends on your proxy pool and target sites. A safe baseline: 10 req/min per API key for JavaScript-rendered scraping, 60 req/min for static HTML. For internal services on fast infrastructure, 200–500 req/min per key is common. Always set limits lower than your proxy provider's IP rotation speed.

Next Steps

Your scraping API now has solid rate limiting in place. From here:

• Add authentication: Use JWT or API key validation middleware before the rate limiter to enable per-client tier enforcement.
• Quota dashboards: Expose a /me/usage endpoint that returns current rate limit state from Redis — clients can self-monitor without hitting limits.
• Webhook alerts: Notify clients via webhook when they reach 80% of their quota — reduces frustrated support tickets.
• Upgrade your infrastructure: If Redis becomes a bottleneck, look at Redis Cluster or a managed Redis service (Redis Cloud, ElastiCache) — your rate limiting code won't change.

For managed VPS hosting that handles OS-level security and monitoring while you focus on your scraping logic, Liquid Web's VPS plans are worth evaluating. Their support team can help configure Redis for high-availability deployments.