Nginx Reverse Proxy for Scraping APIs: SSL, Rate Limiting 2026

Contains affiliate links. Disclosure.

Use Nginx as a reverse proxy for your scraping API: SSL termination, rate limiting, load balancing. Put Nginx in front of FastAPI, Flask, or Node.js scrapers. For managed APIs with no server setup, Apify exposes scrapers via REST API.

Why Use Nginx as Reverse Proxy

• SSL termination — Terminate TLS at Nginx; backend stays HTTP.
• Rate limiting — Protect backend from abuse and respect target quotas.
• Load balancing — Distribute traffic across multiple scraper instances.
• Hiding backend — Backend IP and port stay internal.

Basic proxy_pass Configuration

# /etc/nginx/sites-available/scraping-api
upstream scraping_api {
    server 127.0.0.1:3000;
    # Add more for load balancing:
    # server 127.0.0.1:3001;
    # server 127.0.0.1:3002;
}

server {
    listen 80;
    server_name api.scraper.example.com;

    location / {
        proxy_pass http://scraping_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}

Rate Limiting with limit_req

# In http block (e.g. /etc/nginx/nginx.conf)
http {
    # 10 req/s per IP, burst 20
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req_status 429;
}

# In server block
server {
    location / {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://scraping_api;
        # ... other proxy directives
    }

    # Stricter limit for heavy endpoints
    location /crawl {
        limit_req zone=api burst=5 nodelay;
        proxy_pass http://scraping_api;
    }
}

SSL with Certbot

# Install
sudo apt install certbot python3-certbot-nginx -y

# Obtain certificate (Certbot edits Nginx config)
sudo certbot --nginx -d api.scraper.example.com

# Auto-renewal (usually pre-configured)
sudo systemctl status certbot.timer

Certbot adds a listen 443 ssl block and configures certificate paths. See SSL security for scraping servers for hardening.

Full Example: HTTP + HTTPS

upstream scraping_api {
    server 127.0.0.1:3000;
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name api.scraper.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name api.scraper.example.com;

    ssl_certificate /etc/letsencrypt/live/api.scraper.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.scraper.example.com/privkey.pem;

    limit_req zone=api burst=20 nodelay;

    location / {
        proxy_pass http://scraping_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

API Key or Basic Auth (Optional)

# Basic auth
auth_basic "Scraping API";
auth_basic_user_file /etc/nginx/.htpasswd;

# Or pass API key header to backend for validation
# (backend validates X-API-Key)

Testing

# Test config
sudo nginx -t

# Reload
sudo systemctl reload nginx

# Test upstream
curl -H "Host: api.scraper.example.com" http://127.0.0.1/

Next step

Start with a basic proxy_pass. Add rate limiting before going live. Use Certbot for SSL. For zero server setup, use Apify's API.

Frequently Asked Questions

Run your scraper behind FastAPI, Flask, or Express. Put Nginx in front with proxy_pass. Add rate limiting and SSL with Certbot.

A reverse proxy sits between clients and your backend. It receives requests, forwards them to the backend, and returns responses. Nginx handles SSL and rate limiting at the edge.

Use SSL (Certbot), rate limiting (limit_req), and API keys or basic auth. See our SSL security guide for TLS hardening.