SSL and Security for Scraping Servers: TLS, Certbot 2026

Contains affiliate links. Disclosure.

Secure scraping API servers with Let's Encrypt and Certbot. Use Nginx with TLS 1.2/1.3 only, HSTS, and strong ciphers. Auto-renew certificates with a systemd timer. For managed APIs with built-in SSL, use Apify.

Let's Encrypt with Certbot

Certbot issues and installs free certificates. Standard certificates last 90 days; renewal is automated.

# Ubuntu/Debian
sudo apt update
sudo apt install certbot python3-certbot-nginx -y

# Obtain certificate (Nginx plugin edits config automatically)
sudo certbot --nginx -d api.scraper.example.com -d www.api.scraper.example.com

Certbot configures HTTPS and adds a renewal hook. Certificates go to /etc/letsencrypt/live/<domain>/.

Production Nginx SSL Config

Use TLS 1.2 and 1.3 only. Disable deprecated protocols.

server {
    listen 443 ssl http2;
    server_name api.scraper.example.com;

    ssl_certificate     /etc/letsencrypt/live/api.scraper.example.com/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/api.scraper.example.com/privkey.pem;

    # TLS 1.2 and 1.3 only
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;

    # Modern ciphers (ECDHE, forward secrecy)
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;

    # Session caching
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;

    # HSTS (uncomment after verifying HTTPS works)
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

HTTP to HTTPS Redirect

server {
    listen 80;
    server_name api.scraper.example.com;
    return 301 https://$host$request_uri;
}

Keep the HTTP block for ACME challenges. Certbot uses /.well-known/acme-challenge/ for validation.

Auto-Renewal with systemd

Certbot installs a systemd timer. Verify it's active:

sudo systemctl status certbot.timer
sudo systemctl enable certbot.timer

Test renewal (dry run):

sudo certbot renew --dry-run

Certbot runs renewal twice daily. After renewal, it reloads Nginx if the nginx plugin was used. For custom deploy hooks:

# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
#!/bin/sh
systemctl reload nginx

Certificate Monitoring

Alert when certificates are close to expiry. Example script:

# Check days until expiry
openssl s_client -connect api.scraper.example.com:443 -servername api.scraper.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -checkend 1209600  # 14 days in seconds
# Exit 0 if valid for 14+ days, 1 otherwise

Use this in cron or Prometheus blackbox exporter for alerting.

Additional Hardening

Action	Purpose
fail2ban	Block IPs after repeated auth failures
SSH key-only auth	Disable password login
ufw/iptables	Restrict ports (22, 80, 443)
Non-root process	Run scraper as dedicated user

See Liquid Web VPS setup for initial server hardening.

Next step

Install Certbot. Obtain certificate. Harden Nginx TLS config. Verify auto-renewal. For managed APIs with SSL handled for you, use Apify.

Frequently Asked Questions

Use Let's Encrypt + Certbot for SSL. Configure Nginx with TLS 1.2/1.3, HSTS, strong ciphers. Restrict SSH, use fail2ban, run scrapers as non-root.

Yes, if you expose a scraping API over the internet. HTTPS protects API keys and data in transit. Internal-only scrapers can use HTTP.

Use API keys or basic auth. Put Nginx in front with rate limiting. Restrict by IP if feasible. See our nginx reverse proxy guide.