Monitoring Scrapers with Grafana: Prometheus Dashboard 2026

Contains affiliate links. Disclosure.

Monitor self-hosted scrapers with Prometheus and Grafana. Track success rate, pages per minute, error rate, latency, and data volume. This guide uses a Docker Compose stack. Grafana Cloud is a managed alternative.

For fully managed scraping with built-in run metrics and logs, Apify provides dashboards and alerting without any setup.

Key Metrics to Track

Metric	Purpose
scraper_requests_total	Total requests (labels: status, target)
scraper_errors_total	Failed requests
scraper_pages_per_minute	Throughput
scraper_request_duration_seconds	Latency (histogram)
scraper_items_scraped_total	Data volume
scraper_queue_depth	Pending URLs (Redis/Celery)

Docker Compose Stack

# docker-compose.monitoring.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.52.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'
    ports:
      - "9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.0.0
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
    restart: unless-stopped

  loki:
    image: grafana/loki:3.0.0
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    ports:
      - "3100:3100"
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:
  loki_data:

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'scrapers'
    static_configs:
      - targets: ['scraper-host:9090']
    metrics_path: /metrics
    scrape_interval: 30s

Exposing Metrics from Your Scraper

Python (Prometheus client):

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter('scraper_requests_total', 'Total requests', ['status'])
DURATION = Histogram('scraper_request_duration_seconds', 'Request latency')

def scrape_page(url):
    with DURATION.time():
        r = requests.get(url)
        REQUESTS.labels(status=str(r.status_code)).inc()
        return r

if __name__ == '__main__':
    start_http_server(9090)
    # ... run scraper

Node.js (prom-client):

const { Counter, Histogram, register } = require('prom-client');

const requests = new Counter({
  name: 'scraper_requests_total',
  help: 'Total requests',
  labelNames: ['status'],
});
const duration = new Histogram({
  name: 'scraper_request_duration_seconds',
  help: 'Request latency',
});

// In scraper: requests.inc({ status: '200' }); duration.observe(0.5);

Grafana Dashboard Panels

1. Success rate — rate(scraper_requests_total{status="200"}[5m]) / rate(scraper_requests_total[5m]) * 100
2. Pages per minute — rate(scraper_requests_total[1m]) * 60
3. Error rate — rate(scraper_errors_total[5m])
4. p99 latency — histogram_quantile(0.99, rate(scraper_request_duration_seconds_bucket[5m]))
5. Queue depth — scraper_queue_depth (if you export it from Redis)

Alerting Rules

# prometheus/rules.yml
groups:
  - name: scrapers
    rules:
      - alert: ScraperHighErrorRate
        expr: rate(scraper_errors_total[5m]) / rate(scraper_requests_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Scraper error rate above 10%"

      - alert: ScraperDown
        expr: up{job="scrapers"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Scraper target unreachable"

Loki for Logs

Forward scraper logs to Loki for centralized search. Use Promtail or your logging driver.

# loki-config.yml (simplified)
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Next step

Add Prometheus metrics to your scraper. Deploy the Docker stack. Create a Grafana dashboard for success rate and latency. For managed scraping with built-in dashboards, use Apify.

Frequently Asked Questions

Expose Prometheus metrics from your scraper (counters, histograms). Scrape with Prometheus. Visualize in Grafana. Alert on error rate and latency.

Track requests total, errors, pages per minute, request duration (p95/p99), and queue depth if using a job queue.

Run Prometheus + Grafana via Docker Compose. Add Prometheus as data source. Create panels for success rate, throughput, and latency.

Grafana Cloud is a managed option. Self-hosted is free but you manage upgrades and retention. Both work with the same Prometheus metrics.