OpenClaw Monitoring & Observability

Contains affiliate links. Disclosure.

TL;DR

• Catches the failures OpenClaw reports as "thinking…" and nothing else: an exhausted API key, a tool that fails on every call, spend that doubles overnight.
• Metrics come from the diagnostics-prometheus plugin at /api/diagnostics/prometheus, behind Gateway operator auth. There is no unauthenticated /metrics endpoint.
• Add Prometheus + Grafana + Alertmanager to your Compose stack. About 35 extra lines, 225 MB idle.
• The metric most people miss is openclaw_model_cost_usd_total, which reports spend in dollars with model and agent labels.

What does monitoring actually catch?

Three silent failure modes, all of which look identical to the end user. An LLM API key hits its quota and the assistant starts returning empty responses. A tool crashes on every invocation and users just see "OpenClaw is thinking…" forever. Token spend on an expensive model doubles overnight and you find out on the invoice.

Each one has a metric that goes visibly wrong minutes before anyone files a complaint, which is the whole argument for spending an afternoon on this. This guide extends the base stack from the self-hosted AI assistants hub, which also covers Hermes Agent and NemoClaw if OpenClaw isn't the only assistant on your VPS.

Prerequisites

• The Docker Compose stack already running (OpenClaw + Redis + Caddy)
• The diagnostics-prometheus plugin installed and enabled (below), plus a Gateway operator token carrying the operator.read scope for Prometheus to authenticate with
• Ports 9090 (Prometheus) and 3000 (Grafana) available on the host (or proxied through Caddy)

Extended docker-compose.yml

What this block does: runs the three monitoring containers alongside OpenClaw on the same Docker network, so Prometheus can reach the Gateway by service name. Add them to your existing docker-compose.yml. These tags are the current stable releases as of 2026-09-10, and the compose file pins exact versions deliberately rather than floating on latest; no OpenClaw source certifies particular Prometheus, Alertmanager, or Grafana versions, so check each project's own release notes before bumping.

  prometheus:
    image: prom/prometheus:v3.14.0
    restart: unless-stopped
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    networks:
      - openclaw_net

  grafana:
    image: grafana/grafana:13.2.1
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-changeme}
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./monitoring/grafana/datasources:/etc/grafana/provisioning/datasources:ro
    depends_on:
      - prometheus
    networks:
      - openclaw_net

  alertmanager:
    image: prom/alertmanager:v0.34.0
    restart: unless-stopped
    volumes:
      - ./monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    networks:
      - openclaw_net

What this block does: keeps metric history and Grafana's dashboard database across container restarts. Add it under the top-level volumes: key.

  prometheus_data:
  grafana_data:

What this block does: puts Grafana on a real hostname with a certificate instead of a raw port. Add it to your Caddyfile.

grafana.yourdomain.com {
    reverse_proxy grafana:3000
}

prometheus.yml

What this block does: tells Prometheus what to scrape, how to authenticate to the Gateway, and where to send firing alerts. Create it at monitoring/prometheus.yml. The credentials_file holds a Gateway operator token as a single line of plain text.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "openclaw"
    metrics_path: "/api/diagnostics/prometheus"
    authorization:
      credentials_file: /etc/prometheus/openclaw-gateway-token
    static_configs:
      - targets: ["openclaw:18789"]

  - job_name: "redis"
    static_configs:
      - targets: ["redis:9121"]

  - job_name: "caddy"
    static_configs:
      - targets: ["caddy:2019"]
    metrics_path: "/metrics"

Note: Redis metrics require the redis_exporter sidecar. Add it to your Compose file if you want Redis metrics beyond what OpenClaw reports indirectly.

Which OpenClaw metrics are worth a dashboard panel?

Ten of them, out of the several dozen the plugin publishes. Install and enable the plugin first, then restart the Gateway:

openclaw plugins install clawhub:@openclaw/diagnostics-prometheus
openclaw plugins enable diagnostics-prometheus

openclaw plugins enable writes plugins.entries."diagnostics-prometheus".enabled for you, so there is nothing else to edit unless you have explicitly turned diagnostics.enabled off in your Gateway config; it defaults to true.

Confirm the endpoint answers before you point Prometheus at it:

curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
  http://127.0.0.1:18789/api/diagnostics/prometheus

That same output is the authoritative list of label values for your build. These ten are the ones worth a panel, with the labels OpenClaw documents:

Metric	Type	Labels	What it tells you
openclaw_liveness_sessions	Gauge	state	Sessions currently open, broken out by state
openclaw_message_processed_total	Counter	channel, outcome, reason	Messages handled, and why the failures failed
openclaw_model_call_total	Counter	provider, model, outcome, error_category, api, transport, observation_unit	LLM calls and their outcomes. Filter outcome for the error rate
openclaw_model_call_duration_seconds	Histogram	same as above	End-to-end LLM latency
openclaw_model_tokens_total	Counter	provider, model, agent, channel, token_type	Token consumption, split input vs output
openclaw_model_cost_usd_total	Counter	provider, model, agent, channel	Spend, in dollars, straight from the Gateway
openclaw_skill_used_total	Counter	skill, agent, activation, source	Which skills actually fire
openclaw_tool_execution_total	Counter	tool, outcome, error_category, tool_owner, tool_source, params_kind	Tool calls and failures
openclaw_tool_execution_blocked_total	Counter	tool, denied_reason, tool_owner, tool_source, params_kind	Tool calls refused by policy
openclaw_session_stuck_total	Counter	reason, state	Sessions wedged, by cause

openclaw_model_cost_usd_total is the one most people miss. It turns "the bill went up" into a per-agent, per-model graph without any extra instrumentation.

openclaw_skill_used_total and openclaw_tool_execution_total carry a skill and a tool label, so a spike narrows to a specific ClawHub skill rather than the whole instance. If you're the one writing that skill, the OpenClaw skill development guide covers where errors in a SKILL.md-based skill surface.

Two limits to plan around. The exporter keeps at most 2,048 time series in memory across all counters, gauges and histograms; past that it drops new series rather than rewriting labels, and openclaw_prometheus_series_dropped_total starts climbing. And the plugin holds state in memory only, so a Gateway restart resets every counter to zero. Write queries with rate() and increase(), which handle resets, and avoid raw counter values on dashboards.

What should you alert on?

Four things: slow model calls, failing tools, a session spike, and a burst of model-call errors. Everything else is a dashboard you look at deliberately, not a page at 3am.

What this block does: defines those four rules for Prometheus to evaluate. Create it at monitoring/rules/openclaw.yml, matching the rule_files glob above.

groups:
  - name: openclaw
    rules:
      - alert: HighLLMLatency
        expr: histogram_quantile(0.95, sum by (le) (rate(openclaw_model_call_duration_seconds_bucket[5m]))) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OpenClaw LLM P95 latency > 10s for 5 minutes"
          description: "Check LLM provider status and network connectivity from the VPS."

      - alert: HighToolErrorRate
        expr: |
          sum by (tool) (rate(openclaw_tool_execution_total{outcome!="success"}[5m]))
            / sum by (tool) (rate(openclaw_tool_execution_total[5m])) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "OpenClaw tool error rate > 5%"
          description: "Tool {{ $labels.tool }} is failing on more than 5% of calls. Check docker compose logs openclaw."

      - alert: SessionSpike
        expr: sum(openclaw_liveness_sessions) > 50
        for: 1m
        labels:
          severity: info
        annotations:
          summary: "OpenClaw active sessions > 50"
          description: "Unusual session spike. Check for bot traffic or integration misconfiguration."

      - alert: ModelCallFailureSpike
        expr: |
          sum by (provider, model) (increase(openclaw_model_call_total{outcome!="success"}[10m])) > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Model call failures spiking for {{ $labels.provider }}/{{ $labels.model }}"
          description: "Key may be exhausted or rate limited. Check your Anthropic/OpenAI dashboard."

One caveat on both rules above. OpenClaw's reference documents that openclaw_model_call_total and openclaw_tool_execution_total carry outcome and error_category labels, but it does not enumerate the values either label takes, so outcome!="success" is an assumption about the success value's spelling, not something the docs confirm. Before you trust either alert, run the curl scrape above during a real failure and read the label values your build actually emits, then pin the selector to those. A rule matching a string you guessed sits silently at zero forever, which is worse than no rule at all.

alertmanager.yml

What this block does: decides where a firing alert goes and how often it repeats. Create it at monitoring/alertmanager.yml. Swap the slack_configs receiver for email_configs or webhook_configs if you don't use Slack.

route:
  group_by: ["alertname"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "slack"

receivers:
  - name: "slack"
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#openclaw-alerts"
        title: "OpenClaw Alert: {{ .CommonLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"

Add SLACK_WEBHOOK_URL and GRAFANA_ADMIN_PASSWORD to your .env.

Grafana datasource provisioning

What this block does: wires Grafana to Prometheus on first boot so you never click through the data-source form. Create it at monitoring/grafana/datasources/prometheus.yml. The name here must match the data-source name any dashboard JSON you import expects.

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    editable: false

Useful PromQL queries

# LLM P95 latency (last 5 minutes), across all providers
histogram_quantile(0.95, sum by (le) (rate(openclaw_model_call_duration_seconds_bucket[5m])))

# Same, broken out per model
histogram_quantile(0.95, sum by (le, model) (rate(openclaw_model_call_duration_seconds_bucket[5m])))

# Token spend rate (tokens/hour), split input vs output
sum by (token_type) (rate(openclaw_model_tokens_total[1h])) * 3600

# What it is actually costing, per model, per hour
sum by (model) (rate(openclaw_model_cost_usd_total[1h])) * 3600

# Top 5 failing tools
topk(5, sum by (tool) (rate(openclaw_tool_execution_total{outcome!="success"}[10m])))

# Messages per channel (last hour)
sum by (channel) (increase(openclaw_message_processed_total[1h]))

How much RAM does the monitoring stack cost you?

About 225 MB idle, which fits on the 4 GB VPS you are already paying for. No upgrade needed. Measured 2026-05-01 on local Docker (4 vCPU / 4 GB RAM):

Service	Idle RAM
prometheus	80 MB
grafana	120 MB
alertmanager	25 MB
Total monitoring	225 MB

This is acceptable on a 4 GB VPS (total stack: ~460 MB idle with monitoring). If you add a local Ollama backend, see the GPU inference on Liquid Web guide and move to an 8 GB VPS.

When is this setup not enough?

In three cases, and only one of them costs money.

• Multi-instance OpenClaw. Use a shared Prometheus with federation or Thanos for horizontal scale.
• Long-term metric retention > 30 days. Configure remote_write to a hosted Prometheus. Grafana Cloud's free tier allows 10k active series with 14-day retention, which one OpenClaw Gateway cannot exceed because the exporter caps itself at 2,048 series. Going past 14 days means the Pro tier, which starts at $19/month ^{subject to change · verified 2026-09-10} and includes the same 10k series with 13-month retention. Self-hosting your own Prometheus for 30 days, as configured above, stays free.
• Compliance audit trail. Metrics don't satisfy HIPAA/SOC audit requirements. For regulated deployments, add structured log shipping (Loki or Elasticsearch) and retain logs per your compliance policy. See the HIPAA deployment guide.

Frequently Asked Questions

Either add a Caddy reverse proxy entry (as shown above) or use an SSH tunnel: ssh -L 3000:localhost:3000 user@your-vps. The tunnel is sufficient for solo use; the Caddy proxy is better for teams.

Yes. Point your existing Prometheus at /api/diagnostics/prometheus on the Gateway port, give the job an operator token with the operator.read scope, and import the dashboard JSON into your Grafana. Skip the prometheus, grafana, and alertmanager services in the Compose file.

Use openclaw_model_cost_usd_total, which the diagnostics-prometheus plugin reports in dollars with agent, channel, model and provider labels, so you do not have to multiply token counts by a price list that changes under you. sum by (model) (rate(openclaw_model_cost_usd_total[1h])) * 3600 gives you spend per model per hour. If you want the underlying volume too, openclaw_model_tokens_total carries a token_type label so you can separate input from output.