Prometheus metrics

OpenClaw can expose diagnostics metrics through the official diagnostics-prometheus plugin. It listens to trusted diagnostics plus internally tagged, dispatcher-owned diagnostic events (queue, memory, and session-recovery signals), and renders a Prometheus text endpoint at:

GET /api/diagnostics/prometheus

Content type is text/plain; version=0.0.4; charset=utf-8, the standard Prometheus exposition format.

The route uses Gateway authentication (operator scope, trusted-operator surface). Do not expose it as a public unauthenticated /metrics endpoint. Scrape it through the same auth path you use for other operator APIs.

For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, see OpenTelemetry export.

Quick start

1. Install the plugin

   openclaw plugins install clawhub:@openclaw/diagnostics-prometheus
   
2. Enable the plugin

   Config

   {
     plugins: {
       allow: ["diagnostics-prometheus"],
       entries: {
         "diagnostics-prometheus": { enabled: true },
       },
     },
     diagnostics: {
       enabled: true,
     },
   }
   

   CLI

   openclaw plugins enable diagnostics-prometheus
   
3. Restart the Gateway

   The HTTP route is registered at plugin startup, so reload after enabling.

4. Scrape the protected route

   Send the same gateway auth your operator clients use:

   curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
     http://127.0.0.1:18789/api/diagnostics/prometheus
   
5. Wire Prometheus

   # prometheus.yml
   scrape_configs:
     - job_name: openclaw
       scrape_interval: 30s
       metrics_path: /api/diagnostics/prometheus
       authorization:
         credentials_file: /etc/prometheus/openclaw-gateway-token
       static_configs:
         - targets: ["openclaw-gateway:18789"]
   

diagnostics.enabled defaults to true; set it to false only in tightly constrained environments. If it is false, the plugin still registers the HTTP route, but no diagnostic events flow into the exporter, so the response is empty.

Metrics exported

Metric	Type	Labels
openclaw_gateway_rpc_requests_total	counter	method
openclaw_gateway_rpc_first_response_seconds	histogram	method
openclaw_gateway_rpc_handler_seconds	histogram	method
openclaw_gateway_rpc_admission_seconds	histogram	method
openclaw_gateway_rpc_queue_wait_seconds	histogram	method
openclaw_gateway_rpc_outcomes_total	counter	phase, outcome
openclaw_run_completed_total	counter	channel, model, outcome, provider, trigger
openclaw_run_duration_seconds	histogram	channel, model, outcome, provider, trigger
openclaw_model_call_total	counter	api, error_category, model, observation_unit, outcome, provider, transport
openclaw_model_call_duration_seconds	histogram	api, error_category, model, observation_unit, outcome, provider, transport
openclaw_model_failover_total	counter	from_model, from_provider, lane, reason, suspended, to_model, to_provider
openclaw_model_tokens_total	counter	agent, channel, model, provider, token_type
openclaw_gen_ai_client_token_usage	histogram	model, provider, token_type
openclaw_model_cost_usd_total	counter	agent, channel, model, provider
openclaw_model_usage_duration_seconds	histogram	agent, channel, model, provider
openclaw_skill_used_total	counter	activation, agent, skill, source
openclaw_tool_execution_total	counter	error_category, outcome, params_kind, tool, tool_owner, tool_source
openclaw_tool_execution_duration_seconds	histogram	error_category, outcome, params_kind, tool, tool_owner, tool_source
openclaw_tool_execution_blocked_total	counter	denied_reason, params_kind, tool, tool_owner, tool_source
openclaw_harness_run_total	counter	channel, error_category, harness, model, outcome, phase, plugin, provider
openclaw_harness_run_duration_seconds	histogram	channel, error_category, harness, model, outcome, phase, plugin, provider
openclaw_webhook_received_total	counter	channel, webhook
openclaw_webhook_error_total	counter	channel, webhook
openclaw_webhook_duration_seconds	histogram	channel, webhook
openclaw_message_received_total	counter	channel, source
openclaw_message_dispatch_started_total	counter	channel, source
openclaw_message_dispatch_completed_total	counter	channel, outcome, reason, source
openclaw_message_dispatch_duration_seconds	histogram	channel, outcome, reason, source
openclaw_message_processed_total	counter	channel, outcome, reason
openclaw_message_processed_duration_seconds	histogram	channel, outcome, reason
openclaw_message_delivery_started_total	counter	channel, delivery_kind
openclaw_message_delivery_total	counter	channel, delivery_kind, error_category, outcome
openclaw_message_delivery_duration_seconds	histogram	channel, delivery_kind, error_category, outcome
openclaw_talk_event_total	counter	brain, event_type, mode, provider, transport
openclaw_talk_event_duration_seconds	histogram	brain, event_type, mode, provider, transport
openclaw_talk_audio_bytes	histogram	brain, event_type, mode, provider, transport
openclaw_queue_lane_size	gauge	lane
openclaw_queue_lane_wait_seconds	histogram	lane
openclaw_session_state_total	counter	reason, state
openclaw_session_queue_depth	gauge	state
openclaw_session_turn_created_total	counter	agent, channel, trigger
openclaw_session_stuck_total	counter	reason, state
openclaw_session_stuck_age_seconds	histogram	reason, state
openclaw_session_recovery_total	counter	action, active_work_kind, state, status
openclaw_session_recovery_age_seconds	histogram	action, active_work_kind, state, status
openclaw_gateway_event_loop_delay_max_seconds	histogram	none
openclaw_gateway_event_loop_observed_seconds_total	counter	none
openclaw_liveness_warning_total	counter	reason
openclaw_liveness_sessions	gauge	state
openclaw_liveness_event_loop_delay_p99_seconds	histogram	reason
openclaw_liveness_event_loop_delay_max_seconds	histogram	reason
openclaw_liveness_event_loop_utilization_ratio	histogram	reason
openclaw_liveness_cpu_core_ratio	histogram	reason
openclaw_payload_large_total	counter	action, channel, plugin, reason, surface
openclaw_payload_large_bytes	histogram	action, channel, plugin, reason, surface
openclaw_memory_bytes	gauge	kind
openclaw_memory_rss_bytes	histogram	none
openclaw_memory_pressure_total	counter	level, reason
openclaw_telemetry_exporter_total	counter	exporter, reason, signal, status
openclaw_prometheus_series_dropped_total	counter	none
openclaw_diagnostic_async_queue_dropped_total	counter	drop_class
openclaw_diagnostic_async_queue_length	gauge	none

For model-call metrics, observation_unit="request" measures one observable provider request. observation_unit="turn" measures a synthetic Claude Code or Codex CLI agent turn that can contain multiple hidden provider requests. Keep those series separate when comparing latency.

Gateway RPC metrics cover valid authenticated WebSocket requests, including subsequent rejections. first_response measures receipt through the first frame accepted by the sender; unavailable or suppressed sends have no duration sample. handler measures actual handler invocation through return or throw, and admission measures receipt through that invocation. queue_wait measures only operator request start-queue wait, separately from command/session lane metrics. They measure elapsed time, not CPU time. Early acknowledgments and responses after handler return are distinct from completed agent work. See Gateway RPC timing semantics.

RPC method labels contain canonical core method names, other for plugin methods, or unknown. Outcome totals aggregate by phase and outcome without a method dimension. Each method with all four timings occupies five aggregate samples in the shared 2,048-sample cap. A duration histogram occupies one sample but expands into 19 scrape series (buckets, sum, and count). Existing samples keep updating when the cap fills; unseen RPC or other operational samples are refused and increment openclaw_prometheus_series_dropped_total. Monitor that counter: coverage of every core method can fill the cap, so a zero value matters when interpreting totals or latency percentiles. Async diagnostic queue saturation can also drop observations, reported by openclaw_diagnostic_async_queue_dropped_total.

Event-loop observation windows

openclaw_liveness_cpu_core_ratio measures whole-process CPU usage in core equivalents, including worker and native threads, and can exceed 1. Interpret it alongside main-thread delay and utilization; see CPU pressure and event-loop delay.

The event-loop histogram records the maximum delay from each completed Gateway health-monitor window. The counter sums the seconds represented by those windows. Both are cumulative: a later healthy window does not erase an earlier high-delay observation. Readiness, status, and scrape requests consume completed observations without advancing or resetting the sampling window.

The monitor samples elapsed event-loop intervals every 20 milliseconds and completes a window after at least one second, or sooner for a delay warning. It preserves the pending interval across ordinary window resets, so reading health before an overdue sample cannot erase that delay. Histogram counts are window counts, not stall counts. Histogram quantiles describe window maxima, not the sampled event-loop delay distribution or its overall p99. These metrics have no request labels or trace attribution and do not identify the JavaScript function that blocked.

Collection uses the plugin enablement above. It starts when an interested metrics exporter is running; it does not backfill earlier windows. Intentional monitor resets discard the unfinished window. Diagnostic queue drops, the exporter’s series cap, and process restarts can also lose observations. Watch the existing drop counters and the represented-duration counter when assessing coverage. Readiness decisions and persistent liveness-warning thresholds are unchanged.

Label policy

Bounded, low-cardinality labels

Prometheus labels stay bounded and low-cardinality. The exporter does not emit raw diagnostic identifiers such as runId, sessionKey, sessionId, callId, toolCallId, message IDs, chat IDs, or provider request IDs.

Label values are redacted and must match OpenClaw’s low-cardinality character policy. Values that fail the policy are replaced with unknown, other, or none, depending on the metric. Labels that look like scoped agent session keys are also replaced with unknown.

Series cap and overflow accounting

The exporter caps retained time series in memory at 2048 series across counters, gauges, and histograms combined. New series beyond that cap are dropped, and openclaw_prometheus_series_dropped_total increments by one each time.

Watch this counter as a hard signal that an attribute upstream is leaking high-cardinality values. The exporter never lifts the cap automatically; if it climbs, fix the source rather than disabling the cap.

What never appears in Prometheus output

• prompt text, response text, tool inputs, tool outputs, system prompts
• Talk transcripts, audio payloads, call ids, room ids, handoff tokens, turn ids, and raw session ids
• raw provider request IDs (only bounded hashes, where applicable, on spans — never on metrics)
• session keys and session IDs
• hostnames, file paths, secret values

PromQL recipes

# Gateway RPC requests per second by method
sum by (method) (rate(openclaw_gateway_rpc_requests_total[5m]))

# 95th percentile first-response latency by method
histogram_quantile(
  0.95,
  sum by (le, method) (rate(openclaw_gateway_rpc_first_response_seconds_bucket[5m]))
)

# 95th percentile operator request start-queue wait by method
histogram_quantile(
  0.95,
  sum by (le, method) (rate(openclaw_gateway_rpc_queue_wait_seconds_bucket[5m]))
)

# Tokens per minute, split by provider
sum by (provider) (rate(openclaw_model_tokens_total[1m]))

# Spend (USD) over the last hour, by model
sum by (model) (increase(openclaw_model_cost_usd_total[1h]))

# 95th percentile model run duration
histogram_quantile(
  0.95,
  sum by (le, provider, model)
    (rate(openclaw_run_duration_seconds_bucket[5m]))
)

# Queue wait time SLO (95p under 2s)
histogram_quantile(
  0.95,
  sum by (le, lane) (rate(openclaw_queue_lane_wait_seconds_bucket[5m]))
) < 2

# Skill usage, split by bounded source
sum by (skill, source) (increase(openclaw_skill_used_total[24h]))

# Dropped Prometheus series (cardinality alarm)
increase(openclaw_prometheus_series_dropped_total[15m]) > 0

# Completed windows whose maximum delay exceeded one second
increase(openclaw_gateway_event_loop_delay_max_seconds_count[5m])
  - increase(openclaw_gateway_event_loop_delay_max_seconds_bucket{le="1"}[5m])

# Seconds represented by exported event-loop windows
increase(openclaw_gateway_event_loop_observed_seconds_total[5m])

Prefer gen_ai_client_token_usage for cross-provider dashboards: it follows the OpenTelemetry GenAI semantic conventions and is consistent with metrics from non-OpenClaw GenAI services.

Choosing between Prometheus and OpenTelemetry export

OpenClaw supports both surfaces independently. You can run either, both, or neither.

diagnostics-prometheus

• Pull model: Prometheus scrapes /api/diagnostics/prometheus.
• No external collector required.
• Authenticated through normal Gateway auth.
• Surface is metrics only (no traces or logs).
• Best for stacks already standardized on Prometheus + Grafana.

diagnostics-otel

• Push model: OpenClaw sends OTLP/HTTP to a collector or OTLP-compatible backend.
• Surface includes metrics, traces, and logs.
• Bridges to Prometheus through an OpenTelemetry Collector (prometheus or prometheusremotewrite exporter) when you need both.
• See OpenTelemetry export for the full catalog.

Troubleshooting

Empty response body

• Check that diagnostics.enabled is not set to false in config (it defaults to true).
• Confirm the plugin is enabled and loaded with openclaw plugins list --enabled.
• Generate some traffic; counters and histograms only emit lines after at least one event.

401 / unauthorized

The endpoint requires the Gateway operator scope (auth: "gateway" with gatewayRuntimeScopeSurface: "trusted-operator"). Use the same token or password Prometheus uses for any other Gateway operator route. There is no public unauthenticated mode.

openclaw_prometheus_series_dropped_total is climbing

A new attribute is exceeding the 2048-series cap. Inspect recent metrics for an unexpectedly high-cardinality label and fix it at the source. The exporter intentionally drops new series instead of silently rewriting labels.

Prometheus shows stale series after a restart

The plugin keeps state in memory only. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL rate() and increase() to handle resets cleanly.

Related

• Diagnostics export — local diagnostics zip for support bundles
• Health and readiness — /healthz and /readyz probes
• Logging — file-based logging
• OpenTelemetry export — OTLP push for traces, metrics, and logs