From aea78222da3623362ba050d120aba2604e02863e Mon Sep 17 00:00:00 2001 From: seemeroland Date: Thu, 9 Jul 2026 19:13:57 +0000 Subject: [PATCH] docs: add Grafana dashboard import instructions to monitoring page Add a new 'Grafana dashboard' section to the self-hosted worker monitoring page, covering: - Dashboard layout and sections overview - Step-by-step import instructions (Grafana UI, API, and direct URL) - Prometheus alerting rules reference with alert descriptions - Links to the grafana/ directory in oz-agent-worker Also adds the Steps component import needed for the step-by-step import workflow. Addresses REMOTE-2110 (enterprise customer IMC requesting Grafana dashboard templates for self-hosted Oz worker monitoring). Co-Authored-By: Oz --- .../docs/platform/self-hosting/monitoring.mdx | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/src/content/docs/platform/self-hosting/monitoring.mdx b/src/content/docs/platform/self-hosting/monitoring.mdx index a66d52f47..6fb578d63 100644 --- a/src/content/docs/platform/self-hosting/monitoring.mdx +++ b/src/content/docs/platform/self-hosting/monitoring.mdx @@ -7,6 +7,7 @@ description: >- Prometheus, OTLP, or console to track worker health, task throughput, and saturation. --- +import { Steps } from '@astrojs/starlight/components'; The `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the [Oz dashboard](https://oz.warp.dev) for full observability across both the orchestration plane and your self-hosted compute. @@ -148,6 +149,66 @@ All metrics use the `oz_worker_` prefix. Each worker process emits a distinct se --- +## Grafana dashboard + +A ready-to-import Grafana dashboard is published in the [`grafana/` directory of the oz-agent-worker repository](https://github.com/warpdotdev/oz-agent-worker/tree/main/grafana). It covers all five areas of worker observability — health, throughput, latency, failures, and connectivity — and works with any Prometheus datasource. + +### Dashboard sections + +* **Worker Health** — Connected workers, active workers, tasks active, and fleet saturation gauge. +* **Task Throughput** — Task completion rate by result (succeeded/failed/cancelled), 5-minute success rate, and rejection rate. +* **Task Duration** — p50 / p95 / p99 latency over a sliding 5-minute window. +* **Failure Analysis** — Failure rate by reason (e.g., `image_pull`, `container_oom`, `unschedulable`) and by phase (assignment, backend, cleanup). +* **Connection Health** — WebSocket reconnect rate by reason, and a worker fleet info table showing each connected worker's ID, version, and backend. + +The dashboard also includes an **Instance** variable so you can filter to a specific worker pod or see fleet-wide aggregations. + +### Importing the dashboard + + + +1. Enable Prometheus metrics on your workers. See [Enable Prometheus scrape](#enable-prometheus-scrape) above, or set `metrics.enabled=true` in the Helm chart. + +2. In Grafana, go to **Dashboards** → **Import**. + +3. Click **Upload JSON file** and select `oz-agent-worker-overview.json` from the [`grafana/` directory](https://github.com/warpdotdev/oz-agent-worker/tree/main/grafana), or paste the raw file URL: + + ``` + https://raw.githubusercontent.com/warpdotdev/oz-agent-worker/main/grafana/oz-agent-worker-overview.json + ``` + +4. When prompted, choose your Prometheus datasource and click **Import**. + + + +You can also import via the Grafana API: + +```bash +curl -X POST \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $GRAFANA_API_KEY" \ + --data-binary @oz-agent-worker-overview.json \ + http://your-grafana/api/dashboards/import +``` + +### Prometheus alerting rules + +The same `grafana/` directory includes [`alerts.yaml`](https://github.com/warpdotdev/oz-agent-worker/blob/main/grafana/alerts.yaml) — a set of Prometheus alerting rules covering the most common operational issues: + +* **OzWorkerFleetEmpty** (critical) — No workers connected for 5 minutes. +* **OzWorkerDisconnected** (warning) — A specific worker has been disconnected for 2 minutes. +* **OzWorkerReconnectStorm** (warning) — A worker is reconnecting at more than 0.1/s. +* **OzWorkerFleetSaturated** (warning) — Fleet concurrency utilization is above 90%. +* **OzWorkerTasksRejected** (warning) — Workers are actively rejecting tasks. +* **OzWorkerHighFailureRate** (warning) — More than 10% of completed tasks failed. +* **OzWorkerImagePullFailures** (warning) — Container image pulls are failing. +* **OzWorkerContainerOOM** (warning) — Task containers are being OOM-killed. +* **OzWorkerUnschedulablePods** (warning) — Kubernetes task pods cannot be scheduled. + +All thresholds are conservative starting points — tune them for your fleet size and SLOs. See [`grafana/README.md`](https://github.com/warpdotdev/oz-agent-worker/blob/main/grafana/README.md) for instructions on loading the rules into vanilla Prometheus or the Prometheus Operator. + +--- + ## Sample PromQL queries Direct mappings for common operational questions: