diff --git a/README.md b/README.md index 2cbd5cd..c399d0f 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,8 @@ Repository: `git@github.com:technoidentity/temporal-sumologic-observability.git` | |- import-checklist.md | |- metric-to-panel-mapping.md | |- worker-sdk/worker-metrics-dashboard.json -| +- temporal-cloud/temporal-cloud-metrics-dashboard.json +| |- temporal-cloud/temporal-cloud-metrics-dashboard.json +| +- temporal-self-hosted/temporal-server-metrics-dashboard.json +- iac/terraform/eks/ |- versions.tf |- main.tf @@ -190,6 +191,22 @@ Local endpoints: Scenario drivers are documented in `docs/dashboard-scenario-drivers.md`. +### Deployment Combinations + +This repository provides three independently selectable observability paths: + +1. Worker SDK observability +2. Temporal Cloud service observability +3. Temporal Self-Hosted Server observability + +Deployments should combine them as needed: + +* **Temporal Cloud + Java worker** + -> Worker dashboard + Temporal Cloud dashboard + +* **Self-hosted Temporal + Java worker** + -> Worker dashboard + Temporal Self-Hosted Server dashboard + ## Dashboards Importable dashboards: @@ -198,6 +215,7 @@ Importable dashboards: |---|---| | Worker SDK metrics | `dashboards/sumo/worker-sdk/worker-metrics-dashboard.json` | | Temporal Cloud metrics | `dashboards/sumo/temporal-cloud/temporal-cloud-metrics-dashboard.json` | +| Temporal Self-Hosted metrics | `dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json` | The worker dashboard uses a `Service` variable for SDK metrics. Default value: diff --git a/dashboards/sumo/import-checklist.md b/dashboards/sumo/import-checklist.md index 6da6872..9381d03 100644 --- a/dashboards/sumo/import-checklist.md +++ b/dashboards/sumo/import-checklist.md @@ -6,6 +6,7 @@ |---|---| | Worker SDK metrics | `dashboards/sumo/worker-sdk/worker-metrics-dashboard.json` | | Temporal Cloud metrics | `dashboards/sumo/temporal-cloud/temporal-cloud-metrics-dashboard.json` | +| Temporal Self-Hosted Server | `dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json` | ## Import Steps @@ -13,8 +14,9 @@ 2. Set the worker dashboard `Service` variable to the real service dimension. The default is `temporal-java-sumo-observability`. 3. Set `Namespace`, `Task Queue`, and `Workflow Type` variables to real low-cardinality values. 4. For the Cloud dashboard, set `Cloud Service` to `temporal-cloud` unless the collector service label was intentionally changed. -5. Run one workflow through the app before judging empty panels. -6. Validate the exact labels in Sumo before locking the dashboard into a shared folder. +5. For the Self-Hosted dashboard, set the `Service Role` (`service_name`: `frontend`/`history`/`matching`/`worker`, or `*`) and `Temporal Namespace` (`namespace`, or `*`) variables. Do not expect a `service` or `cluster` dimension — scraped server metrics do not carry one. +6. Run one workflow through the app before judging empty panels. +7. Validate the exact labels in Sumo before locking the dashboard into a shared folder. For API imports, post each dashboard JSON to: diff --git a/dashboards/sumo/metric-to-panel-mapping.md b/dashboards/sumo/metric-to-panel-mapping.md index 25a91b0..39abf5c 100644 --- a/dashboards/sumo/metric-to-panel-mapping.md +++ b/dashboards/sumo/metric-to-panel-mapping.md @@ -21,6 +21,37 @@ | Temporal Cloud replication | `temporal_cloud_v1_replication_lag_p50`, `temporal_cloud_v1_replication_lag_p95`, `temporal_cloud_v1_replication_lag_p99` (without aggregation) | | Temporal Cloud actions | `temporal_cloud_v1_total_action_count` (shown as average actions/sec over the panel window; not a cumulative action total) | +## Temporal Self-Hosted Server Dashboard + +Every metric family below is verified against Temporal's metric registry +(`common/metrics/metric_defs.go`) and the official `temporalio/dashboards` +server dashboards. Server roles are split by the native `service_name` tag +(`frontend`, `history`, `matching`, `worker`) — the same tag Temporal's own +server dashboards use. Do not filter self-hosted panels by a `service` +dimension; scraped server metrics do not carry one. + +| Dashboard area | Primary metrics | +|---|---| +| Service health (by role) | `service_requests`, `service_errors`, `service_pending_requests` | +| Service error ratio | `service_errors` / `service_requests` (formula) | +| Service latency (avg) | `service_latency_sum` / `service_latency_count` (formula) | +| Inter-service client health | `client_errors`, `client_requests`, `client_latency_bucket` | +| Workflow & task traffic | `service_requests` by `operation`, `action`, `workflow_success`, `workflow_failed` | +| Task timeouts | `schedule_to_start_timeout`, `start_to_close_timeout` (by `operation`) | +| Server topology | `service_requests` split by `service_name` | +| Matching / task queue | `approximate_backlog_count`, `approximate_backlog_age_seconds`, `no_poller_tasks`, `poll_success`, `poll_success_sync`, `poll_timeouts` | +| Persistence | `persistence_requests`, `persistence_errors`, `persistence_errors_resource_exhausted`, `persistence_latency_sum`/`_count` | +| History cache | `cache_size`, `cache_usage`, `cache_pinned_usage` by `cache_type` | +| Server runtime | `restarts`, `num_goroutines`, `memory_heap` by `temporal_service_type` | +| Kubernetes correlation | `kube_pod_container_status_restarts_total`, `container_cpu_usage_seconds_total`, `container_memory_working_set_bytes` | + +Latency uses the average form `rate(*_latency_sum)/rate(*_latency_count)`: +Sumo has no `histogram_quantile`, so a true p95/p99 from `*_latency_bucket` +must be validated separately. Counter suffixes are framework-dependent — the +default Tally/Prometheus export uses bare names (`service_requests`), while the +OpenTelemetry framework adds `_total`. Inventory the exact landed names in the +Sumo account before locking filters. + The Temporal Cloud dashboard metric names were compared against Temporal's published OpenMetrics Grafana dashboard. Remaining blank panels should be treated as scenario coverage gaps unless the validation queries above also return no matching metric family after the scenario has run. The Java SDK does not emit `temporal_num_pollers`. Any older dashboard panel that queried it has been removed. Current worker availability must be taken from Kubernetes readiness/replica health and corroborated with Temporal Cloud no-poller and backlog signals. diff --git a/dashboards/sumo/temporal-self-hosted/README.md b/dashboards/sumo/temporal-self-hosted/README.md new file mode 100644 index 0000000..e93cfa0 --- /dev/null +++ b/dashboards/sumo/temporal-self-hosted/README.md @@ -0,0 +1,85 @@ +# Temporal Self-Hosted Server Metrics Dashboard + +This directory contains the Sumo Logic dashboard JSON for a self-hosted Temporal +Server (`temporal-server-metrics-dashboard.json`). + +## Intended Usage + +This dashboard gives an operational view of the **Temporal Server** roles +(Frontend, History, Matching, and internal Worker), their persistence tier, and +server runtime. It is **not** a dashboard for your application workers — for +application workers use the `worker-sdk` dashboard. Keep the two separate so +server-role/persistence telemetry is not conflated with worker-side SDK +telemetry. + +Sections: +- **Service Health & Availability** — `service_requests`, `service_errors`, error + ratio, average `service_latency`, `service_pending_requests`, `client_errors`. +- **Workflow & Task Traffic** — frontend `service_requests` by `operation`, + `action`, `workflow_success`/`workflow_failed`, `schedule_to_start_timeout` and + `start_to_close_timeout`. +- **Server Topology** — `service_requests` split per role via `service_name`. +- **Matching / Task Queue Health** — `approximate_backlog_count`, + `approximate_backlog_age_seconds`, `no_poller_tasks`, `poll_success`, + `poll_success_sync`, `poll_timeouts`. +- **Persistence** — `persistence_requests`, `persistence_errors`, error ratio, + average `persistence_latency`, `persistence_errors_resource_exhausted`. +- **History Cache & Server Runtime** — `cache_usage`/`cache_size` by `cache_type`, + `restarts`/`num_goroutines`/`memory_heap` by `temporal_service_type`. +- **Infrastructure Correlation** — Kubernetes pod restarts, CPU, and memory for + Temporal pods (Sumo Kubernetes Collection). + +Every metric family is verified against Temporal's metric registry +(`common/metrics/metric_defs.go`) and the official `temporalio/dashboards` server +dashboards. + +## Dependencies + +The dashboard expects native Temporal Server metrics scraped in-cluster by the +Sumo Kubernetes Collection from the server `/metrics` endpoint via +`prometheus.io/*` pod annotations. A dedicated Cloud-style metrics collector is +**not** required. See `sumo/temporal-self-hosted-collection-values.example.yaml`. + +Primary dimensions: +- `service_name` — the native role tag (`frontend`, `history`, `matching`, + `worker`). This is what Temporal's own dashboards use; `service_role` also + exists but is not used here. +- `operation` — RPC or persistence operation. +- `namespace` — Temporal namespace. +- `cache_type` — history cache family. +- `temporal_service_type` — role tag on server runtime metrics (`restarts`, etc.). + +## Variables + +- `service_name` (Service Role) — default `*`; also `frontend`/`history`/`matching`/`worker`. +- `namespace` (Temporal Namespace) — default `*`. + +There is intentionally **no** `service` variable: scraped server metrics do not +carry a `service` dimension, so filtering by one would blank every panel. + +## Inventory-first caveat + +Exposed names are framework-dependent. The default Tally/Prometheus export uses +bare names (`service_requests`); the OpenTelemetry framework adds `_total`. The +Kubernetes namespace added by collection can also shadow Temporal's own +`namespace` label as `exported_namespace`. Inventory the landed names before +locking filters: + +```text +metric=service_requests | count by service_name, operation +``` + +Latency panels use the average form `rate(*_latency_sum)/rate(*_latency_count)` +because Sumo has no `histogram_quantile`; a true p95/p99 from `*_latency_bucket` +must be validated separately. + +## Validation + +After importing, confirm metrics arrive from more than one `service_name`: + +```text +metric=service_requests service_name=frontend +metric=service_requests service_name=history +``` + +Full validation query set: `sumo/validation-queries.md`. diff --git a/dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json b/dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json new file mode 100644 index 0000000..105cf38 --- /dev/null +++ b/dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json @@ -0,0 +1,1171 @@ +{ + "type": "DashboardV2SyncDefinition", + "name": "Temporal Self-Hosted Server Metrics", + "description": "Self-hosted Temporal Server observability: service-role health, workflow/task traffic, topology, matching/backlog, persistence, history cache, server runtime, and Kubernetes correlation. Metric families verified against Temporal metric_defs.go and the official server dashboards.", + "title": "Temporal Self-Hosted Server Metrics", + "theme": "Light", + "topologyLabelMap": { + "data": {} + }, + "refreshInterval": 60, + "timeRange": { + "type": "BeginBoundedTimeRange", + "from": { + "type": "RelativeTimeRangeBoundary", + "relativeTime": "-3h" + }, + "to": null + }, + "layout": { + "layoutType": "Grid", + "layoutStructures": [ + { + "key": "panel-0", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 0}" + }, + { + "key": "panel-1", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 2}" + }, + { + "key": "panel-2", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 2}" + }, + { + "key": "panel-3", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 2}" + }, + { + "key": "panel-4", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 10}" + }, + { + "key": "panel-5", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 10}" + }, + { + "key": "panel-6", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 10}" + }, + { + "key": "panel-7", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 18}" + }, + { + "key": "panel-8", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 0, \"y\": 20}" + }, + { + "key": "panel-9", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 12, \"y\": 20}" + }, + { + "key": "panel-10", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 0, \"y\": 28}" + }, + { + "key": "panel-11", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 6, \"y\": 28}" + }, + { + "key": "panel-12", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 12, \"y\": 28}" + }, + { + "key": "panel-13", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 18, \"y\": 28}" + }, + { + "key": "panel-14", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 36}" + }, + { + "key": "panel-15", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 0, \"y\": 38}" + }, + { + "key": "panel-16", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 6, \"y\": 38}" + }, + { + "key": "panel-17", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 12, \"y\": 38}" + }, + { + "key": "panel-18", + "structure": "{\"height\": 8, \"width\": 6, \"x\": 18, \"y\": 38}" + }, + { + "key": "panel-19", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 46}" + }, + { + "key": "panel-20", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 48}" + }, + { + "key": "panel-21", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 48}" + }, + { + "key": "panel-22", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 48}" + }, + { + "key": "panel-23", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 56}" + }, + { + "key": "panel-24", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 56}" + }, + { + "key": "panel-25", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 56}" + }, + { + "key": "panel-26", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 64}" + }, + { + "key": "panel-27", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 66}" + }, + { + "key": "panel-28", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 66}" + }, + { + "key": "panel-29", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 66}" + }, + { + "key": "panel-30", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 0, \"y\": 74}" + }, + { + "key": "panel-31", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 12, \"y\": 74}" + }, + { + "key": "panel-32", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 82}" + }, + { + "key": "panel-33", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 84}" + }, + { + "key": "panel-34", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 84}" + }, + { + "key": "panel-35", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 84}" + }, + { + "key": "panel-36", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 0, \"y\": 92}" + }, + { + "key": "panel-37", + "structure": "{\"height\": 8, \"width\": 12, \"x\": 12, \"y\": 92}" + }, + { + "key": "panel-38", + "structure": "{\"height\": 2, \"width\": 24, \"x\": 0, \"y\": 100}" + }, + { + "key": "panel-39", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 0, \"y\": 102}" + }, + { + "key": "panel-40", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 8, \"y\": 102}" + }, + { + "key": "panel-41", + "structure": "{\"height\": 8, \"width\": 8, \"x\": 16, \"y\": 102}" + } + ] + }, + "panels": [ + { + "id": null, + "key": "panel-0", + "title": "", + "panelType": "TextPanel", + "text": "## Service Health & Availability", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-1", + "title": "Service Requests by Role", + "description": "Frontend/History/Matching/Worker request volume (service_requests).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name={{service_name}} | sum by service_name", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-2", + "title": "Service Errors by Role", + "description": "Unexpected server errors (service_errors).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_errors service_name={{service_name}} | sum by service_name", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-3", + "title": "Service Error Ratio", + "description": "service_errors / service_requests. Apply a minimum-volume guard before alerting.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_errors service_name={{service_name}} | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "B", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name={{service_name}} | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "C", + "queryType": "Metrics", + "queryString": "#A / #B", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-4", + "title": "Service Latency (avg, ms)", + "description": "Average = rate(service_latency_sum)/rate(service_latency_count). True p95 needs histogram_quantile, unavailable in Sumo; validate buckets separately.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_latency_sum service_name={{service_name}} | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "B", + "queryType": "Metrics", + "queryString": "metric=service_latency_count service_name={{service_name}} | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "C", + "queryType": "Metrics", + "queryString": "#A / #B", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-5", + "title": "Service Pending Requests", + "description": "In-flight/queued requests inside a service (service_pending_requests gauge).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_pending_requests service_name={{service_name}} | avg by service_name", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-6", + "title": "Inter-service Client Errors", + "description": "Internal client call failures (client_errors) keyed by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=client_errors service_name={{service_name}} | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-7", + "title": "", + "panelType": "TextPanel", + "text": "## Workflow & Task Traffic", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-8", + "title": "Frontend Requests by Operation", + "description": "Frontend RPC mix (StartWorkflowExecution, SignalWorkflowExecution, RespondWorkflowTaskCompleted, ...).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name=frontend | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-9", + "title": "Actions", + "description": "Server action counter (action). Self-hosted has no SaaS action-limit/throttle telemetry.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=action service_name=frontend | rate", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-10", + "title": "Workflow Success", + "description": "Completed workflows (workflow_success).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=workflow_success | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-11", + "title": "Workflow Failed", + "description": "Failed workflows (workflow_failed).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=workflow_failed | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-12", + "title": "Schedule-to-Start Timeouts", + "description": "Task pickup timeouts (schedule_to_start_timeout) by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=schedule_to_start_timeout | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-13", + "title": "Start-to-Close Timeouts", + "description": "Execution timeouts (start_to_close_timeout) by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=start_to_close_timeout | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-14", + "title": "", + "panelType": "TextPanel", + "text": "## Server Topology / Subsystem Health", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-15", + "title": "Frontend Requests", + "description": "service_requests on the frontend role by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name=frontend | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-16", + "title": "History Requests", + "description": "service_requests on the history role by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name=history | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-17", + "title": "Matching Requests", + "description": "service_requests on the matching role by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name=matching | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-18", + "title": "Worker (internal) Requests", + "description": "service_requests on the internal worker role by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=service_requests service_name=worker | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-19", + "title": "", + "panelType": "TextPanel", + "text": "## Matching / Task Queue Health", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-20", + "title": "Task Queue Backlog", + "description": "Approximate backlog depth (approximate_backlog_count) by namespace.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=approximate_backlog_count namespace={{namespace}} | max by namespace", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-21", + "title": "Oldest Backlog Age (s)", + "description": "Age of oldest queued task (approximate_backlog_age_seconds).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=approximate_backlog_age_seconds namespace={{namespace}} | max by namespace", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-22", + "title": "No-Poller Tasks", + "description": "Tasks arriving with no recent poller (no_poller_tasks). Corroborate with worker/K8s health.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=no_poller_tasks namespace={{namespace}} | sum by namespace", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-23", + "title": "Poll Success", + "description": "Matching poll successes (poll_success).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=poll_success | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-24", + "title": "Sync Match (poll_success_sync)", + "description": "Sync-matched polls (poll_success_sync); low share vs poll_success implies backlog.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=poll_success_sync | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-25", + "title": "Poll Timeouts", + "description": "Long-poll timeouts (poll_timeouts).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=poll_timeouts | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-26", + "title": "", + "panelType": "TextPanel", + "text": "## Persistence", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-27", + "title": "Persistence Requests", + "description": "Persistence request volume (persistence_requests) by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=persistence_requests | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-28", + "title": "Persistence Errors", + "description": "Persistence errors (persistence_errors) by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=persistence_errors | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-29", + "title": "Persistence Error Ratio", + "description": "persistence_errors / persistence_requests.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=persistence_errors | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "B", + "queryType": "Metrics", + "queryString": "metric=persistence_requests | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "C", + "queryType": "Metrics", + "queryString": "#A / #B", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-30", + "title": "Persistence Latency (avg, ms)", + "description": "rate(persistence_latency_sum)/rate(persistence_latency_count) by operation.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=persistence_latency_sum | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "B", + "queryType": "Metrics", + "queryString": "metric=persistence_latency_count | sum by operation", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + }, + { + "queryKey": "C", + "queryType": "Metrics", + "queryString": "#A / #B", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-31", + "title": "Persistence Resource Exhausted", + "description": "Capacity/resource-exhaustion persistence errors (persistence_errors_resource_exhausted).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=persistence_errors_resource_exhausted | sum", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-32", + "title": "", + "panelType": "TextPanel", + "text": "## History Cache & Server Runtime", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-33", + "title": "History Cache Usage", + "description": "cache_usage by cache_type (validate cache_type values for the release).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=cache_usage | max by cache_type", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-34", + "title": "History Cache Size", + "description": "cache_size by cache_type.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=cache_size | max by cache_type", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-35", + "title": "Server Restarts", + "description": "Process restarts (restarts) by temporal_service_type; corroborate with K8s restart/OOM.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=restarts | sum by temporal_service_type", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-36", + "title": "Goroutines", + "description": "num_goroutines by temporal_service_type.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=num_goroutines | avg by temporal_service_type", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-37", + "title": "Heap Memory", + "description": "memory_heap by temporal_service_type; compare against pod memory limits.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"line\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=memory_heap | avg by temporal_service_type", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-38", + "title": "", + "panelType": "TextPanel", + "text": "## Infrastructure Correlation (Kubernetes)", + "description": "", + "keepVisualSettingsConsistentWithParent": true + }, + { + "id": null, + "key": "panel-39", + "title": "Pod Restarts (temporal*)", + "description": "kube_pod_container_status_restarts_total for Temporal pods (Sumo K8s Collection).", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"column\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=kube_pod_container_status_restarts_total pod=*temporal* | max by pod", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-40", + "title": "CPU Usage (temporal*)", + "description": "container_cpu_usage_seconds_total rate by pod.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=container_cpu_usage_seconds_total pod=*temporal* | rate | sum by pod", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + }, + { + "id": null, + "key": "panel-41", + "title": "Memory Working Set (temporal*)", + "description": "container_memory_working_set_bytes by pod.", + "visualSettings": "{\"general\": {\"mode\": \"timeSeries\", \"type\": \"area\", \"displayType\": \"stacked\"}, \"series\": {}}", + "keepVisualSettingsConsistentWithParent": true, + "panelType": "SumoSearchPanel", + "queries": [ + { + "queryKey": "A", + "queryType": "Metrics", + "queryString": "metric=container_memory_working_set_bytes pod=*temporal* | max by pod", + "metricsQueryMode": "Advanced", + "metricsQueryData": null, + "tracesQueryData": null, + "parseMode": "Auto", + "timeSource": "Message" + } + ], + "timeRange": null, + "coloringRules": [] + } + ], + "variables": [ + { + "name": "service_name", + "displayName": "Service Role", + "defaultValue": "*", + "sourceDefinition": { + "variableSourceType": "CsvVariableSourceDefinition", + "values": "*,frontend,history,matching,worker" + }, + "allowMultiSelect": false, + "includeAllOption": true, + "hideFromUI": false + }, + { + "name": "namespace", + "displayName": "Temporal Namespace", + "defaultValue": "*", + "sourceDefinition": { + "variableSourceType": "MetadataVariableSourceDefinition", + "filter": "metric=service_requests", + "key": "namespace" + }, + "allowMultiSelect": false, + "includeAllOption": true, + "hideFromUI": false + } + ], + "filters": [], + "coloringRules": [] +} diff --git a/sumo/metric-field-mapping.md b/sumo/metric-field-mapping.md index 8c99d66..12d1d4c 100644 --- a/sumo/metric-field-mapping.md +++ b/sumo/metric-field-mapping.md @@ -57,9 +57,53 @@ For the EKS path, the Cloud scrape collector adds: | `service` | `temporal-cloud` | | `temporal_namespace` | Temporal Cloud namespace requested from `metrics.temporal.io`. | +## Temporal Self-Hosted Server Metrics + +These are native Temporal **server** metrics, emitted by the Frontend, History, +Matching, and internal Worker roles at the server `/metrics` endpoint. They are +**not** prefixed with `temporal_server_` (that prefix does not exist). Every name +below is verified against Temporal's metric registry +(`common/metrics/metric_defs.go`) and the official `temporalio/dashboards` server +dashboards. + +| Metric | Why it matters | +|---|---| +| `service_requests` | Server RPC request volume by `service_name`/`operation`. Request denominator. | +| `service_errors` / `service_error_with_type` | Server errors (untyped / typed via `error_type`). Build ratios against `service_requests`. | +| `service_latency` (`_bucket`/`_sum`/`_count`) | Server RPC latency histogram. Average = `rate(_sum)/rate(_count)`. | +| `service_pending_requests` | In-flight/pending request pressure (gauge). | +| `client_requests` / `client_errors` / `client_latency` | Internal inter-service client calls, keyed by role/operation. | +| `approximate_backlog_count` | Matching task-queue backlog depth. Direct backlog signal. | +| `approximate_backlog_age_seconds` | Age of the oldest queued task. Queue-delay SLOs. | +| `no_poller_tasks` | Tasks arriving with no recent poller. Corroborate with worker/K8s health. | +| `poll_success` / `poll_success_sync` / `poll_timeouts` | Matching poll and sync-match behavior. | +| `persistence_requests` | Persistence request volume by `operation`. | +| `persistence_errors` / `persistence_error_with_type` | Persistence errors (untyped / typed). | +| `persistence_errors_resource_exhausted` | Capacity/resource-exhaustion persistence errors, separated from general errors. | +| `persistence_latency` (`_bucket`/`_sum`/`_count`) | Persistence latency histogram. | +| `cache_size` / `cache_usage` / `cache_pinned_usage` | History cache families, grouped by `cache_type`. | +| `restarts` | Server process restart counter (by `temporal_service_type`). | +| `num_goroutines` / `memory_heap` / `memory_heapinuse` | Go runtime pressure; compare against pod CPU/memory limits. | +| `action` | Server action counter. Self-hosted has no SaaS action-limit/throttle telemetry. | +| `schedule_to_start_timeout` / `start_to_close_timeout` | Task pickup / execution timeout counters (by `operation`). | + +Recommended dimensions: `service_name` (`frontend`/`history`/`matching`/`worker`), +`operation`, `namespace`, `cache_type`, `temporal_service_type`. Temporal's native +role tag is `service_name`; `service_role` also exists but Temporal's own +dashboards group by `service_name`. Do **not** rely on an invented `temporal_role` +tag. + +Collection and naming caveats for the EKS path: + +| Concern | Note | +|---|---| +| No dedicated collector | Server metrics are scraped in-cluster by the Sumo Kubernetes Collection via `prometheus.io/*` pod annotations; there is no Cloud-style collector and no synthetic `service=temporal-self-hosted` dimension. | +| Counter suffixes | Tally/Prometheus export uses bare names (`service_requests`); the OpenTelemetry framework adds `_total`. Inventory the landed names first. | +| Namespace label collision | Temporal's own `namespace` label can collide with the Kubernetes namespace added by collection. Confirm whether the Temporal namespace lands as `namespace` or `exported_namespace` before filtering. | + ## Callout -Do not collapse worker metrics and Temporal Cloud metrics into one dashboard. They answer different questions: +Do not collapse worker metrics, Temporal Cloud metrics, and Temporal Self-Hosted metrics into one dashboard. They answer different questions: - Worker metrics show application-side poller, slot, workflow, activity, and SDK request health. -- Temporal Cloud metrics show service-side namespace/task-queue behavior exposed by Temporal Cloud. +- Temporal Cloud and Self-Hosted metrics show service-side namespace/task-queue behavior exposed by the server. diff --git a/sumo/temporal-self-hosted-collection-values.example.yaml b/sumo/temporal-self-hosted-collection-values.example.yaml new file mode 100644 index 0000000..5a90fe9 --- /dev/null +++ b/sumo/temporal-self-hosted-collection-values.example.yaml @@ -0,0 +1,73 @@ +# Example Sumo Logic Kubernetes Collection values for a Self-Hosted Temporal Server. +# +# Keep this file credential-free. Pass real credentials from your shell, CI +# secret store, or a private values file that is not committed. +# +# Recommended path (per the guide): let the Sumo Logic Kubernetes Collection +# discover Temporal Server pods through their prometheus.io/* annotations. The +# official Temporal Helm chart already exposes a Prometheus listener and enables +# metrics annotations by default, so a dedicated Cloud-style metrics collector is +# NOT required for the server metrics path. +# +# Docs: +# https://www.sumologic.com/help/docs/send-data/kubernetes/collecting-metrics/ +# https://github.com/temporalio/helm-charts/blob/main/charts/temporal/values.yaml + +sumologic: + accessId: "${SUMO_ACCESS_ID}" + accessKey: "${SUMO_ACCESS_KEY}" + clusterName: "replace-with-eks-cluster-name" + + logs: + enabled: true # Temporal Server container stdout/stderr (membership, persistence, requests) + + metrics: + enabled: true + collector: + otelcol: + # Scrape frequently; Temporal Server emits at high frequency. + scrapeInterval: 15s + +# --------------------------------------------------------------------------- +# Companion Temporal chart values (apply to the TEMPORAL Helm release, not this +# Sumo release). Makes the metrics listener and annotation behavior explicit and +# release-controlled even where the chart already defaults to equivalent values. +# +# server: +# config: +# metrics: +# prometheus: +# listenAddress: "0.0.0.0:9090" +# handlerPath: "/metrics" +# # framework: tally (bare metric names) OR opentelemetry (adds _total) +# metrics: +# annotations: +# enabled: true # emits prometheus.io/scrape, prometheus.io/port, prometheus.io/path +# --------------------------------------------------------------------------- + +# OPTIONAL fallback: an explicit scrape job, only if annotation discovery is not +# used in your environment. honor_labels: true PRESERVES Temporal's native +# labels (service_name, operation, namespace) so they are not overwritten by the +# Kubernetes metadata below. Kubernetes metadata is mapped to distinct label +# names to avoid colliding with Temporal's own `namespace` label. +prometheus: + prometheusSpec: + additionalScrapeConfigs: + - job_name: 'temporal-server' + honor_labels: true + metrics_path: '/metrics' + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] + action: keep + regex: temporal + - source_labels: [__meta_kubernetes_namespace] + action: replace + target_label: kubernetes_namespace + - source_labels: [__meta_kubernetes_pod_name] + action: replace + target_label: pod + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component] + action: replace + target_label: kubernetes_component diff --git a/sumo/validation-queries.md b/sumo/validation-queries.md index f820774..25968a2 100644 --- a/sumo/validation-queries.md +++ b/sumo/validation-queries.md @@ -131,3 +131,76 @@ metric=temporal_cloud_v1_schedule_action_success_count service=temporal-cloud te ``` Blank panels are expected until the matching scenario occurs. Examples: schedule overrun, missed catchup, and rate-limited panels require those schedule conditions, cancellation panels require a workflow that is actually canceled, timeout panels require exported timeout counters, resource exhausted panels require throttling, and replication lag panels require a replicated namespace. + +## Temporal Self-Hosted Server Metrics + +Use these to validate a self-hosted Temporal Server (e.g. Helm chart). Server +metrics are scraped in-cluster by the Sumo Kubernetes Collection from the +Temporal `/metrics` endpoint via `prometheus.io/*` pod annotations — there is no +dedicated Cloud-style collector. + +**Inventory first.** Before locking dashboard filters, confirm the exact landed +metric and tag names, because the exposed names are framework-dependent (Tally +export uses bare names like `service_requests`; the OpenTelemetry framework adds +`_total`), and Kubernetes metadata added by collection can shadow Temporal's own +`namespace` label as `exported_namespace`: + +```text +metric=service_requests | count by service_name, operation +``` + +```text +metric=persistence_* | count by metric, operation +``` + +Service health and topology (roles split by the native `service_name` tag): + +```text +metric=service_requests service_name=frontend +metric=service_requests service_name=history +metric=service_requests service_name=matching +metric=service_requests service_name=worker +``` + +```text +metric=service_errors | sum by service_name, operation +``` + +Matching / task-queue backlog and poller health: + +```text +metric=approximate_backlog_count | max by namespace +metric=approximate_backlog_age_seconds | max by namespace +metric=no_poller_tasks | sum by namespace +metric=poll_success OR metric=poll_success_sync +``` + +Persistence and history cache: + +```text +metric=persistence_requests | sum by operation +metric=persistence_errors | sum by operation +metric=persistence_errors_resource_exhausted +metric=cache_usage | max by cache_type +``` + +Server runtime: + +```text +metric=restarts | sum by temporal_service_type +metric=num_goroutines | avg by temporal_service_type +``` + +A practical smoke gate should verify: + +- `StartWorkflowExecution` produces frontend `service_requests` +- workflow-task and activity-task traffic appear as `service_requests` by `operation` +- `persistence_requests` is non-zero +- server metrics are visible from more than one `service_name` +- latency panels resolve via `service_latency_sum` / `service_latency_count` + +Blank panels are expected for scenario-driven families (backlog, no-poller, +resource-exhausted, timeouts) until the matching condition occurs. A blank panel +whose family also returns nothing in the inventory queries above indicates a +naming/collection gap rather than a scenario gap — adapt the filter to the landed +name.