Add Temporal Self-Hosted Server observability plane

README.md

@@ -61,7 +61,8 @@ Repository: `git@github.com:technoidentity/temporal-sumologic-observability.git`
 |  |- import-checklist.md
 |  |- metric-to-panel-mapping.md
 |  |- worker-sdk/worker-metrics-dashboard.json
-|  +- temporal-cloud/temporal-cloud-metrics-dashboard.json
+|  |- temporal-cloud/temporal-cloud-metrics-dashboard.json
+|  +- temporal-self-hosted/temporal-server-metrics-dashboard.json
 +- iac/terraform/eks/
    |- versions.tf
    |- main.tf
@@ -190,6 +191,22 @@ Local endpoints:
 
 Scenario drivers are documented in `docs/dashboard-scenario-drivers.md`.
 
+### Deployment Combinations
+
+This repository provides three independently selectable observability paths:
+
+1. Worker SDK observability
+2. Temporal Cloud service observability
+3. Temporal Self-Hosted Server observability
+
+Deployments should combine them as needed:
+
+* **Temporal Cloud + Java worker**
+  -> Worker dashboard + Temporal Cloud dashboard
+
+* **Self-hosted Temporal + Java worker**
+  -> Worker dashboard + Temporal Self-Hosted Server dashboard
+
 ## Dashboards
 
 Importable dashboards:
@@ -198,6 +215,7 @@ Importable dashboards:
 |---|---|
 | Worker SDK metrics | `dashboards/sumo/worker-sdk/worker-metrics-dashboard.json` |
 | Temporal Cloud metrics | `dashboards/sumo/temporal-cloud/temporal-cloud-metrics-dashboard.json` |
+| Temporal Self-Hosted metrics | `dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json` |
 
 The worker dashboard uses a `Service` variable for SDK metrics. Default value:
 

dashboards/sumo/import-checklist.md

@@ -6,15 +6,17 @@
 |---|---|
 | Worker SDK metrics | `dashboards/sumo/worker-sdk/worker-metrics-dashboard.json` |
 | Temporal Cloud metrics | `dashboards/sumo/temporal-cloud/temporal-cloud-metrics-dashboard.json` |
+| Temporal Self-Hosted Server | `dashboards/sumo/temporal-self-hosted/temporal-server-metrics-dashboard.json` |
 
 ## Import Steps
 
 1. Import the JSON through Sumo Logic Library import.
 2. Set the worker dashboard `Service` variable to the real service dimension. The default is `temporal-java-sumo-observability`.
 3. Set `Namespace`, `Task Queue`, and `Workflow Type` variables to real low-cardinality values.
 4. For the Cloud dashboard, set `Cloud Service` to `temporal-cloud` unless the collector service label was intentionally changed.
-5. Run one workflow through the app before judging empty panels.
-6. Validate the exact labels in Sumo before locking the dashboard into a shared folder.
+5. For the Self-Hosted dashboard, set the `Service Role` (`service_name`: `frontend`/`history`/`matching`/`worker`, or `*`) and `Temporal Namespace` (`namespace`, or `*`) variables. Do not expect a `service` or `cluster` dimension — scraped server metrics do not carry one.
+6. Run one workflow through the app before judging empty panels.
+7. Validate the exact labels in Sumo before locking the dashboard into a shared folder.
 
 For API imports, post each dashboard JSON to:
 

dashboards/sumo/metric-to-panel-mapping.md

@@ -21,6 +21,37 @@
 | Temporal Cloud replication | `temporal_cloud_v1_replication_lag_p50`, `temporal_cloud_v1_replication_lag_p95`, `temporal_cloud_v1_replication_lag_p99` (without aggregation) |
 | Temporal Cloud actions | `temporal_cloud_v1_total_action_count` (shown as average actions/sec over the panel window; not a cumulative action total) |
 
+## Temporal Self-Hosted Server Dashboard
+
+Every metric family below is verified against Temporal's metric registry
+(`common/metrics/metric_defs.go`) and the official `temporalio/dashboards`
+server dashboards. Server roles are split by the native `service_name` tag
+(`frontend`, `history`, `matching`, `worker`) — the same tag Temporal's own
+server dashboards use. Do not filter self-hosted panels by a `service`
+dimension; scraped server metrics do not carry one.
+
+| Dashboard area | Primary metrics |
+|---|---|
+| Service health (by role) | `service_requests`, `service_errors`, `service_pending_requests` |
+| Service error ratio | `service_errors` / `service_requests` (formula) |
+| Service latency (avg) | `service_latency_sum` / `service_latency_count` (formula) |
+| Inter-service client health | `client_errors`, `client_requests`, `client_latency_bucket` |
+| Workflow & task traffic | `service_requests` by `operation`, `action`, `workflow_success`, `workflow_failed` |
+| Task timeouts | `schedule_to_start_timeout`, `start_to_close_timeout` (by `operation`) |
+| Server topology | `service_requests` split by `service_name` |
+| Matching / task queue | `approximate_backlog_count`, `approximate_backlog_age_seconds`, `no_poller_tasks`, `poll_success`, `poll_success_sync`, `poll_timeouts` |
+| Persistence | `persistence_requests`, `persistence_errors`, `persistence_errors_resource_exhausted`, `persistence_latency_sum`/`_count` |
+| History cache | `cache_size`, `cache_usage`, `cache_pinned_usage` by `cache_type` |
+| Server runtime | `restarts`, `num_goroutines`, `memory_heap` by `temporal_service_type` |
+| Kubernetes correlation | `kube_pod_container_status_restarts_total`, `container_cpu_usage_seconds_total`, `container_memory_working_set_bytes` |
+
+Latency uses the average form `rate(*_latency_sum)/rate(*_latency_count)`:
+Sumo has no `histogram_quantile`, so a true p95/p99 from `*_latency_bucket`
+must be validated separately. Counter suffixes are framework-dependent — the
+default Tally/Prometheus export uses bare names (`service_requests`), while the
+OpenTelemetry framework adds `_total`. Inventory the exact landed names in the
+Sumo account before locking filters.
+
 The Temporal Cloud dashboard metric names were compared against Temporal's published OpenMetrics Grafana dashboard. Remaining blank panels should be treated as scenario coverage gaps unless the validation queries above also return no matching metric family after the scenario has run.
 
 The Java SDK does not emit `temporal_num_pollers`. Any older dashboard panel that queried it has been removed. Current worker availability must be taken from Kubernetes readiness/replica health and corroborated with Temporal Cloud no-poller and backlog signals.

dashboards/sumo/temporal-self-hosted/README.md

@@ -0,0 +1,85 @@
+# Temporal Self-Hosted Server Metrics Dashboard
+
+This directory contains the Sumo Logic dashboard JSON for a self-hosted Temporal
+Server (`temporal-server-metrics-dashboard.json`).
+
+## Intended Usage
+
+This dashboard gives an operational view of the **Temporal Server** roles
+(Frontend, History, Matching, and internal Worker), their persistence tier, and
+server runtime. It is **not** a dashboard for your application workers — for
+application workers use the `worker-sdk` dashboard. Keep the two separate so
+server-role/persistence telemetry is not conflated with worker-side SDK
+telemetry.
+
+Sections:
+- **Service Health & Availability** — `service_requests`, `service_errors`, error
+  ratio, average `service_latency`, `service_pending_requests`, `client_errors`.
+- **Workflow & Task Traffic** — frontend `service_requests` by `operation`,
+  `action`, `workflow_success`/`workflow_failed`, `schedule_to_start_timeout` and
+  `start_to_close_timeout`.
+- **Server Topology** — `service_requests` split per role via `service_name`.
+- **Matching / Task Queue Health** — `approximate_backlog_count`,
+  `approximate_backlog_age_seconds`, `no_poller_tasks`, `poll_success`,
+  `poll_success_sync`, `poll_timeouts`.
+- **Persistence** — `persistence_requests`, `persistence_errors`, error ratio,
+  average `persistence_latency`, `persistence_errors_resource_exhausted`.
+- **History Cache & Server Runtime** — `cache_usage`/`cache_size` by `cache_type`,
+  `restarts`/`num_goroutines`/`memory_heap` by `temporal_service_type`.
+- **Infrastructure Correlation** — Kubernetes pod restarts, CPU, and memory for
+  Temporal pods (Sumo Kubernetes Collection).
+
+Every metric family is verified against Temporal's metric registry
+(`common/metrics/metric_defs.go`) and the official `temporalio/dashboards` server
+dashboards.
+
+## Dependencies
+
+The dashboard expects native Temporal Server metrics scraped in-cluster by the
+Sumo Kubernetes Collection from the server `/metrics` endpoint via
+`prometheus.io/*` pod annotations. A dedicated Cloud-style metrics collector is
+**not** required. See `sumo/temporal-self-hosted-collection-values.example.yaml`.
+
+Primary dimensions:
+- `service_name` — the native role tag (`frontend`, `history`, `matching`,
+  `worker`). This is what Temporal's own dashboards use; `service_role` also
+  exists but is not used here.
+- `operation` — RPC or persistence operation.
+- `namespace` — Temporal namespace.
+- `cache_type` — history cache family.
+- `temporal_service_type` — role tag on server runtime metrics (`restarts`, etc.).
+
+## Variables
+
+- `service_name` (Service Role) — default `*`; also `frontend`/`history`/`matching`/`worker`.
+- `namespace` (Temporal Namespace) — default `*`.
+
+There is intentionally **no** `service` variable: scraped server metrics do not
+carry a `service` dimension, so filtering by one would blank every panel.
+
+## Inventory-first caveat
+
+Exposed names are framework-dependent. The default Tally/Prometheus export uses
+bare names (`service_requests`); the OpenTelemetry framework adds `_total`. The
+Kubernetes namespace added by collection can also shadow Temporal's own
+`namespace` label as `exported_namespace`. Inventory the landed names before
+locking filters:
+
+```text
+metric=service_requests | count by service_name, operation
+```
+
+Latency panels use the average form `rate(*_latency_sum)/rate(*_latency_count)`
+because Sumo has no `histogram_quantile`; a true p95/p99 from `*_latency_bucket`
+must be validated separately.
+
+## Validation
+
+After importing, confirm metrics arrive from more than one `service_name`:
+
+```text
+metric=service_requests service_name=frontend
+metric=service_requests service_name=history
+```
+
+Full validation query set: `sumo/validation-queries.md`.