Cloud Action Costs Dashboard by lainecsmith · Pull Request #81 · temporalio/dashboards

lainecsmith · 2026-07-23T15:24:43Z

What was changed

Adds a new community Grafana dashboard for Temporal Cloud, sourced entirely from the OpenMetrics endpoint, plus a companion alert rules file:

temporal_cloud_action_costs.json — dashboard with two sections:

Summary: 1-day / 7-day / month-to-date Action totals.
Action Spike Detection: current Action rate vs. the same time 24h and 7d ago, percent-change indicators, a breakdown by Workflow Type and Action Type, and a ranked table of top Workflow Types by Action volume.

temporal_cloud_action_costs_alerts.yaml — optional Grafana unified-alerting provisioning file with 2 rules (action rate spike vs 24h ago, vs 7d ago).

Scope is intentionally limited to Action volume — Temporal Cloud doesn't expose price or credit-ledger balance via metrics, so this dashboard doesn't show cost in dollars or track spend against a commitment. That's called out explicitly in the dashboard's intro panel, with a pointer to Billing Center / the Billing API for anyone who needs authoritative $ figures.

Why?

Two of the most common cost-support conversations with customers are (1) someone ships a change or a new workflow that quietly drives a lot more Actions than expected, and nobody notices until the bill, and (2) there's no proactive signal — you only find out by going and looking. This gives customers a self-serve starting point for both: a dashboard to see it, and alert rules so it can page/notify instead of requiring someone to be staring at Grafana when it happens.

Checklist

How was this tested:

Created a Temporal Cloud Service Account with the Metrics Read-Only role and generated an API key; verified the OpenMetrics endpoint directly with curl before touching Grafana.
Ran Prometheus in a homelab Docker Compose stack, added a scrape job against metrics.temporal.io with Bearer auth, and confirmed the temporal-cloud job showed UP in Prometheus's own Targets page.
Imported the dashboard into a homelab Grafana instance against that Prometheus data source; confirmed the temporal_namespace variable populates from label_values(), and confirmed (separately, via a direct endpoint query) that namespaces without recent Action traffic simply don't appear — expected behavior given OpenMetrics' sparse reporting, not a bug — and added a note about this to the dashboard's intro panel so it's not mistaken for one.
Verified the account has 85 namespaces total but only 2 have recent traffic; confirmed the dashboard's namespace variable correctly surfaces those 2 and that switching to "All" sums correctly against real data.
Sanity-checked the Actions MTD figure against the Cloud UI's own Usage Dashboard for the same namespace/timeframe — same order of magnitude (exact match not expected; different counting methodology, called out in the docs).

Any docs updates needed?
Updated cloud/README.md in this PR with a link to the new dashboard the alerts.
No docs.temporal.io changes made as part of this PR.

…al_cloud_action_costs.json

prathyushpv · 2026-07-23T17:54:44Z

+        "uid": "${datasource}",
+        "type": "prometheus"
+      },
+      "description": "Percent change in current action rate vs the same time yesterday. Large positive swings flag an unannounced usage spike. Paired with an alert rule of the same name in temporal_cloud_alert_rules.yaml.",


does this file temporal_cloud_alert_rules.yaml exist?

It does, except I renamed it and forgot to update here! Fixed, thanks!

prathyushpv · 2026-07-23T17:56:14Z

+        "sortBy": [
+          {
+            "desc": true,
+            "displayName": "Actions/sec"


Should it be Estimated Actions?

Sure, changed.

Copilot

Pull request overview

Adds a new community Grafana dashboard (plus optional unified-alerting provisioning) to help Temporal Cloud users monitor Action volume month-to-date and detect sudden Action-rate spikes using the OpenMetrics endpoint.

Changes:

Added a new Grafana dashboard for Action MTD totals and spike-detection breakdowns.
Added Grafana unified-alerting provisioning rules to alert on action-rate spikes vs 24h/7d baselines.
Updated cloud README to link the new dashboard and sample alerts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
cloud/temporal_cloud_action_costs.json	New Grafana dashboard for Action MTD totals, spike comparisons vs 24h/7d ago, and workflow/action-type breakdowns.
cloud/temporal_cloud_action_costs_alerts.yaml	New Grafana unified-alerting provisioning file with two spike-detection alert rules.
cloud/README.md	Adds links to the new dashboard and sample alert rules.

Suppressed comments (3)

cloud/temporal_cloud_action_costs.json:206

This panel description says "billable" actions, but the query uses temporal_cloud_v1_total_action_count (not ..._billable_action_count). Either switch the query to the billable metric or adjust the description to avoid implying it is billable.

      "description": "Estimated total billable actions over the last 7 days",

cloud/temporal_cloud_action_costs.json:614

This percent-change expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN in the Stat panel and making the paired alerting behavior noisy. Clamp the denominator to a tiny positive value to keep the expression finite.

          "expr": "(sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"}) - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d)) / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d) * 100",

cloud/temporal_cloud_action_costs_alerts.yaml:115

This alert expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN and making the alert noisy. Clamp the denominator to a tiny positive value to keep the evaluation finite.

                (sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"})
                - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d))
                / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d) * 100

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+          "showLineNumbers": false,
+          "showMiniMap": false
+        },
+        "content": "# Temporal Cloud \u2014 Actions Spent (MTD) & Spike Detection\n\nScope, on purpose: this dashboard answers two questions from OpenMetrics alone \u2014 **\"how many Actions have we used this month?\"** and **\"did something just start using way more Actions than usual?\"** It does not attempt to track spend against a commit or credit balance \u2014 Temporal Cloud doesn't expose price or credit-ledger balance via metrics, only Action volume (`temporal_cloud_v1_total_action_count`, `temporal_cloud_v1_billable_action_count`).\n\nSet the dashboard time range to **This month** for the MTD panels to read correctly.\n\n**Alerting**: the delta panels below (Action Rate \u0394 vs 24h/7d Ago) are paired with Grafana alert rules \u2014 see the companion `temporal_cloud_alert_rules.yaml` \u2014 so a spike notifies your team instead of only being visible to whoever happens to be looking at the dashboard.\n\n\n**Only seeing a few namespaces in the dropdown?** That's expected, not a bug. OpenMetrics uses sparse reporting - a namespace only appears in a given scrape if it generated at least one billable Action in that window. Idle/unused namespaces simply won't show up here, no matter how wide you set the time range, until they have real traffic.\n",


+        "uid": "${datasource}",
+        "type": "prometheus"
+      },
+      "description": "Estimated total billable actions over the last 1 day",


+            "type": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "(sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"}) - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 1d)) / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 1d) * 100",


+          "fields": [
+            "Value"
+          ],


+        "sortBy": [
+          {
+            "desc": true,
+            "displayName": "Estimated Actions/sec"


+                (sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"})
+                - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 1d))
+                / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 1d) * 100


+# temporal-cost to it. The rules intentionally don't hardcode a contact point, since that's specific to your team.
+# 
+# Rules included
+# Action rate spike vs 24h ago: Fires when current action rate is >50% above the same time yesterday, sustained for	10 min


Update README, add temporal_cloud_action_costs_alerts.yaml and tempor…

7994b5e

…al_cloud_action_costs.json

lainecsmith requested a review from a team as a code owner July 23, 2026 15:24

prathyushpv reviewed Jul 23, 2026

View reviewed changes

lainecsmith added 2 commits July 23, 2026 14:15

Update alert rule references in JSON descriptions

9b868dd

Update display name for Actions/sec to Estimated Actions/sec

1022866

lainecsmith requested review from Copilot and dustin-temporal July 31, 2026 14:02

Copilot started reviewing on behalf of lainecsmith July 31, 2026 14:02 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Navigation Menu

Uh oh!

Cloud Action Costs Dashboard - #81

Cloud Action Costs Dashboard#81
lainecsmith wants to merge 3 commits into
masterfrom
lainecsmith/cloud-action-costs

lainecsmith commented Jul 23, 2026

Uh oh!

prathyushpv Jul 23, 2026

Uh oh!

lainecsmith Jul 23, 2026

Uh oh!

prathyushpv Jul 23, 2026

Uh oh!

lainecsmith Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lainecsmith commented Jul 23, 2026

Uh oh!

prathyushpv Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

lainecsmith Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

prathyushpv Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

lainecsmith Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants