Chart the Paperclip runtime run-queue and retry-age gauges (BLO-23450)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Its runtime health is observable through paperclip_* gauges scraped into the cluster Prometheus and rendered by the operator Grafana (orc8r-user-grafana)
• BLO-22094 shipped paperclip_overdue_scheduled_retry_oldest_age_seconds to make an unpromoted parked retry visible, but left one acceptance criterion open: it must be readable next to paperclip_queued_run_oldest_age_seconds
• That pairing is the whole point — the queued gauge deliberately excludes time spent parked in scheduled_retry, so each number alone understates how long work has actually been stalled
• BLO-29904 then blocked that panel on the belief that no dashboard-as-code path existed. Investigating it showed the belief was false: the path shipped 2026-08-08 in feat(observability): ship the GitHub review-request delivery funnel dashboard (BLO-20171) #1182 and is live today
• This pull request charts both gauges side by side using that existing mechanism, and writes the runbook so the next person does not re-derive it
• The benefit is that the park window and the queue window can finally be read together, and that adding a Paperclip panel is now a reviewable PR rather than a request for a human with Grafana UI access

Linked Issues or Issue Description

• Refs BLO-29904 — the as-code path (premise corrected; no new mechanism added)
• Refs BLO-23450 — the panel this ships
• Refs BLO-22094 — the gauge, and the outstanding "side by side" criterion
• Related merged PR: feat(observability): ship the GitHub review-request delivery funnel dashboard (BLO-20171) #1182 (BLO-20171) built the dashboard-as-code mechanism this reuses. No open duplicate found.

What Changed

• dashboards/runtime-run-queue-health.json — new dashboard Paperclip runtime — run queue & retry health (uid: paperclip-runtime-queues), 5 panels: the two age gauges side by side, plus overdue-agent count, worst queued age, and a refresher-health tile.
• templates/grafana-dashboard-runtime.yaml — renders it as a grafana_dashboard="1" ConfigMap. A separate file from grafana-dashboard.yaml, deliberately (see below).
• tests/grafana-dashboard-runtime.test.mjs — 9 tests.
• runbooks/grafana-dashboard-as-code.md + index entry — the add-a-panel-by-PR flow, the traps, and the deploy gate.

Three choices worth a reviewer's attention:

• Separate template file, not a glob over dashboards/. A glob derives ConfigMap names from filenames, which would rename the live paperclip-grafana-dashboard-review-request-funnel ConfigMap — deleting and recreating a resource currently serving a dashboard, to save twelve lines. It also keeps --show-only templates/grafana-dashboard.yaml in the existing tests reading exactly the document they were written against.
• max by (agent_id), never sum(). Both gauges are emitted by every control-plane replica with identical values (measured 2026-09-02: 3 pods across service=paperclip ×2 and service=paperclip-workers ×1, 46 agent series each). sum() would report 3× the true age and still look plausible. A test pins this.
• A refresher-health panel on min(paperclip_overdue_scheduled_retry_age_metrics_refresh_success). If the refresh loop dies the gauges serve their last value forever, and a frozen gauge is pixel-identical to a healthy one on every other panel.

Verification

helm template paperclip deploy/helm/paperclip --namespace paperclip \
  -f deploy/helm/paperclip/values.blockcast.yaml \
  --show-only templates/grafana-dashboard-runtime.yaml

node --test deploy/helm/paperclip/tests/grafana-dashboard-runtime.test.mjs   # 9 pass
node --test deploy/helm/paperclip/tests/grafana-dashboard.test.mjs           # 11 pass, unchanged

• Render produces valid JSON, grafana_dashboard: "1" present, __PAPERCLIP_DS_UID__ substituted to cluster.
• Series confirmed live against the cluster Prometheus: paperclip_queued_run_oldest_age_seconds 139 series, max 922.137s; paperclip_overdue_scheduled_retry_oldest_age_seconds 138 series; ..._refresh_success = 1 on all 3 replicas.
• The overdue gauge currently reads all-zero. That is the healthy steady state — its contract is an explicit 0 rather than an absent series — not "no data".
• No screenshot: rendering the live panel needs Grafana UI access, which is exactly the dependency this work removes. The ConfigMap and the resolving series are the evidence.

Risks

Low. Additive only — one new ConfigMap; no existing resource is renamed, modified, or deleted.

• The new ConfigMap is adopted by the same sidecar already serving the funnel dashboard; a malformed dashboard would fail to render in Grafana without affecting the other.
• Both uid (paperclip-runtime-queues) and ConfigMap name are new — no collision with anything live.
• Worst realistic case is a panel that shows nothing, which is visible on sight and revertable by dropping one file.
• Deploy note, not a risk of this diff: dashboards in this chart do not sync through Argo — they ride the manual paperclip-api deploy. Merged is not live. The runbook documents the check.

Model Used

Claude Opus 5 (claude-opus-5), 1M context, extended thinking, with tool use — Paperclip MCP, read-only Kubernetes, Prometheus, and GitHub. Cluster facts in this PR were measured directly rather than recalled.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — n/a; the Grafana render needs UI access this work is removing the need for
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green — not yet; will confirm before requesting merge
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not yet reviewed
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-23450
🔗 Paperclip issue: BLO-29904
🔗 Paperclip issue: BLO-22094
🔗 Paperclip issue: BLO-20171

[allyblockcast]

🔗 Paperclip issue: BLO-23450
🔗 Paperclip issue: BLO-29904
🔗 Paperclip issue: BLO-22094
🔗 Paperclip issue: BLO-20171

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: a7427e2

Strong PR. I re-verified the load-bearing claims against the live cluster Prometheus rather than taking the description's word for them, and they hold:

claim	independent check
all three charted metrics exist	paperclip_queued_run_oldest_age_seconds 138 series, paperclip_overdue_scheduled_retry_oldest_age_seconds 138, ..._refresh_success 3
agent_id label is real	46 distinct agent_id values on both age gauges
replica multiplication is real, so max by not sum is correct	92 series on service=paperclip + 46 on service=paperclip-workers; refresh gauge resolves on exactly 3 pods (paperclip-0, paperclip-api-…-h2xml, paperclip-api-…-thtlp)
the stat-panel PromQL actually evaluates	count(max by (agent_id) (…) > 0) or vector(0) → 0
every grafanaDashboard.* values key referenced by the template exists	enabled/sidecarLabel/sidecarLabelValue/datasourceUid/folder/extraLabels all present in values.yaml:480-492
the new test file is actually executed by CI	.github/workflows/pr.yml:292 globs deploy/helm/paperclip/tests/*.test.mjs
no uid collision today	existing dashboard is paperclip-review-request-funnel, new one paperclip-runtime-queues

Critical Issues (0)

None.

Important Issues (1)

• [gstack/review + native-codex] deploy/helm/paperclip/templates/grafana-dashboard-runtime.yaml:25 (and the same instruction at runbooks/grafana-dashboard-as-code.md:57) — the "add another dashboard" recipe tells the reader to change exactly two things (the ConfigMap name suffix and the .Files.Get path) and does not mention the dashboard's internal uid. Grafana's file provisioner keys dashboards by uid, not by filename or ConfigMap name, so a reader who follows this literally — copying runtime-run-queue-health.json as a starting point, which is the natural move given "Drop the JSON in dashboards/" — ships two dashboards claiming paperclip-runtime-queues. One silently overwrites or fails to provision the other; the ConfigMaps both apply cleanly and both kubectl get cm checks in the runbook pass.

  This is precisely the "deploys cleanly, renders wrong, fails nowhere" class the PR itself enumerates, and it is the one member of that class missing from the Traps list at runbooks/grafana-dashboard-as-code.md:62. It is also currently unguarded: no test in deploy/helm/paperclip/tests/ asserts anything about dashboard uids — the existing uid assertions in grafana-dashboard.test.mjs:113 and :338 are about the datasource uid, which is a different field.

  • Add uid to both copy-this-file instructions ("change three things"), add a bullet to the Traps section, and pin it with a test in the style already used here — render every dashboards/*.json through the chart and assert the set of .uid values has no duplicates. That test costs one render and makes the third dashboard safe rather than the second one lucky.

Suggestions (2)

• [native-codex] deploy/helm/paperclip/dashboards/runtime-run-queue-health.json:105 — the refresher-health panel is min(paperclip_overdue_scheduled_retry_age_metrics_refresh_success) with no or vector(0). Its sibling stat panel at :36 defends exactly this case. If the series disappears entirely — refresh loop removed, metric renamed, all three replicas down — min() returns empty and the panel reads "No data" rather than STALE, even though total absence is strictly worse than the 0 case the panel was built to catch. It does fall back to the base threshold colour (red), so it fails in the alarming direction rather than the reassuring one, which is why this is a suggestion and not a blocker. min(...) or vector(0) makes the worst case read as the worst case.

• [pr-review-toolkit: tests] deploy/helm/paperclip/dashboards/runtime-run-queue-health.json:70 — max(max by (agent_id) (paperclip_queued_run_oldest_age_seconds)) is exactly equivalent to max(paperclip_queued_run_oldest_age_seconds); the outer bare max() already collapses every label including agent_id, so the inner grouping is a no-op on this panel. Harmless as PromQL, but worth knowing that the "never summed across replicas" test at grafana-dashboard-runtime.test.mjs:446 is, for this one panel, matching a string that has no effect — the panel is correct for a different reason (the outer max) than the one the test asserts. The two timeseries panels are the cases where that test is doing real work.

Strengths

• The reasoning is in the tree, not just the PR body. Every test carries a comment naming the failure mode it prevents, and the template header explains the one-file-per-dashboard choice — including that folding into a glob would rename and therefore delete/recreate the live paperclip-grafana-dashboard-review-request-funnel ConfigMap. That is the kind of decision that is invisible and expensive to rediscover.
• The refresher-health panel exists at all. "A stale gauge is pixel-identical to a healthy one" is the correct instinct, and most dashboards never get it.
• The CRD-vs-ConfigMap test (:371) pins a non-obvious RBAC constraint — that a CRD-backed kind 403s the entire helm upgrade, not just its own resource — which is the same trap keeping templates/prometheusrule.yaml disabled.
• The template is a faithful structural copy of grafana-dashboard.yaml, so it inherits the reviewed label/annotation/.Files.Get handling rather than re-deriving it. editable: false on the new dashboard is the right call for an as-code artifact.
• The premise correction is worth more than the panel. "A code-search zero is not evidence of absence on a private repo" generalises well past this ticket, and the "merged is not live" note at the bottom of the runbook pre-empts the most likely follow-up confusion.

Recommended Action

1. No Critical issues — nothing blocking merge on correctness grounds.
2. Address the uid guidance gap this cycle (docs bullet in two places + a duplicate-uid render test). It is cheap now and is a silent production failure the second time someone uses this path.
3. Consider the two suggestions opportunistically.