Give the CIP-104 reward automation a health signal by gyorgybalazsi · Pull Request #325 · DLC-link/decentralization-manager

gyorgybalazsi · 2026-08-13T13:37:18Z

What this PR does

Old behavior

decman runs a background loop that reassigns CIP-104 reward coupons for a decentralized party. Holding that party's credentials only puts it in the loop's list: the node reassigns nothing unless an on-ledger CouponReassignmentDelegation exists for that party and names this node's member party as an assigner. Nothing reported on that loop. If it panicked, returned, or hung, the process stayed up and /healthz kept answering 200 while coupons expired unassigned. A coupon past its expiry is unrecoverable value, so the failure that costs the most was the one nothing could see.

New behavior

The node exposes eight Prometheus metrics on a dedicated port, and a monitoring system can now answer three questions it could not before: is the loop alive, are coupons approaching expiry unassigned, and are sweeps failing. Every long-lived background task also reports its own death, whether it panics or simply returns.

Two of those answers are now trustworthy in cases where they were not. A sweep that assigns nothing counts as such however it ended, not only when the ledger refused each coupon one by one. And the expiry countdown refreshes on its own clock, so a long sweep interval no longer makes the countdown that stale.

The change that enables it

Ten instruments live beside the loop that moves them, registered at boot so a family exists before its first event.
The loop beats every 60 seconds and sweeps a decparty when its configured interval has elapsed. The heartbeat shares the task with the sweep, so a hung sweep stops the beat and becomes visible.
The expiry gauge is recomputed on every heartbeat from a timestamp the loop remembers, so it keeps falling between sweeps, during a stall, and after a failed sweep.
A second clock runs a read-only pass between sweeps, refreshing that gauge without assigning anything.
Two counters report a decparty this node cannot authenticate for and is a named assigner for. The delegation is the enablement signal, so a decparty nobody enabled moves nothing.
GET /metrics is served by its own listener on its own port, because the Ingress forwards every path to the API port.
spawn_supervised wraps every long-lived task in start_server and names what the operator loses if one dies.

Caveats

This ships no alert rules. They are dlc-infra#192, and they must land after this deploys or they evaluate an empty series.
A counter's first increment after a restart is invisible to increase, because no scrape observed the prior value. Every rule reading an increase under-counts by one per restart. decman-reward-automation-stalled catches a crash-looping node through pod absence instead.
A decparty with no delegation at all clears the gauge and raises nothing. That is the deliberate off switch, and mainnet is in it today. Telling it apart from a decparty nobody enabled by mistake needs configuration saying Mode A is expected — #334.
The read-only pass has no integration coverage. The localnet harness sets the sweep to 15 seconds, so the read clock never binds and that path never executes in CI. Its clock arbitration has unit tests; the read itself does not.
Scraping needs pod annotations that live in dlc-infra. The deployment guide documents them for an outside operator; BitSafe's own pod templates take the same three additions separately.
New dependency: prometheus 0.13, default features off. Nothing in the tree produced text exposition, which is what the collector scrapes. attestor-stack runs the same crate at the same major version with the same feature setting.
A delegation naming this node for a decparty with no party_credentials row is not covered. The loop walks existing rows, so it never visits that decparty and nothing counts. Accepted rather than filed.

Details

What changed

reward_automation.rs — the ten instruments and register_metrics; the heartbeat loop; the pure helpers is_due, due_pass, seconds_until, oldest_expiry and sweep_assigned_nothing; and apply_sweep_outcome, whose arms are deliberately asymmetrical. A backlog writes the timestamp, an empty backlog removes it and its gauge series, and every failure leaves it alone. That last arm is the point: clearing the countdown on a failure would switch off the only rule that survives a run of failing sweeps.
run_reassign_once and run_once_for_party now report what they saw. run_reassign_once returns the oldest unassigned expiresAt that has not passed, taken before the drain. run_once_for_party returns the sweep's outcome, and takes a Pass saying whether to assign or only to read.
read_oldest_expiry is the read half of run_reassign_once, used by the read-only pass.
A credentials failure gets its own outcome rather than being treated as an empty backlog. A Keycloak failure while a backlog drains would otherwise delete the gauge series and silence the at-risk alert at exactly the moment value is about to be lost. It does not increment the failure counter, so a node whose auth has not loaded cannot trip the sweep-failing alert.
The sweep timer takes the smallest of the heartbeat, the sweep interval and the read interval, so an interval below 60 seconds is no longer silently floored.
config.rs, cli.rs, main.rs — metrics_port, default 9464, 0 serves no metrics. A failed bind logs an error and the node serves on: losing a signal is not a reason to stop serving governance. A metrics_port that collides with the API or Noise port yields rather than binding, for the same reason. Also reward_expiry_read_interval_secs, default 3600, set by DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS, which now warns on 0 as the sweep interval already did.
README.md, docs/DEPLOYMENT_GUIDE.md — the four DECPM_REWARD_* variables, none of which were documented. DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS matters most, because it bounds how stale the expiry gauge gets and both expiry rules read that gauge. The deployment guide also now tells an outside operator that the metrics listener answers any caller who reaches it, so port 9464 has to stay inside the cluster.
The end-to-end harness gives each test node its own metrics port, on both targets.
docs/CONTRIBUTING.md, docs/DEPLOYMENT_GUIDE.md, README.md — the endpoint, the port, the scrape annotations and both configuration tables.

Design decisions worth knowing

The stall rule's window is 15 minutes, not 10. Rolling the three devnet nodes took the 10-minute heartbeat bucket to 2, 3 and 4, against a threshold of 1. The rule would not have fired, but one sample is not a margin. absentFor takes the same number so a deploy cannot trip one half while sparing the other. Detection goes from about 15 minutes to about 20, against a 36h coupon TTL.
The heartbeat shares a task with the sweep on purpose. A hung sweep therefore stops the beat, which is the only way a hang becomes visible — no panic occurs and nothing returns.
The sweep interval is a transaction-cost knob, and detection should not inherit it. One Delegation_Assign carries reward_max_creates / beneficiary_count coupons, so a longer interval fills a bigger batch and buys fewer transactions. Devnet runs 6 hours for that reason. But the gauge counted down from a timestamp only a sweep refreshed, so a node could not see that another assigner had already taken the coupon it was counting. The two clocks separate the two concerns. The gauge refreshes at whichever interval is shorter, so mainnet's 300-second sweep sees no change.
A read that fails leaves the sweep counters alone. Counting it would let a problem on the signal path fire the sweep-failure alert.
The zero-assigned counter changed meaning, and devnet measured why. It read assigned == 0 && skipped > 0, so it only counted a sweep whose coupons the ledger refused individually. The contention path ends a sweep by break, leaving both tallies at zero, which read as a sweep with nothing to do. On 2026-08-15 devnet spent 30 hours in exactly that state: sweeps ran on their normal cadence and assigned nothing, every failure counter stayed flat, and the expiry gauge fell to 533 seconds before the backlog cleared. The counter now reads assignable > 0 && assigned == 0, so any sweep that found coupons and assigned none of them counts, whatever ended it.
The expiry gauge reports the oldest coupon seen unassigned at the last read. In health a sweep assigns the coupon it just reported, so the value is briefly stale, but it never falls far enough to be read as risk. In every failure mode it is accurate or conservative.
The gauge ignores coupons that have already expired, and this was measured rather than assumed. A RewardCouponV2 does not leave the ledger at expiresAt — it is signatory dso with no self-archiving choice, and the DSO's reaper skips batches below vetted amulet 0.1.19. Devnet held 408,584 active coupons already past expiry on 2026-08-13, the oldest by 35 days, against 12,042 ever archived. Taking the minimum over all of them would pin the gauge to the oldest dead coupon and fire both expiry rules forever — the metric failing in exactly the case it exists to detect. So it reports the oldest coupon still worth saving. A coupon inside the assignability margin still counts, because it is about to be lost.
.workers(1) on the metrics listener. actix gives each HttpServer a worker per logical CPU by default, which is waste for one route answered every five minutes.

Verification

DECMAN_SKIP_FRONTEND=1 cargo test -p decman: 431 lib, 5 binary, 44 integration (2 ignored), all passing. The baseline before this branch was 389 lib.
cargo fmt --all and DECMAN_SKIP_FRONTEND=1 cargo clippy -p decman --all-targets -- -D warnings both clean. No #[allow] anywhere.
New unit tests cover every pure decision: the sweep gate on a fresh process and at the interval boundary, the countdown's sign and sub-second precision, the earliest-of-several, empty, all-expired and expired-ignored cases, the zero-assigned classifier, and the three outcome arms — including that a failure and a credentials failure both leave the countdown running.
Five tests cover the two clocks: the first tick sweeps, a due sweep beats a due read, a read runs between sweeps, neither due runs nothing, and a read interval longer than the sweep never binds.
One test drives drain_assignable through the contention path and asserts the zero-assigned counter moved. It uses its own decparty fixture, because these tests share a process-wide registry and run in parallel.
The /metrics handler is tested for 200, the text content type, and the heartbeat family in the body.
Both auth arms have a test that drives run_once_for_party and asserts the outcome: the registry arm through an AppState with no auth, the token arm through a Keycloak stand-in that answers once and then fails. The counting decision is a pure counts_as_auth_fault, tested for an enabled decparty, an unenabled one, an unread set, and every non-auth outcome. Removing the gate fails two of those, checked by removing it. The AppState test builder moved out of governance.rs's test module to an AppState::for_test beside the struct, so both modules share one copy.
The localnet reassignment phase, which runs on every CI run, now reads /metrics after a real sweep and asserts the assigned and skipped counters moved. It sums across both listed assigners, because which one wins a round is a race, and it never compares the skipped counter for equality, because nothing quarantines the deliberately-unassignable seeded coupon.

Verified on devnet through live alert rules, 2026-08-19

Devnet nodes 1, 2 and 4 ran this branch at acbc3b2. The SigNoz rules from the companion dlc-infra
change read these metrics. Every fault below was injected, not waited for.

decman_reward_heartbeat_total drives the stall alert correctly. Scaling node-2 to zero
replicas made decman-reward-automation-stalled fire 17.5 minutes later. The alert carried the
right k8s_namespace_name label. Restoring the node resolved it 43 seconds after the pod went
ready. It then stayed inactive for 24 minutes, so it does not flap.
A deploy's counter reset does not raise a false alarm. A fleet-wide roll took the heartbeat from
1388 to 0. increase absorbed the reset as a dip to 1, 2 and 3, never a negative. The stall alert
stayed inactive. The 15-minute window was widened for exactly this case, and it holds.
decman_reward_sweep_failed_total drives the sweep-failure alert. Pointing node-4's
DECPM_CANTON_LEDGER_HOST at an unroutable name failed every sweep. The counter moved. The rule
then fired at three visible failures, once its own matchType was corrected in dlc-infra. That
correction fixes the alert, not the metric. The rule had required its condition to hold in every
bucket of a 30-minute window, which no episodic failure does.
decman_reward_oldest_unassigned_expires_in_seconds tracked a real stall and a real recovery.
Assignment stopped for about two hours, while the nodes ran a build without these metrics. On
restore the gauge read 121,386 seconds. That is 33.7 hours of runway on a coupon 2.3 hours old, and
it fired decman-reward-coupons-stopped-draining as a true positive. Node-1 then logged
reassigned coupon batch … count: 1. The gauge jumped to 129,320 seconds within the minute.
CredentialsUnavailable keeps the countdown falling, as intended. apply_sweep_outcome treats
it like a failure and keeps the remembered expiry. The served list comes from the configured
credential entries, not from the tokens that succeeded. So a token outage leaves the decparty
tracked, and the gauge still crosses both expiry thresholds. It is not a silent path.

The wrong-assigner path is exercised deliberately. A governance vote replaced
the three assigners with one party the fleet does not hold. Every node then entered ReportOnly.
Three results came out of it.

role_for and the ReportOnly arm work. All three nodes logged the warning within one sweep. The
expiry gauge went from one series to three, because a non-assigner now reads and publishes the
backlog instead of clearing it. It then fell at 1.0000 second per second across 15 monotonic
samples, so the countdown tracked the stall exactly. On the code before this change the same vote
returned Empty from all three nodes. That dropped the series and raised nothing at all, while the
coupons aged.

Both expiry thresholds fire at their configured values, and the escalation gap is right. The warning
fired 55 minutes after assignment stopped, and the critical 11 hours 30 minutes after the warning.
The design estimates "about eleven and a half hours" between them, so the two thresholds are spaced
correctly against a 36-hour TTL.

The countdown's own arithmetic is predictable. Firing times were predicted in advance from
(threshold_age − age_at_fault) + evalWindow and both landed within one evaluation cycle of the
prediction.

decman_reward_coupons_skipped_total is driven by a real fault. It cannot be
forced on demand, because it only moves when an individual coupon submit fails non-transiently, and
contention is classified transient. It does occur naturally: the ledger refused a coupon on node-4 at
14:10Z and decman-reward-coupon-rejected fired at 14:22:10Z with the right decparty label. Devnet
saw two such events in the preceding seven days, the earlier one 25/25/3 across all three nodes during
a sequencer wobble.

The localnet harness also drives this counter on every CI run. It seeds a coupon below splice's
transfer threshold, which the ledger admits and then refuses to assign, so the drain's fan-out runs for
real and the counter is asserted.

A misclassification the same signal exposes. The error was
MEDIATOR_SAYS_TX_TIMED_OUT, which was not in TRANSIENT_ASSIGN_ERROR_IDS, so the drain read a
mediator timeout as a bad coupon. The same coupon assigned cleanly on the next sweep. Four costs
followed: the log line blamed the coupon, the skipped counter overstated real rejections, the alert
fired on a self-healing condition, and the drain kept submitting into a mediator that was already
timing out rather than ending the sweep. The allowlist already carries four sibling timeouts on exactly
this reasoning, so this one joins them.

This is worth knowing about the allowlist's shape. assign_failure_is_transient treats any unlisted
error id as attributable to the coupon, so each new Canton error id arrives misclassified until someone
adds it. Devnet found this one in a day. Mainnet would find it as a page.

Notes

The countdown must be recomputed on the heartbeat rather than the sweep. A gauge written once per sweep freezes when sweeps stop producing a value, so the two expiry rules would stay silent for the coupons' whole 36-hour life — exactly when they are needed.
Metric names are fixed. The alert rules in dlc-infra are written against these exact strings. The zero-assigned counter keeps its name and gains a new help string.
sweep_is_due is now is_due, and all_coupons_refused is now sweep_assigned_nothing. Both names described behavior the functions no longer have.

Design: cip-104/docs/superpowers/specs/2026-08-11-cip104-reward-automation-observability-design.md · Follows decman #316 (JSON logs) · Companion change in dlc-infra: the five alert rules are committed and pushed to devnet SigNoz (5e118d7, d858c94); the log pipeline, pod annotations and dashboard panels still follow

A stalled reward-reassignment loop or coupons drifting toward expiry are currently invisible: nothing surfaces them until a decparty's rewards have already lapsed unclaimed. This adds the seven Prometheus instruments (heartbeat, sweep/assign/skip/fail counters, oldest-unassigned-expiry gauge) that later tasks wire into the loop and expose on a metrics endpoint, plus the startup call that registers them so they exist from boot rather than only once a decparty is first swept.

Record three Prometheus counters where reward drain already counts: assigned coupons, skipped coupons, and sweeps where all coupons were refused. Introduces all_coupons_refused helper and three tests. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

Adds metrics_port field to NodeConfig with default 9464, supporting override via DECPM_METRICS_PORT env var or --metrics-port CLI flag. Task 5 will bind the /metrics listener to this port.

The reward automation now emits Prometheus counters, but nothing scrapes them: decman had no /metrics route. Bind a second, single- worker HttpServer on NodeConfig::metrics_port, separate from the API server so the collector's endpoint isn't gated behind the same auth and CORS surface as the rest of the app. Gives each of the three e2e nodes its own metrics port (localnet and devnet) so Task 6's chaos-phase assertions have a port to read, and so three dec-party-manager processes on one host don't collide on the Prometheus default.

Every other test in this plan is a unit test over a pure helper. None proves that a counter moves when a real sweep assigns a real coupon on the wire, so a label built from the wrong value or a counter wired to the wrong code path would ship undetected. Carry each node's metrics port into the fixture, add a text GET for the unauthenticated /metrics route, and sum decman_reward_coupons_assigned_total / _skipped_total across nodes 1 and 2 (either can win the assign race) after the existing localnet split assertion. assigned is compared with >=, and skipped is never compared with ==, since nothing quarantines the deliberately-unassignable seeded coupon and every sweep on the short e2e interval re-counts it.

The outer task awaits the inner task's JoinHandle and logs any failure. This catches two cases the panic hook in main cannot: - a clean return from the infinite loop (equally broken, equally silent) - task-specific context: this log says *which* task died, not just that the process panicked Nothing respawns. The loop never returns, so a crash loop would hammer the ledger and hide the bug. Logging and stopping lets the responder fix it. Design §6, change 6, explains what this closes.

"Tick" collided with tokio's own timer API once the loop grew a distinct heartbeat and reward cadence (Tasks 1-7): a reader could no longer tell "reassignment cycle" from "60s wake" apart from context. Settling on sweep for the cycle and heartbeat for the wake removes the ambiguity everywhere the words appear — log lines, doc comments, and test names — while leaving tokio's tick()/MissedTickBehavior alone, since those name the timer API rather than either concept. Comment and identifier renames only; no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Adds documentation for the Prometheus metrics endpoint at /metrics on DECPM_METRICS_PORT (default 9464), separate from the API port. Includes Kubernetes manifest snippets, pod-template annotations and labels for Prometheus scrape configuration, and environment-variable reference table entries in both DEPLOYMENT_GUIDE and README. Also updates the devnet test harness comment to document the per-participant metrics ports (9464/9465/9466). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The loop's heartbeat timer was hardcoded to 60s, so sweep_is_due could only ever see 60s-sized steps: any configured interval below that was silently floored, and the localnet e2e's 15s interval quietly ran at 60s. Build the timer from HEARTBEAT_INTERVAL.min(sweep_interval) instead, so a short interval beats at its own rate; beating faster is safe because the stall alert's `increase < 1 over 10m` is a lower bound. Two in-tree comments assumed the fast path already worked and are corrected to say why it now does. A Keycloak token failure returned through get_party_credentials was indistinguishable from "no active delegation" or "not a listed assigner" — all three cleared the expiry gauge as if the automation were deliberately off. That silences decman-reward-coupons-at-risk at the exact moment a backlog is draining and auth breaks. Give the credentials arm its own SweepOutcome so it holds the countdown steady like a failed sweep, without counting toward decman_reward_sweep_failed_total (a node with credentials but no loaded token yet would otherwise trip decman-reward-sweep-failing on every tick). get_party_credentials itself is untouched — its signature change is out of scope here. Also: rename the loop's local oldest_expiry to oldest (it shadowed the module-level fn of the same name), drop a test comment that only restated sweep_is_due's own doc, and note in-place that the heartbeat and sweep share one task (so a blocking sweep costs heartbeats) and that last_sweep records wake time rather than tick deadline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Once assigned reached the seeded count, the probe hard-failed if skipped was still 0. That's reachable, not just theoretical: the drain's fan-out can break 'chunks on a transient error, and if that lands on the unassignable coupon's isolated submission after the healthy coupons already committed, the sweep ends with assigned complete and skipped at 0. Several TRANSIENT_ASSIGN_ERROR_IDS are plausible on a loaded CI box. Log and retry within the scenario's deadline instead — nothing quarantines that coupon, so the next sweep re-finds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The metrics HttpServer binds before the API server, so a metrics_port that matches the API or noise port makes the node die reporting the wrong listener as the cause. A startup warning names the collision; it stays a warning, not a fatal check, since losing the metrics signal is not a reason to stop serving governance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Metrics line claimed the listed ports match DECPM_METRICS_PORT in the per-participant .env files; nothing puts them there. It was copied from the Noise entry above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Devnet PQS shows 408,584 of 429,529 active RewardCouponV2 coupons already past expiresAt (oldest 35 days), because the DSO's reaper does not archive them promptly. oldest_expiry took the minimum over every unassigned coupon including expired ones, so once a decparty lost coupons the metric pinned to the oldest corpse and both expiry alerts fired forever instead of tracking the backlog still worth saving.

socket-security · 2026-08-13T13:38:01Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/prometheus@0.13.4

View full report

Copilot

Pull request overview

Adds Prometheus-based health signaling for the CIP-104 reward automation so monitoring can detect stalls, approaching-expiry unassigned coupons, and sweep failures, including cases where the task returns or panics while the process stays up.

Changes:

Introduces reward-automation Prometheus metrics (heartbeat, sweep/coupon counters, and oldest-expiry gauge) and serves /metrics on a dedicated configurable port.
Refactors the reward automation loop into a heartbeat + due-sweep model and adds a supervisor task that logs if the automation task returns or dies.
Updates integration tests and documentation to configure and validate the new metrics endpoint.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
README.md	Documents `DECPM_METRICS_PORT` configuration.
integration-tests/smoke-noise-errors.sh	Passes per-node metrics ports when starting nodes.
integration-tests/run.sh	Exports metrics port env vars for e2e phases.
integration-tests/env.sh	Defines default per-node metrics ports.
integration-tests/devnet.env.sh	Adds devnet metrics port documentation and exports.
integration-tests/common.sh	Includes metrics ports in port-availability checks and node startup.
docs/DEPLOYMENT_GUIDE.md	Documents scrape annotations, container port, and env var for metrics.
docs/CONTRIBUTING.md	Adds contributor guidance for metrics instrumentation conventions.
crates/decman/tests/common/processes.rs	Propagates metrics port into spawned node processes.
crates/decman/tests/common/phases/seed_reward_coupons.rs	Updates terminology from tick → sweep in comments.
crates/decman/tests/common/phases/coupon_reassignment.rs	Adds end-to-end assertions by reading `/metrics` and updates sweep terminology.
crates/decman/tests/common/mod.rs	Extends fixture port configuration to include metrics ports.
crates/decman/tests/common/http.rs	Adds a helper to GET plain text responses (used for `/metrics`).
crates/decman/src/server/reward_automation.rs	Implements metrics, heartbeat/sweep loop changes, and outcome tracking for gauge updates.
crates/decman/src/server/mod.rs	Registers metrics at boot, starts a dedicated metrics listener, and supervises the automation task.
crates/decman/src/server/handlers/mod.rs	Re-exports the new `metrics` handler.
crates/decman/src/server/handlers/config.rs	Implements `/metrics` handler and tests Prometheus text exposition.
crates/decman/src/main.rs	Wires `metrics_port` CLI/env into runtime config.
crates/decman/src/config.rs	Adds `metrics_port` to `NodeConfig` with default 9464.
crates/decman/src/cli.rs	Adds `--metrics-port` / `DECPM_METRICS_PORT` CLI/env flag.
crates/decman/Cargo.toml	Adds `prometheus` dependency to decman crate.
Cargo.toml	Adds workspace `prometheus` dependency configuration.
Cargo.lock	Locks the new `prometheus` dependency version.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The fix wave made the loop wake on `HEARTBEAT_INTERVAL.min(sweep_interval)` so a configured interval below 60s is no longer floored. The metric's help text still claimed one heartbeat per minute, which is wrong for any such interval — the e2e suite runs at 15s. Production is unaffected: 300s and 21600s both floor to 60. The metric name is untouched, so no alert rule changes. Reported by Copilot on #325. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

gyorgybalazsi · 2026-08-13T14:10:21Z

Copilot review loop — disposition summary

Round	Commits	Findings	Fixed	Won't-fix
1	`8beb33c…c2a3809`	2	1	1

By category

Category	Findings	Fixed	Won't-fix
Functionality	1	0	1
Cosmetic	1	1	0

Fixed — the heartbeat metric's HELP string still said "one per minute". This branch's own fix wave made the loop wake on HEARTBEAT_INTERVAL.min(sweep_interval), so the text is wrong for any interval below 60s, and the e2e suite runs at 15s. Corrected in c2a3809; the metric name is untouched.

Won't fix — prune_unserved's linear contains is quadratic in principle. Both collections hold one entry per decparty this node serves, which is single digits, so it is about a hundred string comparisons once per sweep. A HashSet built for ten elements would cost more than the scan.

Outcome: converged in one round. CI is green on 8beb33c, including the 28-minute integration suite that exercises the new metrics ports and the counter assertion for the first time.

A counter that only appears on its first event leaves a dashboard unable to tell a node that never failed from an instrument that is missing. Touching the four outcome counters where the sweep counter is already incremented creates each child at zero, so the panels draw a flat line rather than a no-data marker. No alert changes: every rule tests an increase of at least one, which a series sitting at zero never satisfies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…-metrics

The zero-assigned counter keyed on the refusal tally, so it only counted a sweep whose coupons the ledger refused one by one. The contention path ends a sweep with both tallies at zero, which read as a sweep that had nothing to do. Devnet spent 30 hours in that state on 2026-08-15. Sweeps ran on their normal cadence and assigned nothing, every failure counter stayed flat, and the expiry gauge fell to 533 seconds before the backlog cleared. The alert this counter feeds would not have fired. The counter now keys on the backlog it was given, so any sweep that found coupons and assigned none of them counts, whatever ended it.

…-metrics

The sweep interval sizes a Delegation_Assign batch, so it is a transaction-cost choice. Detection inherited it: the gauge counts down from a timestamp only a sweep refreshes, so at devnet's 6h cadence a node cannot see that another assigner already took the coupon it is counting. A second clock now drives a read-only pass between sweeps. It reads the backlog and refreshes the gauge without assigning anything. A sweep reads the ledger anyway, so it satisfies the read clock too, and the gauge refreshes at whichever interval is shorter. Mainnet's 300s sweep therefore sees no change. DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS sets it, defaulting to 3600. A read that fails leaves the sweep counters alone, so a problem on the signal path cannot fire the sweep-failure alert.

A participant with no rewards DAR cannot host a delegation, so the automation is off for that node. The loop still counted it: the error arm incremented the sweep-failure counter twelve times an hour while #335's trace line kept the log quiet. That is what blocks the sweep-failing alert. The new Off outcome counts nothing, logs at trace and touches nothing. It is deliberately not Empty: Empty drops the gauge series, which asserts that nothing is left to save. A node that held the DAR and lost it would then go silent instead of keeping its countdown. Robert settled this on 2026-08-17, so the opt-in config field the plan proposed is no longer needed to silence these nodes.

Three conflicts, all two-sided additions rather than disagreements. #340 added a debug line to the no-assignable-coupons return in run_reassign_once, which this branch had changed to return the oldest expiry. Kept both: the log and the returned timestamp. #337 added probe_get_json beside this branch's get_text, and probe_reward_coupons beside counter_total. Kept all four.

…-metrics # Conflicts: # crates/decman/src/server/handlers/mod.rs

MEDIATOR_SAYS_TX_TIMED_OUT was not in the transient allowlist, so a mediator that did not gather confirmations in time was read as a bad coupon. Observed on devnet on 2026-08-20. The drain isolated one coupon, logged it at ERROR, counted it skipped, and fired the coupon-rejected alert. The same coupon assigned cleanly on the next sweep, so nothing was wrong with it. Four costs followed from the misclassification. The log line blamed the coupon. The skipped counter overstated real rejections. The alert fired on a condition that heals itself. And the drain carried on submitting into a mediator that was already timing out, instead of ending the sweep to re-read as it does for every other timeout. The list already carries four sibling timeouts on exactly this reasoning: LOCAL_VERDICT_TIMEOUT, NOT_SEQUENCED_TIMEOUT, SEQUENCER_BACKPRESSURE and REQUEST_TIME_OUT. This one belongs with them. The new test pins the reason rather than the string, and records the gRPC code the error actually arrived under. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…-metrics # Conflicts: # crates/decman/src/server/reward_automation.rs

get_party_credentials now separates a token fetch that failed from a decparty the auth registry does not know. Both were treated as one sweep's transient problem, which keeps the remembered expiry so the countdown carries on falling. That is right for a failed token fetch and wrong for the other case. The registry is rebuilt whenever party config changes, so a decparty that read a backlog earlier can stop resolving to credentials and stay that way. Its countdown then falls to zero on a reading nothing can refresh, and both expiry rules fire for a decparty this node no longer serves. prune_unserved cannot catch it: that keys on the party_credentials rows, which still exist. SweepOutcome::Unserved drops the timestamp and its gauge series, the way an empty backlog does. The two are kept apart because they assert different things: an empty backlog says nothing is left to save, while this says only that the backlog is unreadable here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

schronck

Read the whole diff plus the branch source. The evidence here is unusually good, the devnet fault injection covers the paths that matter, and the two-clock arbitration and the outcome arms hold up. Two I checked closely and think are right: oldest_expiry excluding an already-expired coupon, and the assignable > 0 && assigned == 0 rewrite against the contention break.

One defect I'd want fixed before this merges, on the Unserved arm. Details inline. Short version: a decparty whose auth init failed is still in party_credentials but missing from the registry, so it hits Ok(None) on every sweep forever, loses its gauge series, and increments nothing.

Rest is smaller:

The Noise inbound listener in run_heartbeat (crates/decman/src/server/mod.rs:1354) is still a detached spawn. spawn_supervised("heartbeat", ...) wraps the outer fn, which awaits run_peer_ping_loop. The accept loop can still die quietly. Fine as a follow-up, but the caveat in the description says the heartbeat gained supervision, and that's half of it.
DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS isn't in the README or the deployment guide. It sets how stale the expiry gauge can get, and both expiry rules read that gauge, so it's the one reward knob an operator actually needs. DECPM_METRICS_PORT got both tables.
The deployment guide is written for outside node operators, and this opens a third port on their host, unauthenticated on 0.0.0.0. Worth a line telling them to firewall 9464 or set it to 0.
counter_total in the reassignment phase matches on starts_with(name), so a longer family sharing the prefix would sum in. Nothing collides today.

AuthRegistry::new skips a party whose token manager fails to initialise and logs a warn, but the party_credentials row survives. The reward loop iterates those rows, so a decparty with a bad Keycloak secret resolved to no credentials on every pass and dropped its expiry gauge series. Its coupons expired with both expiry rules silent and no counter moving. prune_unserved already forgets a decparty whose row is gone, so this arm can only mean "row present, registry entry missing" — a fault, not an off switch. The countdown now stays, and a new counter carries the fault, because a decparty whose auth failed at boot never read a backlog and so has no series to fire on.

The two comments still cited a 10-minute window after the stalled rule moved to 15 minutes, so anyone sizing a change against that rule read the wrong number out of the code. reward_expiry_read_interval_secs also lacked the zero warning its sibling has. A 0 becomes 1s, which is a full ACS read per decparty per second.

The metrics listener bound first, so a metrics_port that collided with the API or noise port won the race and killed the other listener. It now yields the port and serves no metrics, which is the rule the bind-failure arm beside it already kept. The metrics server and the Noise inbound accept loop were also the two long-lived tasks in start_server that skipped spawn_supervised. A metrics server that dies leaves every alert rule reading an empty series while the node looks healthy, and a dead accept loop refuses every inbound peer in silence.

counter_total tested a bare prefix, so a longer family name starting with the one asked for would have summed in too.

None of the four DECPM_REWARD_* variables appeared in the README or the deployment guide. DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS matters most: it bounds how stale the expiry gauge gets, and both expiry alert rules read that gauge. The deployment guide also targets outside node operators, and it opened a third port on their host without saying that the listener answers any caller that reaches it.

The credentials arm keeps the expiry countdown, which is right, but it incremented nothing. A decparty whose Keycloak client is gone fails its token fetch on every pass and only writes one warn line, and a decparty broken from boot never read a backlog so it has no gauge series to fall either. Devnet measures the case: on 2026-08-25 nodes 1 and 2 hit this arm for 28 node/decparty pairs, on every pass, all of them stale test decparties. Nothing counted it.

gyorgybalazsi · 2026-08-25T09:13:14Z

All six items are in, plus one I found while measuring the blocker on devnet. The commits are e6f2d74, 1c9b3a6, aaeb841, 433564d, 4bd3b2f and 06922da. The inline threads carry the detail; this covers the four from your summary.

The Noise inbound listener. Wrapped in spawn_supervised rather than deferred (aaeb841). start_server now spawns no long-lived task any other way, and the PR description no longer needs the caveat.

DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS. In both tables now, along with the other three DECPM_REWARD_* variables — none of the four was documented (4bd3b2f). The expiry-read row says what it costs: it bounds how stale the gauge gets, and both expiry rules read that gauge.

Port 9464 on an outside operator's host. The deployment guide now says the listener answers any caller who reaches it, names the firewall and security-group options, and points at DECPM_METRICS_PORT=0 for an operator with no collector. The pod-template comment and the config table say it too, so nobody meets the port in only one place.

counter_total. Now matches the family exactly — the character after the name has to be { or a space (433564d).

The one you did not ask for. Measuring your finding on devnet turned up the same silence on the sibling arm. CredentialsUnavailable kept the countdown, which was right, but counted nothing, and it is the arm devnet takes: 28 node/decparty pairs on nodes 1 and 2 hit it on every pass, while your arm logged zero lines in 7 days. TokenManager::new authenticates when it is built, so a restart on this build moves all 28 onto your arm. Both arms now have a counter, one rule reads them both, and the rule fires on devnet the day this deploys. That last part is deliberate — Gyorgy chose it over gating the rule on cleaning those party_credentials rows, which are filed as a follow-up.

The rule is dlc-infra#192, already live on devnet SigNoz. The design doc gained the sixth condition, which also settles the §1-says-six-§7-says-five count you may have noticed.

Both increments sat on a path no test reached, so removing either one left the suite green. Each arm now has a case that drives run_once_for_party and reads the counter back: the registry arm through an AppState with no auth, the token arm through a Keycloak stand-in that answers once and then fails. Removing an increment fails its test. The AppState builder moves from governance's test module to an AppState::for_test beside the struct, so the two test modules share one copy and a new field breaks one place.

The two auth counters moved for any decparty this node could not authenticate for, whether or not that decparty ever expected Mode A. Devnet holds one active CouponReassignmentDelegation, for cbtc-network, so all 28 firing series were decparties with no delegation and nothing to lose. Every alert was a false positive. The delegation is the enablement signal, and it was unreachable here because reading it needed the token that had just failed. It does not: the template makes its assigners observers, so the node reads the delegations naming it under any working token and learns which decparties matter before deciding to count. An unread set counts nothing. `None` means the node could not ask, not that nothing is enabled, so guessing either way is worse than staying quiet until a read lands.

gyorgybalazsi · 2026-08-26T13:42:43Z

Update since the disposition summary above. One more commit, and it changes something I told you.

d53d9b6 — the two auth counters are now gated on enablement. They previously moved for any decparty this node could not authenticate for. That predicate was wrong, and Robert Tera caught it the first time the rule fired: devnet holds exactly one active CouponReassignmentDelegation (for cbtc-network, across all four package versions), so the rule named 28 decparties with no delegation and nothing to lose, while the one that mattered was healthy. 28 alerts, 0 findings.

A delegation naming this node is what turns a credentials failure into a loss. Reading it looks impossible on that path, because it needs the token that just failed — except CouponReassignmentDelegation declares observer assigners, so the node reads the delegations naming it under any working token it already holds. No new configuration, so this did not have to wait on #334.

The counting decision is now a pure counts_as_auth_fault(outcome, decparty, enabled) rather than a condition inside the loop, with four tests: an enabled decparty counts, an unenabled one does not, an unread set counts nothing, and no non-auth outcome counts. Removing the gate fails two of them, checked by removing it.

Verified on devnet. Nodes 1/2/4 rolled to d53d9b6: 0 firing series where the ungated build had 14/14/0, both counters at sum 0 across 51 sweeps, and node 1 still logging 14 no auth registered warnings per pass — the fault is still detected, just not counted where nothing is at stake. The alert has been silent for 19 hours.

One thing worth knowing if you review the deploy. A windowed counter rule keeps firing for its full window after the fix lands. The gated build reached devnet at 14:45 UTC and neither counter moved again, but the alert stayed lit until 15:07 because increase over 20 minutes still held pre-fix samples. I also re-enabled the rule inside that window and generated three avoidable Slack messages doing it. The check that answers the question is "has anything incremented since the deploy", read from the pod's /metrics rather than a range query wide enough to reach back past it.

Also unproven and knowingly accepted: a delegation naming this node for a decparty with no party_credentials row at all. The loop only walks existing rows, so it never visits that decparty and nothing counts. enabled - configured would detect it; Gyorgy decided not to implement or file it, and devnet is empty on it today.

…-metrics

Conflicts resolved in favor of the decman-lib architecture. Main's semantic changes are preserved: #364's DomainConfirmation DTO replaces the placeholder action, #370's field_record helper moved into decman-lib, #369's token-standard handler split stands, and #325's reward-automation health signal stands.

gyorgybalazsi and others added 14 commits August 13, 2026 12:49

Beat every 60s and count each sweep

a6361cc

Add a metrics port to NodeConfig

895943a

Adds metrics_port field to NodeConfig with default 9464, supporting override via DECPM_METRICS_PORT env var or --metrics-port CLI flag. Task 5 will bind the /metrics listener to this port.

gyorgybalazsi requested a lite review from Copilot August 13, 2026 13:46

Copilot started reviewing on behalf of gyorgybalazsi August 13, 2026 13:47 View session

Copilot AI reviewed Aug 13, 2026

View reviewed changes

Comment thread crates/decman/src/server/reward_automation.rs Outdated

Comment thread crates/decman/src/server/reward_automation.rs

gyorgybalazsi mentioned this pull request Aug 13, 2026

get_party_credentials swallows token errors and conflates three failures #326

Closed

This was referenced Aug 13, 2026

Devnet nodes 1 and 2: 14 decparties fail Keycloak M2M auth at startup #330

Closed

Reward automation warns every 5 minutes on any node whose participant lacks the rewards DAR #334

Open

Log an absent-package sweep failure at trace #335

Merged

gyorgybalazsi and others added 2 commits August 14, 2026 17:14

Merge remote-tracking branch 'origin/main' into feature/decman-reward…

b1613fe

…-metrics

schronck mentioned this pull request Aug 17, 2026

reward automation has no health signal: silent stop, failing assigns, coupons nearing expiry #278

Open

gyorgybalazsi added 3 commits August 17, 2026 12:32

Merge remote-tracking branch 'origin/main' into feature/decman-reward…

37f0298

…-metrics

gyorgybalazsi added 2 commits August 18, 2026 16:57

gyorgybalazsi mentioned this pull request Aug 18, 2026

No UI signal when a participant lacks the rewards DAR #356

Open

schronck mentioned this pull request Aug 19, 2026

fix(api): distinguished a token-fetch failure from an unconfigured party #360

Merged

gyorgybalazsi marked this pull request as ready for review August 19, 2026 13:50

gyorgybalazsi requested review from a team and sosaucily August 19, 2026 13:50

Merge remote-tracking branch 'origin/main' into feature/decman-reward…

b3ead80

…-metrics # Conflicts: # crates/decman/src/server/handlers/mod.rs

gyorgybalazsi requested a review from scolear August 19, 2026 14:44

gyorgybalazsi and others added 3 commits August 20, 2026 16:42

Merge remote-tracking branch 'origin/main' into feature/decman-reward…

b3bd141

…-metrics # Conflicts: # crates/decman/src/server/reward_automation.rs

schronck reviewed Aug 25, 2026

View reviewed changes

gyorgybalazsi added 6 commits August 25, 2026 10:46

Match the metric family exactly when summing it

433564d

counter_total tested a bare prefix, so a longer family name starting with the one asked for would have summed in too.

gyorgybalazsi added 2 commits August 25, 2026 11:22

schronck assigned gyorgybalazsi Aug 25, 2026

Merge remote-tracking branch 'origin/main' into feature/decman-reward…

5f0f684

…-metrics

scolear approved these changes Aug 26, 2026

View reviewed changes

scolear merged commit 78af4d7 into main Aug 27, 2026
10 checks passed

scolear deleted the feature/decman-reward-metrics branch August 27, 2026 14:50

hubagaspar91 mentioned this pull request Aug 27, 2026

feat(decman-lib): extract the governance domain model and gRPC codecs #382

Draft

11 tasks

Navigation Menu

Uh oh!

Give the CIP-104 reward automation a health signal - #325

Give the CIP-104 reward automation a health signal#325
scolear merged 40 commits into
mainfrom
feature/decman-reward-metrics

gyorgybalazsi commented Aug 13, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

gyorgybalazsi commented Aug 13, 2026

Uh oh!

schronck left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gyorgybalazsi commented Aug 25, 2026

Uh oh!

gyorgybalazsi commented Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

gyorgybalazsi commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Details

Uh oh!

socket-security Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

gyorgybalazsi commented Aug 13, 2026

Copilot review loop — disposition summary

Uh oh!

schronck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gyorgybalazsi commented Aug 25, 2026

Uh oh!

gyorgybalazsi commented Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gyorgybalazsi commented Aug 13, 2026 •

edited

Loading

socket-security Bot commented Aug 13, 2026 •

edited

Loading