Log an absent-package sweep failure at trace - #335
Merged
Conversation
Any node whose participant does not hold governance-rewards-automation-v1 logs this line once per decparty per sweep, and the interval defaults to 300 seconds. Devnet nodes 3, 5 and 6 have produced 864 lines a day between them, unbroken for at least 48 hours, because the delegation query cannot resolve the package name. This lowers the volume and fixes nothing. Issue #334 carries the cause and the options for removing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit lowered the whole statement to trace, which also silenced a genuine sweep failure. Only one cause is unactionable: the participant does not carry governance-rewards-automation-v1, so the delegation query is rejected with PACKAGE_NAMES_NOT_FOUND on every tick. Classify on that error id and lower only that case. Everything else -- auth, transport, a rejected assign -- stays at warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyorgybalazsi
changed the title
A participant can hold the DAR and still not use it. The name may resolve to a package without the template or the interface the read asks for, or the participant may hold the implementation and not have vetted it. Match the five read-path package error ids instead of one. Every error this loop catches comes from a read, because drain_assignable absorbs each submission error, so the submission-side vetting ids cannot reach this statement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyorgybalazsi
changed the title
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Classifies unusable-package sweep failures so routine failures are suppressed at default log levels.
Changes:
- Adds Canton package-error classification.
- Logs classified failures at
trace; retainswarnotherwise. - Adds classification tests.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The delegation read is the loop's first ledger call, so a participant without the DAR is rejected there with PACKAGE_NAMES_NOT_FOUND and the later reads never run. Canton resolves a read filter's package name from uploaded packages, not vetted ones, so an unvetted DAR raises nothing here either. The other four ids can only appear past that point, or not at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyorgybalazsi
changed the title
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scolear
approved these changes
Aug 17, 2026
gyorgybalazsi
added a commit
that referenced
this pull request
Aug 17, 2026
#335 lowered the absent-package failure to trace and landed on the same error arm this branch had just restructured for the two clocks. Kept both: the package-absent line stays at trace, and the warn arm below it now distinguishes a failed sweep from a failed expiry read. An absent package still increments the sweep-failure counter, which is what the blockers findings expect while #334 is open — the metric carries the signal the log no longer does. #335's wording says tick where this branch says sweep, so the merged lines take the sweep name the rename gives them.
gyorgybalazsi
added a commit
that referenced
this pull request
Aug 18, 2026
A participant with no rewards DAR cannot host a delegation, so the automation is off for that node. The loop still counted it: the error arm incremented the sweep-failure counter twelve times an hour while #335's trace line kept the log quiet. That is what blocks the sweep-failing alert. The new Off outcome counts nothing, logs at trace and touches nothing. It is deliberately not Empty: Empty drops the gauge series, which asserts that nothing is left to save. A node that held the DAR and lost it would then go silent instead of keeping its countdown. Robert settled this on 2026-08-17, so the opt-in config field the plan proposed is no longer needed to silence these nodes.
scolear
pushed a commit
that referenced
this pull request
Aug 27, 2026
* Add the reward automation's metric instruments A stalled reward-reassignment loop or coupons drifting toward expiry are currently invisible: nothing surfaces them until a decparty's rewards have already lapsed unclaimed. This adds the seven Prometheus instruments (heartbeat, sweep/assign/skip/fail counters, oldest-unassigned-expiry gauge) that later tasks wire into the loop and expose on a metrics endpoint, plus the startup call that registers them so they exist from boot rather than only once a decparty is first swept. * Beat every 60s and count each sweep * Count assigned, skipped and zero-assigned sweeps Record three Prometheus counters where reward drain already counts: assigned coupons, skipped coupons, and sweeps where all coupons were refused. Introduces all_coupons_refused helper and three tests. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add a metrics port to NodeConfig Adds metrics_port field to NodeConfig with default 9464, supporting override via DECPM_METRICS_PORT env var or --metrics-port CLI flag. Task 5 will bind the /metrics listener to this port. * Serve /metrics on its own port The reward automation now emits Prometheus counters, but nothing scrapes them: decman had no /metrics route. Bind a second, single- worker HttpServer on NodeConfig::metrics_port, separate from the API server so the collector's endpoint isn't gated behind the same auth and CORS surface as the rest of the app. Gives each of the three e2e nodes its own metrics port (localnet and devnet) so Task 6's chaos-phase assertions have a port to read, and so three dec-party-manager processes on one host don't collide on the Prometheus default. * Assert the reward counters in the e2e Every other test in this plan is a unit test over a pure helper. None proves that a counter moves when a real sweep assigns a real coupon on the wire, so a label built from the wrong value or a counter wired to the wrong code path would ship undetected. Carry each node's metrics port into the fixture, add a text GET for the unauthenticated /metrics route, and sum decman_reward_coupons_assigned_total / _skipped_total across nodes 1 and 2 (either can win the assign race) after the existing localnet split assertion. assigned is compared with >=, and skipped is never compared with ==, since nothing quarantines the deliberately-unassignable seeded coupon and every sweep on the short e2e interval re-counts it. * Report the reward task's death The outer task awaits the inner task's JoinHandle and logs any failure. This catches two cases the panic hook in main cannot: - a clean return from the infinite loop (equally broken, equally silent) - task-specific context: this log says *which* task died, not just that the process panicked Nothing respawns. The loop never returns, so a crash loop would hammer the ledger and hide the bug. Logging and stopping lets the responder fix it. Design §6, change 6, explains what this closes. * Call a reward cycle a sweep "Tick" collided with tokio's own timer API once the loop grew a distinct heartbeat and reward cadence (Tasks 1-7): a reader could no longer tell "reassignment cycle" from "60s wake" apart from context. Settling on sweep for the cycle and heartbeat for the wake removes the ambiguity everywhere the words appear — log lines, doc comments, and test names — while leaving tokio's tick()/MissedTickBehavior alone, since those name the timer API rather than either concept. Comment and identifier renames only; no behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Document the metrics endpoint Adds documentation for the Prometheus metrics endpoint at /metrics on DECPM_METRICS_PORT (default 9464), separate from the API port. Includes Kubernetes manifest snippets, pod-template annotations and labels for Prometheus scrape configuration, and environment-variable reference table entries in both DEPLOYMENT_GUIDE and README. Also updates the devnet test harness comment to document the per-participant metrics ports (9464/9465/9466). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix the reward sweep's silent 60s floor and credential gaps The loop's heartbeat timer was hardcoded to 60s, so sweep_is_due could only ever see 60s-sized steps: any configured interval below that was silently floored, and the localnet e2e's 15s interval quietly ran at 60s. Build the timer from HEARTBEAT_INTERVAL.min(sweep_interval) instead, so a short interval beats at its own rate; beating faster is safe because the stall alert's `increase < 1 over 10m` is a lower bound. Two in-tree comments assumed the fast path already worked and are corrected to say why it now does. A Keycloak token failure returned through get_party_credentials was indistinguishable from "no active delegation" or "not a listed assigner" — all three cleared the expiry gauge as if the automation were deliberately off. That silences decman-reward-coupons-at-risk at the exact moment a backlog is draining and auth breaks. Give the credentials arm its own SweepOutcome so it holds the countdown steady like a failed sweep, without counting toward decman_reward_sweep_failed_total (a node with credentials but no loaded token yet would otherwise trip decman-reward-sweep-failing on every tick). get_party_credentials itself is untouched — its signature change is out of scope here. Also: rename the loop's local oldest_expiry to oldest (it shadowed the module-level fn of the same name), drop a test comment that only restated sweep_is_due's own doc, and note in-place that the heartbeat and sweep share one task (so a blocking sweep costs heartbeats) and that last_sweep records wake time rather than tick deadline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Retry the reward e2e's skip assertion instead of failing Once assigned reached the seeded count, the probe hard-failed if skipped was still 0. That's reachable, not just theoretical: the drain's fan-out can break 'chunks on a transient error, and if that lands on the unassignable coupon's isolated submission after the healthy coupons already committed, the sweep ends with assigned complete and skipped at 0. Several TRANSIENT_ASSIGN_ERROR_IDS are plausible on a loaded CI box. Log and retry within the scenario's deadline instead — nothing quarantines that coupon, so the next sweep re-finds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Warn when metrics_port collides with another listener The metrics HttpServer binds before the API server, so a metrics_port that matches the API or noise port makes the node die reporting the wrong listener as the cause. A startup warning names the collision; it stays a warning, not a fatal check, since losing the metrics signal is not a reason to stop serving governance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop a false claim from the devnet ports comment The Metrics line claimed the listed ports match DECPM_METRICS_PORT in the per-participant .env files; nothing puts them there. It was copied from the Noise entry above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Stop the expiry gauge pinning to an unreaped corpse Devnet PQS shows 408,584 of 429,529 active RewardCouponV2 coupons already past expiresAt (oldest 35 days), because the DSO's reaper does not archive them promptly. oldest_expiry took the minimum over every unassigned coupon including expired ones, so once a decparty lost coupons the metric pinned to the oldest corpse and both expiry alerts fired forever instead of tracking the backlog still worth saving. * Correct the heartbeat metric's HELP string The fix wave made the loop wake on `HEARTBEAT_INTERVAL.min(sweep_interval)` so a configured interval below 60s is no longer floored. The metric's help text still claimed one heartbeat per minute, which is wrong for any such interval — the e2e suite runs at 15s. Production is unaffected: 300s and 21600s both floor to 60. The metric name is untouched, so no alert rule changes. Reported by Copilot on #325. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Export the outcome counters at zero from the first sweep A counter that only appears on its first event leaves a dashboard unable to tell a node that never failed from an instrument that is missing. Touching the four outcome counters where the sweep counter is already incremented creates each child at zero, so the panels draw a flat line rather than a no-data marker. No alert changes: every rule tests an increase of at least one, which a series sitting at zero never satisfies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Count a sweep that ended before assigning anything The zero-assigned counter keyed on the refusal tally, so it only counted a sweep whose coupons the ledger refused one by one. The contention path ends a sweep with both tallies at zero, which read as a sweep that had nothing to do. Devnet spent 30 hours in that state on 2026-08-15. Sweeps ran on their normal cadence and assigned nothing, every failure counter stayed flat, and the expiry gauge fell to 533 seconds before the backlog cleared. The alert this counter feeds would not have fired. The counter now keys on the backlog it was given, so any sweep that found coupons and assigned none of them counts, whatever ended it. * Refresh the expiry gauge on its own clock The sweep interval sizes a Delegation_Assign batch, so it is a transaction-cost choice. Detection inherited it: the gauge counts down from a timestamp only a sweep refreshes, so at devnet's 6h cadence a node cannot see that another assigner already took the coupon it is counting. A second clock now drives a read-only pass between sweeps. It reads the backlog and refreshes the gauge without assigning anything. A sweep reads the ledger anyway, so it satisfies the read clock too, and the gauge refreshes at whichever interval is shorter. Mainnet's 300s sweep therefore sees no change. DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS sets it, defaulting to 3600. A read that fails leaves the sweep counters alone, so a problem on the signal path cannot fire the sweep-failure alert. * Drop design-doc pointers from code comments A section number points into a file the reader does not have open, in a different repo, and it rots the moment a section is renumbered. It also reads as deferral: the comment gestures at a reason instead of giving one. Each comment now states the reason it was pointing at, or says nothing where the reason belongs only in the design. The RFC citations stay: those name a public standard, not our document. * Count the reads that refresh the expiry gauge A read-only pass incremented nothing and logged nothing on success, so the read clock was observable only through its effect: the gauge re-anchoring. Where a backlog is stable that effect is invisible, because re-reading the same coupon yields the same expiry instant and the countdown continues at the same slope. decman_reward_expiry_read_total makes the reader answerable on its own. It is zero-initialised with the outcome counters, which matters most here: a node whose sweep interval is shorter than its read interval never increments it, and absence would read as a fault. * Supervise the heartbeat and the peer listener too Both dropped their JoinHandle, so either could panic or return and the process would carry on with no signal. The heartbeat stops updating peer liveness; the peer listener stops accepting jobs. Both look healthy from outside, because /healthz keeps answering 200. The reward loop already had an outer task reporting its death. That block is now a helper the three share, so each names what the operator loses rather than repeating the mechanism. * Report the backlog when the delegation omits this node A delegation that exists and does not name this node returned Empty, which cleared the expiry gauge and took both expiry alerts down with it. So a governance vote naming the wrong party produced a node that swept, counted its sweeps, and reported nothing while the coupons expired. The only log line was at debug, which the filter drops. A vote naming the wrong party is a mistake, not an off switch. The node now reads the backlog, reports it, and assigns nothing. The line moves to warn, because fixing the vote is the action. A decparty with no delegation at all still clears the series. That is the deliberate off switch, and mainnet is in it today. * Make a missing rewards DAR a no-op A participant with no rewards DAR cannot host a delegation, so the automation is off for that node. The loop still counted it: the error arm incremented the sweep-failure counter twelve times an hour while #335's trace line kept the log quiet. That is what blocks the sweep-failing alert. The new Off outcome counts nothing, logs at trace and touches nothing. It is deliberately not Empty: Empty drops the gauge series, which asserts that nothing is left to save. A node that held the DAR and lost it would then go silent instead of keeping its countdown. Robert settled this on 2026-08-17, so the opt-in config field the plan proposed is no longer needed to silence these nodes. * Treat a mediator confirmation timeout as transient MEDIATOR_SAYS_TX_TIMED_OUT was not in the transient allowlist, so a mediator that did not gather confirmations in time was read as a bad coupon. Observed on devnet on 2026-08-20. The drain isolated one coupon, logged it at ERROR, counted it skipped, and fired the coupon-rejected alert. The same coupon assigned cleanly on the next sweep, so nothing was wrong with it. Four costs followed from the misclassification. The log line blamed the coupon. The skipped counter overstated real rejections. The alert fired on a condition that heals itself. And the drain carried on submitting into a mediator that was already timing out, instead of ending the sweep to re-read as it does for every other timeout. The list already carries four sibling timeouts on exactly this reasoning: LOCAL_VERDICT_TIMEOUT, NOT_SEQUENCED_TIMEOUT, SEQUENCER_BACKPRESSURE and REQUEST_TIME_OUT. This one belongs with them. The new test pins the reason rather than the string, and records the gRPC code the error actually arrived under. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the expiry countdown when a decparty loses auth get_party_credentials now separates a token fetch that failed from a decparty the auth registry does not know. Both were treated as one sweep's transient problem, which keeps the remembered expiry so the countdown carries on falling. That is right for a failed token fetch and wrong for the other case. The registry is rebuilt whenever party config changes, so a decparty that read a backlog earlier can stop resolving to credentials and stay that way. Its countdown then falls to zero on a reading nothing can refresh, and both expiry rules fire for a decparty this node no longer serves. prune_unserved cannot catch it: that keys on the party_credentials rows, which still exist. SweepOutcome::Unserved drops the timestamp and its gauge series, the way an empty backlog does. The two are kept apart because they assert different things: an empty backlog says nothing is left to save, while this says only that the backlog is unreadable here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Count the auth-registry gap and keep its countdown AuthRegistry::new skips a party whose token manager fails to initialise and logs a warn, but the party_credentials row survives. The reward loop iterates those rows, so a decparty with a bad Keycloak secret resolved to no credentials on every pass and dropped its expiry gauge series. Its coupons expired with both expiry rules silent and no counter moving. prune_unserved already forgets a decparty whose row is gone, so this arm can only mean "row present, registry entry missing" — a fault, not an off switch. The countdown now stays, and a new counter carries the fault, because a decparty whose auth failed at boot never read a backlog and so has no series to fire on. * Name the stall window the rules actually use, and guard 0 The two comments still cited a 10-minute window after the stalled rule moved to 15 minutes, so anyone sizing a change against that rule read the wrong number out of the code. reward_expiry_read_interval_secs also lacked the zero warning its sibling has. A 0 becomes 1s, which is a full ACS read per decparty per second. * Stop a lost signal from taking down governance The metrics listener bound first, so a metrics_port that collided with the API or noise port won the race and killed the other listener. It now yields the port and serves no metrics, which is the rule the bind-failure arm beside it already kept. The metrics server and the Noise inbound accept loop were also the two long-lived tasks in start_server that skipped spawn_supervised. A metrics server that dies leaves every alert rule reading an empty series while the node looks healthy, and a dead accept loop refuses every inbound peer in silence. * Match the metric family exactly when summing it counter_total tested a bare prefix, so a longer family name starting with the one asked for would have summed in too. * Document the reward knobs and the metrics port's reach None of the four DECPM_REWARD_* variables appeared in the README or the deployment guide. DECPM_REWARD_EXPIRY_READ_INTERVAL_SECS matters most: it bounds how stale the expiry gauge gets, and both expiry alert rules read that gauge. The deployment guide also targets outside node operators, and it opened a third port on their host without saying that the listener answers any caller that reaches it. * Count a token fetch that fails on every pass The credentials arm keeps the expiry countdown, which is right, but it incremented nothing. A decparty whose Keycloak client is gone fails its token fetch on every pass and only writes one warn line, and a decparty broken from boot never read a backlog so it has no gauge series to fall either. Devnet measures the case: on 2026-08-25 nodes 1 and 2 hit this arm for 28 node/decparty pairs, on every pass, all of them stale test decparties. Nothing counted it. * Test that both auth counters actually count Both increments sat on a path no test reached, so removing either one left the suite green. Each arm now has a case that drives run_once_for_party and reads the counter back: the registry arm through an AppState with no auth, the token arm through a Keycloak stand-in that answers once and then fails. Removing an increment fails its test. The AppState builder moves from governance's test module to an AppState::for_test beside the struct, so the two test modules share one copy and a new field breaks one place. * Count an auth failure only where value is at risk The two auth counters moved for any decparty this node could not authenticate for, whether or not that decparty ever expected Mode A. Devnet holds one active CouponReassignmentDelegation, for cbtc-network, so all 28 firing series were decparties with no delegation and nothing to lose. Every alert was a false positive. The delegation is the enablement signal, and it was unreachable here because reading it needed the token that had just failed. It does not: the template makes its assigners observers, so the node reads the delegations naming it under any working token and learns which decparties matter before deciding to count. An unread set counts nothing. `None` means the node could not ask, not that nothing is enabled, so guessing either way is worse than staying quiet until a read lands. --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Old behavior
Every decman node runs the reward automation loop, because
reward_automation_interval_secsdefaultsto 300 seconds. A node whose participant does not hold
governance-rewards-automation-v1cannotresolve that package name, so every sweep fails and logs a warning. That is one line per decparty,
twelve times an hour. Devnet nodes
dec-party-manager-3,-5and-6have produced 864 such lines aday between them.
New behavior
The loop now reads the Canton error id before it picks a level. It logs at
tracewhen theparticipant does not hold a package the automation reads, and the deployed filter
dec_party_manager=infodrops that event before it is formatted. Every other failure still logs atwarn, so a genuine sweep failure keeps its warning.Here is the quiet case on devnet. An operator recovers it by adding
dec_party_manager::server::reward_automation=tracetoRUST_LOG, decparty and ledger error intact:The change that enables it
package_absent()helper answers one question: does the error carry the Canton error idPACKAGE_NAMES_NOT_FOUND?traceand names the causein its message; the other branch keeps the original
warnline unchanged.Caveats
sweep still fails. #334 carries
the cause and the options for removing it, for the team to decide.
failure is unactionable, so that is the intent. The metric
decman_reward_sweep_failed_totalin #325still counts it once that lands.
drain_assignable, per coupon, aterror. This PR does not touch that.Details
What changed
One file,
crates/decman/src/server/reward_automation.rs. The classification reuses the existingcanton_error_id(), which reads the id from the message prefix of atonic::Status. That helperalready backs the transient-assign rule, and it searches the whole
anyhowchain, so a.context(..)added later does not turn the line back into a warning.
Why one error id is enough. Canton raises three package errors on a read, and the loop can only
ever see the first.
IndexServiceImpl.checkNameTypeConRefresolves a filter's package name againstthe participant's uploaded packages, and rejects an unknown name with
PACKAGE_NAMES_NOT_FOUNDbefore it checks anything else. The delegation read is also the loop's first ledger call, so the
coupon read never runs on an affected node. The remaining ids need a package that resolves by name
but lacks the template or the interface, which our own DARs cannot do.
Vetting does not reach this statement either. The read resolves names from uploaded packages, so a
DAR that a participant holds and has not vetted raises nothing here. A submission that fails on
vetting is caught by
drain_assignable(), which absorbs every submission error and returns a count.The failure happens on the read, not on a submission.
active_delegation()looks upGovernance.Rewards.CouponReassignmentDelegationunder the package name#governance-rewards-automation-v1, whichdefault_package_config()hardcodes. The ledger APIrejects that query with
PACKAGE_NAMES_NOT_FOUNDwhen the participant has never had the DAR uploaded.A missing delegation is a different case:
active_delegation()returnsOk(None), the sweepno-ops, and nothing is logged either way.
The DAR reaches a participant through the
Darsworkflow, which an operator runs. Nothing uploads itat startup, and the image carries only the binary. So a fresh node running any release with this loop
warns until that upload happens.
Verification
DECMAN_SKIP_FRONTEND=1 cargo test -p decmanpasses: 396 lib, 5 binary and 44 integration tests, 2ignored. That is the
mainbaseline plus the one test below. The branch carriesorigin/mainmerged in.
cargo clippy -p decman --all-targets -- -D warningsis clean, andcargo fmt --allmade nochanges.
only_an_absent_package_is_quietcovers both branches. It feeds the devnet messageverbatim and asserts the quiet classification, including after a
.context(..)wrapper. It thenasserts that a different ledger rejection and a non-ledger failure both stay at
warn.Verified on devnet. Node
dec-party-manager-5now runs this branch's commit image, for decpartytest-token-devnet-5. Its participant does not hold the DAR, so it is one of the three noisy nodes.dec_party_manager=info, the pod logged 15 lines in thefirst 6 minutes. All 15 are startup lines, and the reward loop logged nothing.
dec_party_manager::server::reward_automation=tracetoRUST_LOG, the same failurecame back at
TRACE. It carries the decparty and the fullPACKAGE_NAMES_NOT_FOUNDmessage.and failing through the quiet window. The line was suppressed, not absent.
severity_text = TRACEandseverity_number = 1. An operator whowidens the filter can therefore query them by severity, not only through
kubectl.Two earlier measurements from devnet also stand behind the change.
This statement accounts for all of the noise. Grouping every log line from the three affected
devnet nodes by message and severity, over six hours, returns exactly one row:
reward automation tick failed,WARN, 216 lines. There is no second message and no other severity.It is the only site that can log this failure.
main'sreward_automation.rshas fivewarnsites. Four sit in the assign path, which these nodes never reach, because they fail at the delegation
query. The fifth is the
interval_secs == 0guard at startup.#325 adds the failure metric ·
#334 removes the loop from nodes
that cannot run it