From 7af742cef2d2a9ac8165a8f203c25347e12171f2 Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Sun, 20 Sep 2026 04:06:21 +0000 Subject: [PATCH 1/5] fix(metrics): pre-seed backstop counters at 0 so absence can't read as "no sweep" (BLO-29763) The startup pre-seed loop covered only the gauge. Both backstop counters were left unseeded, so their series are absent until the first .inc() -- reproducing the exact silence defect BLO-29763 was opened to remove, one metric over. Measured live 2026-09-20 on paperclip-0: paperclip_backstop_deferred_candidates -> 6 series (5 at 0, 1 at 20) paperclip_backstop_sweep_completed_total -> EXACTLY 1 series issue_graph_liveness.backstop absent entirely; depth flat at 20 for 3h paperclip_backstop_candidates_skipped_total -> 7 of 12 reasons present Consequences this fixes: - AC2 ("did this loop finish a sweep in the last N minutes", answerable from metrics alone) is unanswerable while the series is missing. - AC3 cannot distinguish "saw and skipped" from "never reached" for the 5 skip reasons that have never fired. - The onprem-k8s#3332 alert is `depth > 0 unless increase(counter[2h]) > 0`. `unless` cannot exclude on an absent arm, so the alert fires on process age rather than on a stalled backstop. paperclip-0 is recreated every few hours (4 pod IPs in 14h, restartCount 0), so the counter is absent more often than not, and the rule would be firing right now on a healthy loop. Uses the .inc(labels, 0) idiom already used for projectPrimaryWorkspaceFallback. Bounded: 2 + (2 x 12) = 26 series, no per-issue or per-agent labels (AC5). Guard is mutation-verified: reverting the counter pre-seed alone fails exactly the new test and no other. Co-Authored-By: Paperclip --- server/src/__tests__/metrics-service.test.ts | 23 ++++++++++++++++++++ server/src/services/metrics.ts | 12 ++++++++++ 2 files changed, 35 insertions(+) diff --git a/server/src/__tests__/metrics-service.test.ts b/server/src/__tests__/metrics-service.test.ts index 50ce10bef8ff..cfc9b46613d2 100644 --- a/server/src/__tests__/metrics-service.test.ts +++ b/server/src/__tests__/metrics-service.test.ts @@ -4,6 +4,8 @@ import { AUTH_REQUEST_METRIC, BACKSTOP_CANDIDATES_SKIPPED_METRIC, BACKSTOP_DEFERRED_CANDIDATES_METRIC, + BACKSTOP_SKIP_REASONS, + BACKSTOP_SOURCES, BACKSTOP_SWEEP_COMPLETED_METRIC, CONCURRENT_RUN_BLOCKED_METRIC, DEP_BLOCKED_WAKEUP_METRIC, @@ -1319,6 +1321,27 @@ describe("setQueuedRunOldestAgeMetrics (BLO-21116)", () => { }); describe("backstop metrics (BLO-29763)", () => { + it("pre-seeds the COUNTERS at zero too, not just the gauge, so absence never reads as 'no sweep'", async () => { + // Measured live 2026-09-20 on paperclip-0: the gauge was present for both streams + // (one at 20, one at 0) while `paperclip_backstop_sweep_completed_total` carried + // EXACTLY ONE series -- issue_graph_liveness.backstop was absent entirely, because + // that process had not yet completed a sweep. AC2 asks that "did this loop finish a + // sweep in the last N minutes" be answerable from metrics alone; an absent series + // cannot answer it, and `depth > 0 unless increase(counter[2h]) > 0` cannot exclude + // on a missing arm, so the alert fires on process age rather than on a stalled loop. + // This asserts the healthy-but-untouched state BEFORE anything is recorded. + const body = (await renderMetrics()).body; + + for (const source of BACKSTOP_SOURCES) { + expect(body).toContain(`${BACKSTOP_SWEEP_COMPLETED_METRIC}{source="${source}"} 0`); + for (const reason of BACKSTOP_SKIP_REASONS) { + expect(body).toContain( + `${BACKSTOP_CANDIDATES_SKIPPED_METRIC}{source="${source}",reason="${reason}"} 0`, + ); + } + } + }); + it("publishes both bounded streams at zero, then records depth, completion, and skips", async () => { let body = (await renderMetrics()).body; expect(body).toContain(`${BACKSTOP_DEFERRED_CANDIDATES_METRIC}{source="issue_graph_liveness.backstop"} 0`); diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 682f3eba77ff..89a760d6b5a5 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -3109,8 +3109,20 @@ function ensureRegistry(): { labelNames: ["source", "reason"], registers: [registry], }); + // Pre-seed every backstop series so "healthy" reads as a literal 0 rather than an + // absent series. The gauge already did this; the counters did not, which reproduced + // the exact silence defect BLO-29763 was opened to remove, one metric over: on a + // freshly-started process `paperclip_backstop_sweep_completed_total` is missing for + // any stream that has not yet completed a sweep, so "did this loop finish a sweep in + // the last N minutes" is unanswerable, and a `depth > 0 unless increase(...) > 0` + // alert cannot exclude on the missing arm and fires on process age instead of on a + // stalled backstop. Bounded: 2 sources, 2 + (2 x 12) = 26 series. for (const source of BACKSTOP_SOURCES) { backstopDeferredCandidates.set({ source }, 0); + backstopSweepCompleted.inc({ source }, 0); + for (const reason of BACKSTOP_SKIP_REASONS) { + backstopCandidatesSkipped.inc({ source, reason }, 0); + } } pluginWebhookDeliveryRejected = new Counter({ name: PLUGIN_WEBHOOK_DELIVERY_REJECTED_METRIC, From 7076a87f4f3126ff845382743deb4e3f5e3dbe75 Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Sun, 20 Sep 2026 19:42:38 +0000 Subject: [PATCH 2/5] docs(metrics): correct the pre-seed rationale -- it enables `== 0`, it does not fix `unless > 0` (BLO-29763) Ally's Important finding on #1946 is correct and I had independently reached the same conclusion on the issue before the review landed. The recorded rationale claimed pre-seeding fixes a `depth > 0 unless increase(counter[2h]) > 0` alert that "cannot exclude on the missing arm". It does not: `> 0` discards a zero-valued sample exactly as it discards an absent series, so that expression returns the LHS and fires identically before and after this change. What pre-seeding actually buys is a stall predicate that was previously unwritable -- `increase(...[2h]) == 0` now returns a sample for a stream that has never swept, where absent it returned nothing and the condition was undetectable. Corrected both copies to state that, and to tell anyone tuning an alert to key on `== 0` / `absent_over_time` rather than `unless ... > 0`. Also from the same review: - series arithmetic: the loop seeds the gauge too, so it emits 28 series (2 sources x (1 gauge + 1 counter + 12 reasons)), not the 26 counted. - the ~8-line rationale was duplicated verbatim in the test; the test now keeps the live measurement and points at ensureRegistry for the mechanism. - noted that the seed is unconditional at registry construction, so an `== 0` stall alert should be scoped by job selector or gated on `paperclip_backstop_deferred_candidates > 0`. Topology Ally could not verify: both loops are driven from the same recoveryService and the server has a single entrypoint, so no process exports these without running them. Comment-only: `git diff -U0` shows zero changed lines outside `//` comments. Co-Authored-By: Paperclip --- server/src/__tests__/metrics-service.test.ts | 6 ++---- server/src/services/metrics.ts | 19 +++++++++++++++---- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/server/src/__tests__/metrics-service.test.ts b/server/src/__tests__/metrics-service.test.ts index cfc9b46613d2..524a1014d5b8 100644 --- a/server/src/__tests__/metrics-service.test.ts +++ b/server/src/__tests__/metrics-service.test.ts @@ -1325,10 +1325,8 @@ describe("backstop metrics (BLO-29763)", () => { // Measured live 2026-09-20 on paperclip-0: the gauge was present for both streams // (one at 20, one at 0) while `paperclip_backstop_sweep_completed_total` carried // EXACTLY ONE series -- issue_graph_liveness.backstop was absent entirely, because - // that process had not yet completed a sweep. AC2 asks that "did this loop finish a - // sweep in the last N minutes" be answerable from metrics alone; an absent series - // cannot answer it, and `depth > 0 unless increase(counter[2h]) > 0` cannot exclude - // on a missing arm, so the alert fires on process age rather than on a stalled loop. + // that process had not yet completed a sweep. Mechanism and the alert shape this + // does and does not fix: see the pre-seed loop in `ensureRegistry` (metrics.ts). // This asserts the healthy-but-untouched state BEFORE anything is recorded. const body = (await renderMetrics()).body; diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 89a760d6b5a5..e8c465c37252 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -3113,10 +3113,21 @@ function ensureRegistry(): { // absent series. The gauge already did this; the counters did not, which reproduced // the exact silence defect BLO-29763 was opened to remove, one metric over: on a // freshly-started process `paperclip_backstop_sweep_completed_total` is missing for - // any stream that has not yet completed a sweep, so "did this loop finish a sweep in - // the last N minutes" is unanswerable, and a `depth > 0 unless increase(...) > 0` - // alert cannot exclude on the missing arm and fires on process age instead of on a - // stalled backstop. Bounded: 2 sources, 2 + (2 x 12) = 26 series. + // any stream that has not yet completed a sweep, so AC2 ("did this loop finish a + // sweep in the last N minutes", answerable from metrics alone) has no series to read. + // + // What this buys, precisely: `increase(...[2h]) == 0` now RETURNS A SAMPLE for a + // stream that has never swept. Absent, it returned nothing and the stall was + // undetectable. It does NOT change `depth > 0 unless increase(...) > 0` -- `> 0` + // discards a zero-valued sample exactly as it discards an absent series, so that + // form returns the LHS and fires either way. An alert consuming this must key on + // `== 0` / `absent_over_time`, never on `unless ... > 0`. + // + // The seed is unconditional here, so a process that registers these and never runs a + // loop would export a permanently-zero counter. Not live today (both loops are driven + // from the same recoveryService, single entrypoint), but gate an `== 0` stall alert on + // a job selector or on `paperclip_backstop_deferred_candidates > 0` so it stays true + // if that changes. Bounded: 2 sources x (1 gauge + 1 counter + 12 reasons) = 28 series. for (const source of BACKSTOP_SOURCES) { backstopDeferredCandidates.set({ source }, 0); backstopSweepCompleted.inc({ source }, 0); From ee28d5f058f7a7c7835ec25861c8d9004551db44 Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Sun, 20 Sep 2026 19:46:37 +0000 Subject: [PATCH 3/5] =?UTF-8?q?docs(metrics):=20Ally's=20unverified=20topo?= =?UTF-8?q?logy=20caveat=20is=20LIVE=20=E2=80=94=20the=20api=20tier=20neve?= =?UTF-8?q?r=20sweeps=20(BLO-29763)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit My previous commit asserted the hazard away: "Not live today (both loops are driven from the same recoveryService, single entrypoint)". That is false, and it is the same class of error Ally had just caught me on -- an unverified claim in a rationale comment -- so it does not get to ship. Measured 2026-09-20 ~19:45Z against live Prometheus: paperclip_backstop_deferred_candidates -> 3 replicas paperclip-0 (service=paperclip-workers) issue_graph_liveness = 13 paperclip-api-* (service=paperclip) both sources flat at 0 paperclip_backstop_candidates_skipped_total -> paperclip-0 ONLY (500) Ally could not establish the process topology from code search and said so. The metrics settle it: only the workers tier runs the loops, and the api replicas carry the gauge purely because `ensureRegistry` pre-seeds it. The tier split is real in code too -- PAPERCLIP_NODE_ROLE=api fences run dispatch (heartbeat.ts:12286) and the plugin worker manager (index.ts:913). So once this PR lands, both api replicas publish `paperclip_backstop_sweep_completed_total{...} 0` forever, and an unscoped `increase(...[2h]) == 0` stall alert would page against them permanently. The comment now states the scoping requirement as a MUST with the selector (`service="paperclip-workers"`) rather than as a conditional caution. Still comment-only: `git diff -U0` shows zero changed lines outside `//`. Co-Authored-By: Paperclip --- server/src/services/metrics.ts | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index e8c465c37252..1b6c4146ec2e 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -3123,11 +3123,17 @@ function ensureRegistry(): { // form returns the LHS and fires either way. An alert consuming this must key on // `== 0` / `absent_over_time`, never on `unless ... > 0`. // - // The seed is unconditional here, so a process that registers these and never runs a - // loop would export a permanently-zero counter. Not live today (both loops are driven - // from the same recoveryService, single entrypoint), but gate an `== 0` stall alert on - // a job selector or on `paperclip_backstop_deferred_candidates > 0` so it stays true - // if that changes. Bounded: 2 sources x (1 gauge + 1 counter + 12 reasons) = 28 series. + // The seed is unconditional here, and the API tier registers these metrics without + // running either loop -- so after this change every api replica exports a + // permanently-zero completion counter. Measured 2026-09-20 on live Prometheus: the + // gauge is present on all 3 replicas, but `paperclip_backstop_candidates_skipped_total` + // exists on `paperclip-0` (`service="paperclip-workers"`) alone, and the two + // `paperclip-api-*` replicas sit at the seeded 0 for both sources. Run dispatch is + // already fenced on that tier (PAPERCLIP_NODE_ROLE=api, heartbeat.ts). So an `== 0` + // stall alert MUST be scoped -- `service="paperclip-workers"`, or gated on + // `paperclip_backstop_deferred_candidates > 0` -- or it pages forever against a tier + // that legitimately never sweeps. Bounded: 2 sources x (1 gauge + 1 counter + 12 + // reasons) = 28 series. for (const source of BACKSTOP_SOURCES) { backstopDeferredCandidates.set({ source }, 0); backstopSweepCompleted.inc({ source }, 0); From 1b407e564d44d4ee13e2a3842cf69d1dd84ab8a4 Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Mon, 21 Sep 2026 05:18:03 +0000 Subject: [PATCH 4/5] docs(metrics): the `deferred > 0` alert gate is anti-correlated -- drop it, state the mechanism (BLO-29763) Ally's Important finding is correct and I reproduced it against live Prometheus. The gauge and the completion counter are ONE variable with opposite sign -- `setBackstopDeferredCandidates(src, candidateLimitSkipped)` against `if (candidateLimitSkipped === 0) recordBackstopSweepCompleted(src)` -- so `deferred > 0` selects exactly the population where `increase(completed[2h]) == 0` is structurally guaranteed. The gate this comment offered would have reintroduced the permanent-false-page class BLO-29763 exists to remove. Confirmed live: paperclip-0 `issue_graph_liveness.backstop` reads gauge 48 with the completion counter absent, while the drained sibling stream reads gauge 0 / counter 1. I did not adopt the recommended inversion as a new paste-ready predicate, because I measured it firing too: `deferred == 0 and increase(completed[2h]) == 0`, scoped to `service="paperclip-workers"`, returns the healthy drained `stranded_recovery_wake_backstop` stream right now. Two causes -- pre-change the counter series is born at 1 so increase() never sees the 0->1 edge (the seed cures that), and post-change the start-to-first-sweep window still reads exactly like a stall. The latter matters here because paperclip-0 ran under 19 distinct pod UIDs in 24h with 0 container restarts each: the pod is replaced ~hourly, so series are routinely younger than a 2h window. So the comment now states the three constraints as mechanism rather than prescribing PromQL -- two review rounds have each produced a predicate that then measured wrong, because correctness turns on topology and process lifetime a code comment cannot track. Also addresses both Suggestions: names `paperclip_backstop_sweep_completed_total` explicitly in the api-tier clause so "sit at the seeded 0" is unambiguous, and adds the gauge assertion to the "not just the gauge" test so it is self-contained. Mutation-verified: removing `backstopDeferredCandidates.set()` now fails that test at the added line; it passed without it before. Co-Authored-By: Paperclip --- server/src/__tests__/metrics-service.test.ts | 1 + server/src/services/metrics.ts | 45 ++++++++++++++++---- 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/server/src/__tests__/metrics-service.test.ts b/server/src/__tests__/metrics-service.test.ts index 524a1014d5b8..75ecdb9207dc 100644 --- a/server/src/__tests__/metrics-service.test.ts +++ b/server/src/__tests__/metrics-service.test.ts @@ -1331,6 +1331,7 @@ describe("backstop metrics (BLO-29763)", () => { const body = (await renderMetrics()).body; for (const source of BACKSTOP_SOURCES) { + expect(body).toContain(`${BACKSTOP_DEFERRED_CANDIDATES_METRIC}{source="${source}"} 0`); expect(body).toContain(`${BACKSTOP_SWEEP_COMPLETED_METRIC}{source="${source}"} 0`); for (const reason of BACKSTOP_SKIP_REASONS) { expect(body).toContain( diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 1b6c4146ec2e..6f370c598727 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -3125,15 +3125,42 @@ function ensureRegistry(): { // // The seed is unconditional here, and the API tier registers these metrics without // running either loop -- so after this change every api replica exports a - // permanently-zero completion counter. Measured 2026-09-20 on live Prometheus: the - // gauge is present on all 3 replicas, but `paperclip_backstop_candidates_skipped_total` - // exists on `paperclip-0` (`service="paperclip-workers"`) alone, and the two - // `paperclip-api-*` replicas sit at the seeded 0 for both sources. Run dispatch is - // already fenced on that tier (PAPERCLIP_NODE_ROLE=api, heartbeat.ts). So an `== 0` - // stall alert MUST be scoped -- `service="paperclip-workers"`, or gated on - // `paperclip_backstop_deferred_candidates > 0` -- or it pages forever against a tier - // that legitimately never sweeps. Bounded: 2 sources x (1 gauge + 1 counter + 12 - // reasons) = 28 series. + // permanently-zero completion counter. Measured 2026-09-21 on live Prometheus: the + // gauge is present on all 3 replicas, while `paperclip_backstop_sweep_completed_total` + // and `paperclip_backstop_candidates_skipped_total` exist on `paperclip-0` + // (`service="paperclip-workers"`) alone -- on both `paperclip-api-*` replicas those + // two counters are absent pre-change and read a permanent seeded 0 after it. Run + // dispatch is already fenced on that tier (PAPERCLIP_NODE_ROLE=api, heartbeat.ts). + // + // Constraints on a stall alert over these series, given as mechanism rather than a + // paste-ready predicate: two review rounds have each produced a predicate that then + // measured wrong against live data, because correctness here turns on topology and + // process lifetime that a code comment cannot track. + // 1. Scope to `service="paperclip-workers"`, or it pages against a tier that + // legitimately never sweeps. + // 2. The gauge and the completion counter are ONE variable with opposite sign -- + // `setBackstopDeferredCandidates(src, result.candidateLimitSkipped)` against + // `if (result.candidateLimitSkipped === 0) recordBackstopSweepCompleted(src)` + // (recovery/service.ts). So `deferred > 0` and "no completion on that tick" are + // the SAME fact, not two independent signals, and ANDing them pages forever on a + // saturated-but-healthy stream. Corollary: NO predicate over these two series + // detects a stall on a permanently-saturated stream -- that needs the + // zero-deferred-tick fix, not an alert change. + // 3. The seed makes a never-swept process read identical to a drained one: both + // counters sit at 0 until that process completes its first sweep. So an `== 0` + // alert needs a `for:` longer than one sweep interval -- and process lifetime + // here is short. Measured 2026-09-21, `paperclip-0` ran under 19 distinct pod + // UIDs in 24h with 0 container restarts on each: the pod is REPLACED roughly + // hourly, so every series is born fresh well inside a 2h window. + // Worked example, same day: `deferred == 0 and increase(...[2h]) == 0`, scoped + // to the worker tier, DID fire against the healthy, drained + // `stranded_recovery_wake_backstop` stream. Two distinct causes, and only one of + // them is fixed here -- pre-change the counter series is born at 1 so + // `increase()` never sees the 0->1 edge (the seed below cures that); post-change + // the start-to-first-sweep window still reads exactly like a stall (only a + // `for:` longer than the sweep interval covers that one). + // + // Bounded: 2 sources x (1 gauge + 1 counter + 12 reasons) = 28 series. for (const source of BACKSTOP_SOURCES) { backstopDeferredCandidates.set({ source }, 0); backstopSweepCompleted.inc({ source }, 0); From 1c96641d7480cc320db41715989930a938a4453d Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Mon, 21 Sep 2026 10:55:47 +0000 Subject: [PATCH 5/5] docs(metrics): the backstop cadence is one CHAIN completion, not one 30s tick (BLO-29763) Ally's Important finding at 1b407e56 is correct and reproduced independently: `paperclip_backstop_sweep_completed_total` does not increment per sweep, so "an `== 0` alert needs a `for:` longer than one sweep interval" understates the bound by ~35x and would page on every fresh pod. Measured 2026-09-21 on paperclip-0, and going one step past the review, which recorded the measurement and explicitly declined the mechanism: - 7 consecutive pod generations over ~7h each top out at exactly 1. Never 2. - On instance 10.244.3.76 the first sweep lands ~19 min after process start; for the following 30 min BOTH skip counters are frozen (issue_graph 500, stranded 12) and the issue_graph gauge is frozen at 33. One sweep, then nothing. Mechanism: neither loop is driven by the 30s scheduler tick. Both run only inside `reconcileIssueGraphLiveness`, which sits partway down the heartbeat recovery chain behind `reconcileStrandedAssignedIssues` and is gated by the `heartbeatRecoveryChainInFlight` latch (index.ts:1759, BLO-34207/#1897). A tick that finds the chain in flight skips it silently, and the latch declaration already says a sweep "routinely outlives one interval". `reconcileIssueGraphLiveness` has no internal throttle, so cadence is entirely driver-gated. So the fix is not a bigger `for:`. Ally recommended sizing it off the observed ~18 min start-to-first-completion; that quantity is the duration of an unrelated sweep chain and moves with estate size and with the convoy fix, which is exactly the decay that has now produced three wrong predicates in three rounds. The comment states the topology and refuses to carry a constant. Also promotes pod lifetime from reassurance to a stated precondition, per the review, and moves the dated measurements out to BLO-29763 (Ally suggestion 2) rather than adding a fourth dated paragraph to a comment that keeps decaying. Comment-only; no behaviour change. --- server/src/services/metrics.ts | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/server/src/services/metrics.ts b/server/src/services/metrics.ts index 6f370c598727..9017dd4bbff1 100644 --- a/server/src/services/metrics.ts +++ b/server/src/services/metrics.ts @@ -3147,18 +3147,28 @@ function ensureRegistry(): { // detects a stall on a permanently-saturated stream -- that needs the // zero-deferred-tick fix, not an alert change. // 3. The seed makes a never-swept process read identical to a drained one: both - // counters sit at 0 until that process completes its first sweep. So an `== 0` - // alert needs a `for:` longer than one sweep interval -- and process lifetime - // here is short. Measured 2026-09-21, `paperclip-0` ran under 19 distinct pod - // UIDs in 24h with 0 container restarts on each: the pod is REPLACED roughly - // hourly, so every series is born fresh well inside a 2h window. - // Worked example, same day: `deferred == 0 and increase(...[2h]) == 0`, scoped - // to the worker tier, DID fire against the healthy, drained - // `stranded_recovery_wake_backstop` stream. Two distinct causes, and only one of - // them is fixed here -- pre-change the counter series is born at 1 so - // `increase()` never sees the 0->1 edge (the seed below cures that); post-change - // the start-to-first-sweep window still reads exactly like a stall (only a - // `for:` longer than the sweep interval covers that one). + // counters sit at 0 until that process completes its first sweep. Sizing a + // `for:` around that needs the sweep CADENCE -- and the cadence is NOT the 30s + // scheduler tick. Neither loop is driven by that tick: both run only inside + // `reconcileIssueGraphLiveness`, which sits partway down the heartbeat recovery + // chain behind `reconcileStrandedAssignedIssues` and is gated by the + // `heartbeatRecoveryChainInFlight` latch (index.ts, BLO-34207/#1897). A tick + // that finds the chain in flight skips it silently, and the latch declaration + // says outright that a sweep "routinely outlives one interval". So a completion + // is one CHAIN COMPLETION, not one tick, and that period is a property of the + // chain's slowest pass and of estate size -- not of these metrics, and not of + // anything this file can track. Do NOT paste a `for:` constant from here. + // 4. PRECONDITION, not reassurance: `increase([Nh]) == 0` is only + // meaningful while process lifetime stays under N. Because a completion is + // per-chain rather than per-tick, that expression currently reads as a function + // of PROCESS AGE more than of sweep liveness; it is quiet today only because the + // pod is replaced on a shorter period than the range window. Anything that + // lengthens pod lifetime past N makes it fire permanently on a healthy stream. + // Any alert built on it must state that bound, and a `for:` shorter than the + // start-to-first-completion window pages on every fresh pod. + // Measurements behind 3-4, and the standing caveat that this cadence moves: + // BLO-29763. Three review rounds have each produced a predicate that then + // measured wrong -- a number recorded here is the thing that keeps decaying. // // Bounded: 2 sources x (1 gauge + 1 counter + 12 reasons) = 28 series. for (const source of BACKSTOP_SOURCES) {