From f36eff536654189da6ab879674d9afadf0f117d7 Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Wed, 23 Sep 2026 13:40:06 +0000 Subject: [PATCH 1/5] fix(alertmanager): cascade covers behind the commit point too, closing the orphaned [user-cover] race (BLO-33497) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A resolve landing concurrently with the escalation sweep's chain-exhausted rung could strand an open `[user-cover]` with an unresolved member for an alert that had already cleared, permanently — nothing re-triggers `closeCoverIfEligible` for it, and `reconcileStuckCovers` only resumes covers that already won a closing claim. The sweep creates its cover before its compare-and-swap (claiming first would leave a failed `createCover` permanently uncovered), and compensates when that swap is REFUSED. The other half had nothing watching it: the webhook's cascade ran while the cover did not yet exist, then stored `resolvedAt` only after the swap had succeeded — so no compensation ran. Moving the existing cascade behind the state write is the wrong repair, and `worker.test.ts`'s "fails the delivery without marking resolved when cover cleanup fails" pins why: `ctx.state.set` is the commit point, and side effects sit ahead of it so a failure leaves `resolvedAt` unwritten and the retry redoes everything. Moving it swaps a concurrency orphan for a failure orphan. So cascade twice. The pre-commit call makes cover cleanup a precondition of recording the resolution; the post-commit call catches a cover that did not exist when the first ran, since a swap that succeeds means the cover was created before the commit. Neither subsumes the other — removing either fails a different test. The second call is near-free: it early-returns when the alert never joined a cover, and re-marking is `COALESCE(resolved_at, now())`. Verified: the new interleaving test fails against master; removing the post-commit cascade fails it; removing the pre-commit cascade fails the durability guard. 335/335 in the package, typecheck clean. --- .../paperclip-plugin-alertmanager/README.md | 35 +++++++ .../src/__tests__/escalation.test.ts | 95 +++++++++++++++++++ .../src/escalation.ts | 17 +++- .../src/webhook-handler.ts | 44 +++++++++ 4 files changed, 187 insertions(+), 4 deletions(-) diff --git a/packages/plugins/paperclip-plugin-alertmanager/README.md b/packages/plugins/paperclip-plugin-alertmanager/README.md index 7189e2877bd8..f98cdcbb03fb 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/README.md +++ b/packages/plugins/paperclip-plugin-alertmanager/README.md @@ -661,6 +661,41 @@ sibling that is still firing keeps the cover open. The "chain exhausted" comment sits behind the swap, so no announcement is posted for an alert that has already cleared. +That compensation only fires when the swap is **refused**, which left one more +interleaving open (BLO-33497). The webhook's cover cascade ran *before* it +stored `resolvedAt`, so a resolve could cascade while the cover did not yet +exist — nothing to mark — and then store `resolvedAt` only *after* the sweep's +swap had already succeeded. The swap succeeding means no compensation runs, and +no later resolve will ever cascade into that cover again: an open +`[user-cover]` with an unresolved member, for an alert that has cleared, +permanently. + +The obvious repair — move the cascade behind the state write — is wrong, and +the existing tests say so. `ctx.state.set` is the delivery's **commit point**, +and every side effect is deliberately sequenced ahead of it so that a failure +leaves `resolvedAt` unwritten and the retry redoes the lot. Moving the cascade +past it swaps a concurrency orphan for a failure orphan: a cascade that throws +would leave a record asserting the alert is over with its cover uncleaned. + +So `handleResolved` cascades **twice**, and the two calls answer different +failures: + +- **ahead of the commit point** — makes cover cleanup a precondition of + recording the resolution. A throwing cascade aborts the delivery with nothing + recorded. +- **behind the commit point** — catches a cover that did not exist yet when the + first call ran. A swap that *succeeds* means the sweep read, created its cover + and claimed all before the commit, so by the time the second call runs the + cover is there to be closed. + +Between them there is no window: the sweep compensates the refused-swap half, +and the post-commit cascade covers the succeeded-swap half. The second call is +close to free — `recordSourceResolvedAndCloseCovers` early-returns when the +alert never joined a cover (the common case), re-marking is +`COALESCE(resolved_at, now())`, and the close is a single-UPDATE claim only one +caller can win. If it throws, the delivery still fails and the retry's +pre-commit cascade closes the cover, which by then exists. + ### Bearer rotation in a Kubernetes deployment In a typical onprem-k8s deployment the bearer value lives in three places diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts index 66112b3d0a09..f07f79d47a78 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts @@ -857,6 +857,101 @@ describe("BLO-20650 concurrent webhook + sweep on one alert-state record", () => expect(coverRow.cancelled_at).not.toBeNull(); }); + /** + * BLO-33497 — the compensation above is reached only when the swap is + * REFUSED, so it does not cover the interleaving where the swap SUCCEEDS. + * The webhook cascaded only *before* storing `resolvedAt`, which allowed: + * the resolve's cascade runs while the cover still does not exist (no + * membership to mark), the sweep then creates the cover and wins its swap + * against a record the webhook has not written yet, and only afterwards does + * the webhook store `resolvedAt`. Nothing compensates, and no later resolve + * can ever cascade into that cover again — it is orphaned open with an + * unresolved member for an alert that has already cleared. + * + * The fix is a SECOND cascade in `handleResolved`, behind the state write. + * Simply moving the existing one is wrong: `ctx.state.set` is the commit + * point, and `worker.test.ts`'s "fails the delivery without marking resolved + * when cover cleanup fails" pins the cascade ahead of it so a failure leaves + * `resolvedAt` unwritten. Two calls answer the two failures — the first makes + * cleanup a precondition of committing, the second sees a cover that did not + * exist when the first ran, since a swap that succeeds means the cover was + * created before the commit. + */ + it("closes the cover when the resolve cascades before it exists and stores state after the swap", async () => { + const exhausted: AlertStateRecord = { ...unresolved(), escalationAttempt: 1 }; + const covers = buildFakeAlertmanagerStore(); + + // Hold the webhook's authoritative (un-guarded) state write open until the + // sweep has finished, and report when it is reached. That *is* the + // interleaving: everything the webhook does before storing `resolvedAt` + // happens first, and the store itself happens last. Gating on the write + // rather than on a tick count keeps it exact in both orderings. + let reachedStateWrite = false; + let releaseStateWrite!: () => void; + const gate = new Promise((resolve) => { releaseStateWrite = resolve; }); + const base = buildFakeStateStore(exhausted); + const store = { + ...base, + set: vi.fn(async (ref: unknown, value: AlertStateRecord, options?: { ifMatch?: unknown }) => { + if (!options || !("ifMatch" in options)) { + reachedStateWrite = true; + await gate; + } + return base.set(ref, value, options); + }), + } as unknown as ReturnType; + + const { ctx, mocks } = sweepContext(exhausted, null, covers); + mocks.state = store as never; + + // The resolve shares the sweep's covers/members tables, so its cascade is + // real rather than modelled — that is the whole point of this test. Every + // other table keeps the permissive single-delivery answers `resolveContext` + // already uses, so the only difference from the sibling test above is the + // cascade's visibility of the membership. + const resolveDb = { + namespace: "ns", + execute: async (sql: string, params: unknown[] = []) => + sql.includes("cover") ? covers.db.execute(sql, params) : { rowCount: 0 }, + query: async (sql: string, params: unknown[] = []) => + sql.includes("cover") ? covers.db.query(sql, params) : [], + }; + + let webhook!: Promise; + mocks.access.members.list = vi.fn(async () => { + // Start the resolve inside `createCover`, at the same await the + // compensated test uses — after the membership guard, before the cover + // issue exists. + webhook = handleResolved( + { ...(resolveContext(store) as unknown as Record), db: resolveDb } as unknown as PluginContext, + config(), + resolvedAlert, + ); + // Let it run right up to its state write. Bounded, so a change to the + // webhook's shape fails this test rather than hanging it. + for (let i = 0; i < 1000 && !reachedStateWrite; i++) await Promise.resolve(); + expect(reachedStateWrite).toBe(true); + return [{ principalType: "user", principalId: "board-1", status: "active", membershipRole: "owner" }]; + }); + + await runAlertEscalationSweep(ctx, config(), new Date("2026-07-11T01:00:00Z")); + releaseStateWrite(); + await webhook; + + // The swap SUCCEEDED here — this is deliberately not the compensated + // branch, which is what makes it a distinct case from the test above. + expect(store.read().escalationComplete).toBe(true); + expect(store.read().resolvedAt).toBe("2026-07-11T02:00:00Z"); + // No cover may be left open with an unresolved member for a cleared alert. + // Both assertions fail with the cascade moved back ahead of the state + // write: membership stays open, which blocks the closing claim, so + // `reconcileStuckCovers` cannot clean it up either. + const [coverRow] = [...covers.covers.values()]; + expect(coverRow).toBeDefined(); + expect(covers.openMemberCount(coverRow.cover_issue_id)).toBe(0); + expect(coverRow.cancelled_at).not.toBeNull(); + }); + it("refuses a stale sweep write against a record any other writer touched", async () => { // Same guard, non-resolve mutation: any concurrent rewrite must void the // sweep's read. Otherwise this would be a special case for one field diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/escalation.ts b/packages/plugins/paperclip-plugin-alertmanager/src/escalation.ts index 26aa9fea3d4d..d75f14bf0516 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/escalation.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/escalation.ts @@ -460,10 +460,19 @@ async function advanceIssueLadder( const claimed = await casAlertState(ctx, ref, state, { ...state, escalationAttempt: MAX_ATTEMPTS, escalationComplete: true, nextEscalationAt: null }); if (!claimed) { // A webhook won the record while the cover was being created. If it was a - // resolve, its own cascade ran before the cover existed and so could not - // see it — leaving an open board-assigned cover for an alert that has - // already cleared. Re-running the cascade here against the cover we just - // created is the compensating close. + // resolve, its own cascade may have run before the cover existed and so + // could not see it — which would leave an open board-assigned cover for + // an alert that has already cleared. Re-running the cascade here against + // the cover we just created is the compensating close. + // + // BLO-33497: this branch is reached only when the swap is REFUSED, which + // is exactly the half `handleResolved` cannot see for itself. It pairs + // with the second `recordSourceResolvedAndCloseCovers` there, behind its + // commit point: a swap that SUCCEEDS means the webhook had not yet stored + // `resolvedAt` when we claimed, so its post-commit cascade still lies + // ahead of it and will find the cover we just created. Keep the two in + // step — drop that call and this compensation stops being sufficient on + // its own. // // Safe to run unconditionally on a resolved winner: the cascade is // idempotent (`COALESCE(resolved_at, now())` plus the single-UPDATE diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/webhook-handler.ts b/packages/plugins/paperclip-plugin-alertmanager/src/webhook-handler.ts index 05050a8fbe27..e5c6f0a83667 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/webhook-handler.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/webhook-handler.ts @@ -2537,6 +2537,14 @@ export async function handleResolved( // exhausted because the alert kept firing, not because the underlying // issue's status policy says so, so a resolved alert means its membership // in the shared cover is done either way. + // + // Position is load-bearing: this sits AHEAD of the `ctx.state.set` below, + // which is this delivery's commit point. Cover cleanup is therefore a + // precondition of recording the resolution — a throwing cascade aborts the + // delivery with `resolvedAt` still unwritten, so the retry re-runs every + // side effect rather than stranding an uncleaned cover behind a record that + // already claims the alert is over. Do not move it past the commit point; + // the second call below exists precisely so this one does not have to. await recordSourceResolvedAndCloseCovers( ctx, existing.paperclipCompanyId, @@ -2589,6 +2597,42 @@ export async function handleResolved( }; await ctx.state.set(stateRef, updated); + // BLO-33497: cascade a SECOND time, behind the commit point. This is not a + // duplicate of the call above — the two cover different failures, and + // neither subsumes the other: + // + // - the call AHEAD of the commit point makes cover cleanup a precondition + // of recording the resolution, so a cascade failure leaves nothing + // recorded and the retry redoes everything; + // - this one catches a cover that did not exist yet when that call ran. + // + // The escalation sweep's chain-exhausted rung creates its cover BEFORE its + // compare-and-swap (see `escalation.ts`; claiming first would leave a + // failed `createCover` permanently uncovered), so the two paths interleave. + // The sweep compensates when its swap is REFUSED, which is the half where + // this delivery had already stored `resolvedAt`. The other half had nothing + // watching it: the cascade above ran while the cover did not yet exist — + // no membership to mark — and `resolvedAt` landed only after the sweep's + // swap, so the swap SUCCEEDED and the sweep's compensation never ran. That + // stranded an open [user-cover] with an unresolved member for a cleared + // alert, which no later resolve can ever cascade into again. + // + // A swap that succeeds means the sweep read, created its cover and claimed + // all before the write above — so by the time we get here the cover exists + // and this call sees it. Together the two halves leave no window. + // + // Cheap and idempotent: `recordSourceResolvedAndCloseCovers` early-returns + // on `rowCount === 0` (the common case — most alerts never join a cover), + // re-marking is `COALESCE(resolved_at, now())`, and the close is a + // single-UPDATE claim only one caller can win. Failing here still fails the + // delivery, and the retry's pre-commit cascade closes the cover, which by + // then exists. + await recordSourceResolvedAndCloseCovers( + ctx, + existing.paperclipCompanyId, + aggregateResolution.issueId, + ); + await ctx.events.emit( "alertmanager.alert.resolved", existing.paperclipCompanyId, From c2dda1ab84b4ac94c791b14b6c171f1de1d74bd9 Mon Sep 17 00:00:00 2001 From: Omar Ramadan Date: Thu, 24 Sep 2026 07:39:05 +0000 Subject: [PATCH 2/5] test(alertmanager): pin the post-commit cascade test to the claimed branch (BLO-33497) escalationComplete and resolvedAt are both written by the webhook's own update, so the test's assertions held on the refused-swap compensation branch as well as the claimed one it documents. Assert the chain- exhausted comment, which only the claimed path posts. With that comment removed from escalation.ts, only this test fails (29/30). Co-Authored-By: Claude Opus 5.5 (1M context) --- .../src/__tests__/escalation.test.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts index f07f79d47a78..dbc7f0f31a7b 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts @@ -942,6 +942,12 @@ describe("BLO-20650 concurrent webhook + sweep on one alert-state record", () => // branch, which is what makes it a distinct case from the test above. expect(store.read().escalationComplete).toBe(true); expect(store.read().resolvedAt).toBe("2026-07-11T02:00:00Z"); + // The two lines above hold on the refused-swap branch too, since the + // webhook's own write sets both. Only the claimed path posts the + // chain-exhausted comment, so this pins the test to the branch it names. + expect(mocks.issues.createComment).toHaveBeenCalledWith( + "issue-1", expect.stringContaining("Agent chain exhausted"), "company-1", + ); // No cover may be left open with an unresolved member for a cleared alert. // Both assertions fail with the cascade moved back ahead of the state // write: membership stays open, which blocks the closing claim, so From 9dbff21642e04befd9676a0cb1b6d38886439cfc Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Thu, 24 Sep 2026 10:42:16 +0000 Subject: [PATCH 3/5] fix(alertmanager): keep a sweep failure legible, and narrow the README's announcement claim (BLO-33497) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ally suggestions 2 and 3 from the review of f36eff5. - escalation.test.ts: `webhook` was a floating promise between its assignment inside the `members.list` mock and `await webhook` after the sweep. When `expect(reachedStateWrite).toBe(true)` throws out through `createCover` and the sweep rethrows, `releaseStateWrite()` never ran, so the webhook's rejection surfaced unhandled and could mask the real assertion failure. Release and swallow on the unwind path only; the success path still awaits it normally, so a genuine webhook rejection is not hidden. - README: "no announcement is posted for an alert that has already cleared" overstated the guarantee in exactly the interleaving this PR adds. It holds on the refused-swap half; on the succeeded-swap half the resolve has not yet stored `resolvedAt`, so the rung reads the alert as firing and does post. The cover still closes via the post-commit cascade. Suggestion 1 (collapse the four prose sites) declined — see the PR comment. Co-Authored-By: Paperclip --- .../plugins/paperclip-plugin-alertmanager/README.md | 8 ++++++-- .../src/__tests__/escalation.test.ts | 12 +++++++++++- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/packages/plugins/paperclip-plugin-alertmanager/README.md b/packages/plugins/paperclip-plugin-alertmanager/README.md index f98cdcbb03fb..a36bc25cce8d 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/README.md +++ b/packages/plugins/paperclip-plugin-alertmanager/README.md @@ -658,8 +658,12 @@ the alert, runs the cover cascade itself (`recordSourceResolvedAndCloseCovers`). That is idempotent by construction and only cancels a cover whose every member has resolved, so a storm-batched sibling that is still firing keeps the cover open. The "chain exhausted" -comment sits behind the swap, so no announcement is posted for an alert that -has already cleared. +comment sits behind the swap, so a **refused** swap posts no announcement for +an alert that has already cleared. A swap that **succeeds** still can: the +resolve is mid-delivery and has not stored `resolvedAt` yet, so the rung reads +the alert as firing and posts "while alert remains firing" on the source issue. +The cover is still closed by the resolve's post-commit cascade below, so the +announcement is the only residue. That compensation only fires when the swap is **refused**, which left one more interleaving open (BLO-33497). The webhook's cover cascade ran *before* it diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts index dbc7f0f31a7b..befb6b254f77 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts @@ -934,7 +934,17 @@ describe("BLO-20650 concurrent webhook + sweep on one alert-state record", () => return [{ principalType: "user", principalId: "board-1", status: "active", membershipRole: "owner" }]; }); - await runAlertEscalationSweep(ctx, config(), new Date("2026-07-11T01:00:00Z")); + try { + await runAlertEscalationSweep(ctx, config(), new Date("2026-07-11T01:00:00Z")); + } catch (err) { + // The webhook is still parked on the gate. Release it and swallow its + // rejection here only, so an unhandled rejection cannot outlive this test + // and mask the sweep's real failure. On the success path below it is + // awaited normally, so a genuine webhook rejection still surfaces. + releaseStateWrite(); + await webhook?.catch(() => {}); + throw err; + } releaseStateWrite(); await webhook; From 0035439750f8af43c95443920332737bc83d259a Mon Sep 17 00:00:00 2001 From: Omar Ramadan Date: Thu, 24 Sep 2026 19:21:52 +0000 Subject: [PATCH 4/5] test(alertmanager): assert the resolve ordering after the sweep, not inside it (BLO-33497) runAlertEscalationSweep catches and logs a per-issue failure, so the expect(reachedStateWrite) inside the members.list mock was swallowed by the code under test and the test then failed later on the missing cover, naming the wrong cause. The try/catch around the sweep was unreachable in the state its comment described: the sweep's only rethrow is issues.list, before the webhook exists. Drop the try/catch and assert after releasing and awaiting the webhook. With the state-write flag never set, the test now fails on that assertion. Plugin suite: 335 passed; tsc --noEmit clean. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../src/__tests__/escalation.test.ts | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts index befb6b254f77..c873b229baf9 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts +++ b/packages/plugins/paperclip-plugin-alertmanager/src/__tests__/escalation.test.ts @@ -930,23 +930,17 @@ describe("BLO-20650 concurrent webhook + sweep on one alert-state record", () => // Let it run right up to its state write. Bounded, so a change to the // webhook's shape fails this test rather than hanging it. for (let i = 0; i < 1000 && !reachedStateWrite; i++) await Promise.resolve(); - expect(reachedStateWrite).toBe(true); return [{ principalType: "user", principalId: "board-1", status: "active", membershipRole: "owner" }]; }); - try { - await runAlertEscalationSweep(ctx, config(), new Date("2026-07-11T01:00:00Z")); - } catch (err) { - // The webhook is still parked on the gate. Release it and swallow its - // rejection here only, so an unhandled rejection cannot outlive this test - // and mask the sweep's real failure. On the success path below it is - // awaited normally, so a genuine webhook rejection still surfaces. - releaseStateWrite(); - await webhook?.catch(() => {}); - throw err; - } + // The sweep catches and logs a per-issue failure (runAlertEscalationSweep), + // so an assertion inside the members.list mock would be swallowed there. + // Assert the ordering out here instead, after the webhook is released and + // awaited: a webhook rejection surfaces first, then the ordering guard. + await runAlertEscalationSweep(ctx, config(), new Date("2026-07-11T01:00:00Z")); releaseStateWrite(); await webhook; + expect(reachedStateWrite).toBe(true); // The swap SUCCEEDED here — this is deliberately not the compensated // branch, which is what makes it a distinct case from the test above. From 2b2115fd42cc912d3f0063cb012fe8a4e9b8e90b Mon Sep 17 00:00:00 2001 From: PlatformSREEngineer Date: Thu, 24 Sep 2026 20:50:50 +0000 Subject: [PATCH 5/5] docs(alertmanager): bound the succeeded-swap announcement claim to its window (BLO-33497) Ally's remaining Suggestion on the README's BLO-33497 narrative. "A swap that succeeds still can" is correct in context but travels badly: lifted out of the paragraph it reads as though any successful swap risks announcing a cleared alert, when in the ordinary non-racing case the alert really is firing and the announcement is right. Scope it to the interleaving window under discussion. Prose only; no code path touched. escalation.test.ts: 30 passed. Co-Authored-By: Claude --- .../plugins/paperclip-plugin-alertmanager/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/plugins/paperclip-plugin-alertmanager/README.md b/packages/plugins/paperclip-plugin-alertmanager/README.md index a36bc25cce8d..6eee391af03b 100644 --- a/packages/plugins/paperclip-plugin-alertmanager/README.md +++ b/packages/plugins/paperclip-plugin-alertmanager/README.md @@ -659,11 +659,11 @@ the alert, runs the cover cascade itself only cancels a cover whose every member has resolved, so a storm-batched sibling that is still firing keeps the cover open. The "chain exhausted" comment sits behind the swap, so a **refused** swap posts no announcement for -an alert that has already cleared. A swap that **succeeds** still can: the -resolve is mid-delivery and has not stored `resolvedAt` yet, so the rung reads -the alert as firing and posts "while alert remains firing" on the source issue. -The cover is still closed by the resolve's post-commit cascade below, so the -announcement is the only residue. +an alert that has already cleared. A swap that **succeeds** in that same window +still can: the resolve is mid-delivery and has not stored `resolvedAt` yet, so +the rung reads the alert as firing and posts "while alert remains firing" on +the source issue. The cover is still closed by the resolve's post-commit +cascade below, so the announcement is the only residue. That compensation only fires when the swap is **refused**, which left one more interleaving open (BLO-33497). The webhook's cover cascade ran *before* it