fix(heartbeat): bound agent-start-lock against a section that never settles (BLO-23094)

[allyblockcast]

Context

BLO-23094: Ally's queued-run dispatch produced 0 starts for ~49 minutes with 13/15 slots free, while the fleet dispatched normally for every other agent. agent-start-lock.ts was the prime suspect — its own warning ("agent start lock held longer than expected") fired for Ally 112/144 times in 6h.

What I found

Live repro at investigation time: by the time I checked out the issue, the wedge had already self-cleared without a paperclip-0 restart (pod uptime 14h, 0 restarts) — a burst of 13 Ally dispatches landed at 00:57:08Z, and the lock-held warning has not fired for Ally since (16-minute log window checked). Ally is now legitimately saturated (15/15), and the six still-queued runs are ordinary capacity-bound queueing, not a wedge (BLO-23071's original concern, not this one). I could not reproduce the zero-throughput state live.

Code review of the suspected mechanism: agent-start-lock.ts's cross-agent nesting (via the non-agent-scoped orphan reaper: reap → promote → dispatch) is bounded by MAX_NESTED_DISPATCH_DEPTH=4 and is exercised by the existing test suite ("caps nesting depth..." in heartbeat-start-lock.test.ts) — this is a deliberate, tested design, not a bug. Every k8s API call in the dispatch path (k8s-job-liveness.ts) is AbortSignal.timeout-bounded (2s in prod). I found no single unbounded/never-settling await.

The actual gap: the module's own doc comment states "there is no timeout bypass" — true for ordinary contention, but the mutex had zero liveness backstop if fn() (the per-agent dispatch body) ever does hit a genuinely unbounded await (a future bug, not the historical BLO-20396 defect). If that happens, runningByAgent's marker for that agent never resolves, and every coalesced follow-up chained onto it via ensureCoalescedFollowUp's running.then(...) waits forever too — wedging that agent indefinitely with no recovery path short of a process restart. That matches the acceptance criteria's ask directly: "A wedged per-agent section can no longer stall that agent indefinitely — bounded by a timeout, watchdog, or forced release."

The fix

Add LOCK_FORCE_RELEASE_MS (5 minutes — an order of magnitude above LOCK_HELD_WARN_MS and above any duration the bounded, depth-capped, timeout-protected cascade could plausibly produce). Past that ceiling:

• runningByAgent's entry is deleted (guarded by marker identity, same pattern the existing finally block already uses).
• The marker is force-settled, which unblocks any coalesced follow-up already chained on it.
• Logged as error (not warn) — crossing this ceiling means the section is not slow, it is stuck, and that is a bug to chase.

This is deliberately not a reintroduction of the BLO-20396 defect: that bypass fired at 30s (well within ordinary contention) and let two sections run concurrently as routine behavior. This ceiling only fires on a section that has already stopped being merely slow — normal operation, including the worst-case 4-deep nested cascade, should never reach it. The abandoned execution isn't cancelled (there's no way to cancel an arbitrary in-flight await from here); it's left to settle later as a no-op against the bookkeeping. That's an explicit, narrow trade of strict mutual exclusion for liveness, made only in the already-abnormal case where the alternative is permanent starvation.

Also found, filed separately (not fixed here)

heartbeat-run-queue-latency-exporter (referenced in the issue's verifying signal) is deployed and its pg_heartbeat_run_queue_backlog_by_agent query already has agent_id/agent_name labels and an oldest_queued_seconds gauge per agent — but Prometheus has no scrape target for it (up{job=~".*heartbeat.*"} is empty), so none of these metrics are actually queryable/alertable right now despite the exporter running. Filed as a follow-up.

Testing

• npx vitest run server/src/__tests__/heartbeat-start-lock.test.ts — 11/11 pass, including the new never-settles case.
• tsc --noEmit clean on both touched files.

Issue: https://paperclip.blockcast.net/BLO/issues/BLO-23094

[allyblockcast]

🔗 Paperclip issue: BLO-20396
🔗 Paperclip issue: BLO-23071
🔗 Paperclip issue: BLO-23094

[allyblockcast]

🔗 Paperclip issue: BLO-20396
🔗 Paperclip issue: BLO-23071
🔗 Paperclip issue: BLO-23094

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[kkroo]

Changes requested — do not merge

This patch reintroduces the failure mode fixed by BLO-20396: after a timer expires, it deletes the same-agent lock while the original critical section continues, allowing concurrent queue mutation.

The investigation found no never-settling await, while live recovery fits the documented terminal-run / lingering-Job gate. The narrow, reviewed replacement is PR #1110 (head b85cad1), already in the merge queue; it makes the Job liveness guard run-aware while preserving the active/terminating-Pod fence and exact-identity deletion.

Please close this PR as superseded by #1110. Do not replace the safe liveness fix with an unprovable timer bypass.

[kkroo]

Closed as superseded by the approved, merge-queued run-aware liveness fix in #1110. This PR must not replace same-agent mutual exclusion with a timer bypass; the branch is retained for audit.