Skip to content

fix(watch): honor a declared wait on a busy over-age pane - #1898

Open
Ndidi wants to merge 1 commit into
kunchenguid:mainfrom
Ndidi:fm/watcher-long-render-false-wedge
Open

fix(watch): honor a declared wait on a busy over-age pane#1898
Ndidi wants to merge 1 commit into
kunchenguid:mainfrom
Ndidi:fm/watcher-long-render-false-wedge

Conversation

@Ndidi

@Ndidi Ndidi commented Aug 7, 2026

Copy link
Copy Markdown

The problem

Supervision raised repeated false wedge alarms on healthy workers running one long bounded command.
The most recent occurrence produced seven consecutive escalations on a worker that was polling a >2h test lane, and a declared paused: did not stop them.

What the evidence actually showed

The recorded triage reasons for that worker were all busy (no completed turn), and its escalation counter held exactly 7, with no .paused- marker ever written.
That is the signature of the busy_turn_over_age path, not of the ordinary stale path.

The declared wait was masked at two independent layers:

  1. Every sibling pause branch in the watcher's stale loop is gated on a non-busy pane, so handle_paused_stale was unreachable while the pane was busy.
  2. fm-crew-state.sh reads the busy verdict before its status-log fallback, so a busy worker's authoritative state is working · source: pane and pause_state_class can never answer paused for it.

So the busy over-age path called wedge_timer_check with no pause consultation at all.
A late pause looked deliberately ignored, but it was structurally unreachable.

The fix

Route that path through a new busy_over_age_check, which absorbs a declared wait onto the existing bounded pause cadence and otherwise leaves the wedge timer exactly as it was.
The pause is read straight from the status log rather than through pause_state_class, whose liveness reconciliation (agent not dead) a busy verdict already outranks.

Both call sites also stop clearing pause bookkeeping that an absorb just wrote, which would otherwise destroy the re-surface throttle and fire the pause cadence every poll.

The safety bound is preserved, not removed

BUSY_TURN_MAX_SECS exists because a genuinely hung foreground call behind a busy signature once ran undetected for 25h.
That bound still holds: handle_paused_stale re-surfaces the pane every PAUSE_RESURFACE_SECS, so a forgotten or dishonest pause is rechecked on a long cadence instead of never.
This also aligns the watcher with the away-mode daemon, whose documented contract already gives a declared wait the long recheck and "never a wedge escalation".

Shapes considered and rejected

  • A fresh-mtime beacon on whatever the parked command writes. Rejected on the evidence: the worker was already provably busy on every one of those escalations, so a weaker liveness signal would have changed nothing. It would also reintroduce marker mtimes as a state signal, which the semantic busy-state contract in bin/fm-busy-lib.sh deliberately excludes, and it needs a redirect target firstmate does not record.
  • Raising BUSY_TURN_MAX_SECS. Rejected: any value large enough for a >2h lane hides a genuinely hung worker for that long. It trades one failure for another and ignores the worker's own declaration.

Testing

  • Two new portable regressions in tests/fm-watch-triage.test.sh, both pinning FM_FAKE_CREW_STATE to the working · source: pane verdict a real busy pane produces, so a fix routed back through pause_state_class fails them.
  • One asserts the divergence directly: withdraw the declared wait and the same pane wedge-escalates again, so the case cannot pass vacuously.
  • The other proves the bound is preserved by aging the pause past its window and requiring a paused recheck, never a wedge.
  • Verified these fail without the fix (did not record the paused flag) and pass with it.
  • Full tests/fm-watch-triage.test.sh suite green, 52/52.
  • bin/fm-lint.sh clean over the full file set; bin/fm-doc-audience-check.sh ok.

A worker running one long bounded command - a full suite, a render sweep, a
polled lane - kept wedge-escalating every FM_STALE_ESCALATE_SECS for the whole
run, and a declared paused: could not stop it. The most recent occurrence
produced seven consecutive escalations on a healthy worker, every one recorded
as "busy (no completed turn)".

The declared wait was masked at two independent layers. Every sibling pause
branch in the watcher's stale loop is gated on a NON-busy pane, and
fm-crew-state.sh reads the busy verdict before its status-log fallback, so a
busy worker's authoritative state is `working` and pause_state_class can never
answer `paused` for it. The busy over-age path therefore ran wedge_timer_check
with no pause consultation at all, which is why a late pause appeared to be
deliberately ignored rather than structurally unreachable.

Route that path through busy_over_age_check, which absorbs a declared wait onto
the existing bounded pause cadence and otherwise leaves the wedge timer exactly
as it was. The pause is read from the status log rather than through
pause_state_class, whose liveness reconciliation a busy verdict already
outranks.

The duration bound BUSY_TURN_MAX_SECS exists to enforce is preserved rather than
removed: handle_paused_stale still re-surfaces the pane every
PAUSE_RESURFACE_SECS, so a forgotten or dishonest pause is rechecked on a long
cadence instead of never.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant