fix(watch): honor a declared wait on a busy over-age pane - #1898
Open
Ndidi wants to merge 1 commit into
Open
Conversation
A worker running one long bounded command - a full suite, a render sweep, a polled lane - kept wedge-escalating every FM_STALE_ESCALATE_SECS for the whole run, and a declared paused: could not stop it. The most recent occurrence produced seven consecutive escalations on a healthy worker, every one recorded as "busy (no completed turn)". The declared wait was masked at two independent layers. Every sibling pause branch in the watcher's stale loop is gated on a NON-busy pane, and fm-crew-state.sh reads the busy verdict before its status-log fallback, so a busy worker's authoritative state is `working` and pause_state_class can never answer `paused` for it. The busy over-age path therefore ran wedge_timer_check with no pause consultation at all, which is why a late pause appeared to be deliberately ignored rather than structurally unreachable. Route that path through busy_over_age_check, which absorbs a declared wait onto the existing bounded pause cadence and otherwise leaves the wedge timer exactly as it was. The pause is read from the status log rather than through pause_state_class, whose liveness reconciliation a busy verdict already outranks. The duration bound BUSY_TURN_MAX_SECS exists to enforce is preserved rather than removed: handle_paused_stale still re-surfaces the pane every PAUSE_RESURFACE_SECS, so a forgotten or dishonest pause is rechecked on a long cadence instead of never.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Supervision raised repeated false wedge alarms on healthy workers running one long bounded command.
The most recent occurrence produced seven consecutive escalations on a worker that was polling a >2h test lane, and a declared
paused:did not stop them.What the evidence actually showed
The recorded triage reasons for that worker were all
busy (no completed turn), and its escalation counter held exactly7, with no.paused-marker ever written.That is the signature of the
busy_turn_over_agepath, not of the ordinary stale path.The declared wait was masked at two independent layers:
handle_paused_stalewas unreachable while the pane was busy.fm-crew-state.shreads the busy verdict before its status-log fallback, so a busy worker's authoritative state isworking · source: paneandpause_state_classcan never answerpausedfor it.So the busy over-age path called
wedge_timer_checkwith no pause consultation at all.A late pause looked deliberately ignored, but it was structurally unreachable.
The fix
Route that path through a new
busy_over_age_check, which absorbs a declared wait onto the existing bounded pause cadence and otherwise leaves the wedge timer exactly as it was.The pause is read straight from the status log rather than through
pause_state_class, whose liveness reconciliation (agent not dead) a busy verdict already outranks.Both call sites also stop clearing pause bookkeeping that an absorb just wrote, which would otherwise destroy the re-surface throttle and fire the pause cadence every poll.
The safety bound is preserved, not removed
BUSY_TURN_MAX_SECSexists because a genuinely hung foreground call behind a busy signature once ran undetected for 25h.That bound still holds:
handle_paused_stalere-surfaces the pane everyPAUSE_RESURFACE_SECS, so a forgotten or dishonest pause is rechecked on a long cadence instead of never.This also aligns the watcher with the away-mode daemon, whose documented contract already gives a declared wait the long recheck and "never a wedge escalation".
Shapes considered and rejected
bin/fm-busy-lib.shdeliberately excludes, and it needs a redirect target firstmate does not record.BUSY_TURN_MAX_SECS. Rejected: any value large enough for a >2h lane hides a genuinely hung worker for that long. It trades one failure for another and ignores the worker's own declaration.Testing
tests/fm-watch-triage.test.sh, both pinningFM_FAKE_CREW_STATEto theworking · source: paneverdict a real busy pane produces, so a fix routed back throughpause_state_classfails them.did not record the paused flag) and pass with it.tests/fm-watch-triage.test.shsuite green, 52/52.bin/fm-lint.shclean over the full file set;bin/fm-doc-audience-check.shok.