fix(watcher): keep a live idle worker's declared pause on the bounded cadence - #5
Merged
Merged
Conversation
… cadence A crew that appended `paused:` (or carried a durable `captain-held` hold) and then idled with its agent still up was forced to the `none` absorb class on every poll: pause_state_class read backend agent liveness and vetoed the authoritative `paused` verdict unless the agent was confidently dead. `none` surfaced a bare `stale: <window>` wake and re-seeded the pause markers, and any pane churn (a ticking clock, a token counter) cleared the pause cadence entirely, so the next stable hash surfaced bare again - one firstmate turn per escalation window, forever. Observed live at ~60-180s intervals against a worker that was deliberately idling to answer review comments. Split the two directions that liveness read was conflating. Agent liveness may only RECOVER a paused classification the authoritative state no longer reports; it may never veto one that it does: - `working` (active run step or busy pane) still wins over the declared wait, so a crew that resumed is handled normally; - `paused` straight from fm-crew-state is now honored whether the agent is live or exited, because an idle-but-live agent is the normal shape of a declared wait - the worker stays up to answer when it clears; - anything else means fm-crew-state has degraded to stopped or unknown, and only a confidently dead ordinary crew may still be read as the declared hold. A live or unreadable agent under that degraded state stays surfaced. Secondmate handling is unchanged: the parent keeps the authoritative verdict and never reads its endpoint's liveness. The event fast-path and away-mode daemon already absorbed a declared pause with no liveness gate, so this puts all three supervision paths on one rule. Dropping the veto also removes a per-poll backend read from a deliberately idle pane: the recheck window now short-circuits before both the state and backend reads, and a full recheck still runs every FM_STALE_ESCALATE_SECS.
Owner
Author
|
CI note: two red checks, neither from this change.
Every lane that covers this change is green: Lint shell scripts, Test coverage guard, Behavior portable parallel 1-2, Behavior portable serial 1-4, Stock macOS Bash snapshot compatibility, and Repo invariants. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
A worker that declares
paused:(or carries a durablecaptain-heldhold) and then idles with its agent still up was never absorbed into the long-cadence pause recheck. The watcher emitted a barestale: <window>wake on every escalation window, forever, and each one cost a firstmate handling turn.Observed live on 2026-08-11: a worker appended
paused: awaiting captain round-2 reviewand idled with its agent alive (deliberately kept up to answer inline review comments).fm-crew-stateread it correctly asstate: paused · source: status-log, and the watcher still queued barestale: default:w1B:pPwakes at ~60-180s intervals.Root cause
pause_state_classinbin/fm-watch.shreconciles a declared hold against the authoritative crew state. On both of its paths it read backend agent liveness and, for an ordinary (non-secondmate) crew, returnednonewhenever the agent was not confidently dead - overriding an authoritativepausedverdict.That produced a self-sustaining loop:
noneroutes the stale tosurface_nonterminal_stale, which emits the barestale: <window>wake and seeds.paused-<key>/.paused-rechecked-<key>.noneagain.nonecleared all pause tracking - including the.paused-resurfaced-<key>throttle - so the next stable hash surfaced bare again.handle_paused_stale, the intended absorb path with its long cadence and throttled annotated resurface, was unreachable for a live agent.The invariant
Agent liveness may only RECOVER a paused classification that the authoritative state no longer reports; it may never veto one that it does.
The old code conflated two opposite questions behind one liveness read. Split by what
fm-crew-stateactually reports:working(an active run step or busy pane) still wins over the declared wait, whatever the log says, so a worker that resumed is handled normally.pausedis honored regardless of agent liveness. An idle-but-live agent is the normal shape of a declared external wait or captain hold: the worker stays up precisely so it can answer when the wait clears.fm-crew-statehas degraded to stopped or unknown while the log still claims a hold. That is the recovery case the dead-agent gate was written for, and it keeps it: only a confidently dead ordinary crew may still be read as the declared hold, and a live or unreadable agent under that degraded state stays surfaced as genuinely inconclusive.This also puts the three supervision paths on one rule. The Herdr event fast-path (
handle_push_transition) and the away-mode daemon already absorbed a declared pause with no liveness gate; only the watcher's poll loop had the veto.Protections preserved
workingverdict still clears it and restarts the wedge timer.busy_turn_over_ageroutes a busy worker with no completed turn through the same wedge timer whatever its status log says.Side effect
The fast-path recheck window now short-circuits before both the crew-state read and the backend read, so a deliberately idle pane no longer pays a backend liveness probe every poll. A full recheck still runs every
FM_STALE_ESCALATE_SECS, so a resume with no status append is picked up within that bound.What changed
bin/fm-watch.sh:pause_state_classhonors an authoritativepausedverdict regardless of agent liveness, and applies the dead-agent gate only to the recovery case. Comments state the invariant at the function and at the stale-triage call site.docs/architecture.md: the supervision contract now says the cadence follows authoritative state, and that liveness only recovers a degraded hold.tests/fm-watch-triage.test.sh: new coverage, below.Testing
bin/fm-lint.shclean (ShellCheck 0.11.0, pinned):bin/fm-watch.sh,tests/fm-watch-triage.test.sh.bin/fm-doc-audience-check.shclean.tests/fm-watch-triage.test.shgreen end to end (48 assertions).bin/fm-test-run.sh --changedover the watcher/wake/daemon/lock family green.New tests, all driving a real
fm-watch.shsubprocess against fixture state dirs (no livestate/touched):test_live_agent_declared_pause_absorbed_then_resurfaced- the regression. A live idle agent under a declared pause is absorbed with a residual wedge timer discarded rather than fired; stays absorbed across pane churn (the exact step that used to reset the cadence and re-emit a bare stale); re-surfaces once past the long cadence with the annotatedawaiting externalreason and nopossible wedgelabel; and re-arms inside the throttle window append nothing further. Zero barestale: <window>records are asserted against the durable queue throughout.test_live_agent_degraded_state_under_declared_pause_surfaces- the disconfirming case. A live agent whose authoritative state degraded to stopped while the log still claims a hold must still surface.test_exited_declared_pause_is_bounded- the pre-existing dead-agent and captain-held recovery coverage, kept intact. Its third sub-case, which asserted that a live paused gate surfaces on first sight, is the behavior this PR deliberately changes; it is replaced by the two tests above.Working-resume paths are already covered by
test_nonterminal_paused_rechecks_authoritative_state,test_paused_authoritative_working_preserves_wedge_timer, andtest_nonterminal_stale_pause_transitions_reclassify_unchanged_hash, all still green.The regression test was verified to bite: run against the pre-fix
bin/fm-watch.shit fails with exactly the reported symptom.