Skip to content

fix(watcher): keep a live idle worker's declared pause on the bounded cadence - #5

Merged
ecloin merged 1 commit into
mainfrom
fm/watcher-paused-alive-stale
Aug 11, 2026
Merged

fix(watcher): keep a live idle worker's declared pause on the bounded cadence#5
ecloin merged 1 commit into
mainfrom
fm/watcher-paused-alive-stale

Conversation

@ecloin

@ecloin ecloin commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Intent

A worker that declares paused: (or carries a durable captain-held hold) and then idles with its agent still up was never absorbed into the long-cadence pause recheck. The watcher emitted a bare stale: <window> wake on every escalation window, forever, and each one cost a firstmate handling turn.

Observed live on 2026-08-11: a worker appended paused: awaiting captain round-2 review and idled with its agent alive (deliberately kept up to answer inline review comments). fm-crew-state read it correctly as state: paused · source: status-log, and the watcher still queued bare stale: default:w1B:pP wakes at ~60-180s intervals.

Root cause

pause_state_class in bin/fm-watch.sh reconciles a declared hold against the authoritative crew state. On both of its paths it read backend agent liveness and, for an ordinary (non-secondmate) crew, returned none whenever the agent was not confidently dead - overriding an authoritative paused verdict.

That produced a self-sustaining loop:

  1. none routes the stale to surface_nonterminal_stale, which emits the bare stale: <window> wake and seeds .paused-<key> / .paused-rechecked-<key>.
  2. The next poll hits the fresh-marker fast path, which re-ran the same liveness veto and returned none again.
  3. Any pane churn (a ticking clock, a token counter) changed the hash, and the changed-hash branch's none cleared all pause tracking - including the .paused-resurfaced-<key> throttle - so the next stable hash surfaced bare again.

handle_paused_stale, the intended absorb path with its long cadence and throttled annotated resurface, was unreachable for a live agent.

The invariant

Agent liveness may only RECOVER a paused classification that the authoritative state no longer reports; it may never veto one that it does.

The old code conflated two opposite questions behind one liveness read. Split by what fm-crew-state actually reports:

  • working (an active run step or busy pane) still wins over the declared wait, whatever the log says, so a worker that resumed is handled normally.
  • paused is honored regardless of agent liveness. An idle-but-live agent is the normal shape of a declared external wait or captain hold: the worker stays up precisely so it can answer when the wait clears.
  • Anything else means fm-crew-state has degraded to stopped or unknown while the log still claims a hold. That is the recovery case the dead-agent gate was written for, and it keeps it: only a confidently dead ordinary crew may still be read as the declared hold, and a live or unreadable agent under that degraded state stays surfaced as genuinely inconclusive.

This also puts the three supervision paths on one rule. The Herdr event fast-path (handle_push_transition) and the away-mode daemon already absorbed a declared pause with no liveness gate; only the watcher's poll loop had the veto.

Protections preserved

  • Resume: a status line that stops being a hold clears the cadence at the top of the stale loop, and a working verdict still clears it and restarts the wedge timer.
  • Genuine wedge: a busy pane is never on this path at all - busy_turn_over_age routes a busy worker with no completed turn through the same wedge timer whatever its status log says.
  • Dead-agent recovery and secondmate handling are unchanged; a secondmate keeps the authoritative verdict and its endpoint liveness is never read.

Side effect

The fast-path recheck window now short-circuits before both the crew-state read and the backend read, so a deliberately idle pane no longer pays a backend liveness probe every poll. A full recheck still runs every FM_STALE_ESCALATE_SECS, so a resume with no status append is picked up within that bound.

What changed

  • bin/fm-watch.sh: pause_state_class honors an authoritative paused verdict regardless of agent liveness, and applies the dead-agent gate only to the recovery case. Comments state the invariant at the function and at the stale-triage call site.
  • docs/architecture.md: the supervision contract now says the cadence follows authoritative state, and that liveness only recovers a degraded hold.
  • tests/fm-watch-triage.test.sh: new coverage, below.

Testing

bin/fm-lint.sh clean (ShellCheck 0.11.0, pinned): bin/fm-watch.sh, tests/fm-watch-triage.test.sh.
bin/fm-doc-audience-check.sh clean.
tests/fm-watch-triage.test.sh green end to end (48 assertions).
bin/fm-test-run.sh --changed over the watcher/wake/daemon/lock family green.

New tests, all driving a real fm-watch.sh subprocess against fixture state dirs (no live state/ touched):

  • test_live_agent_declared_pause_absorbed_then_resurfaced - the regression. A live idle agent under a declared pause is absorbed with a residual wedge timer discarded rather than fired; stays absorbed across pane churn (the exact step that used to reset the cadence and re-emit a bare stale); re-surfaces once past the long cadence with the annotated awaiting external reason and no possible wedge label; and re-arms inside the throttle window append nothing further. Zero bare stale: <window> records are asserted against the durable queue throughout.
  • test_live_agent_degraded_state_under_declared_pause_surfaces - the disconfirming case. A live agent whose authoritative state degraded to stopped while the log still claims a hold must still surface.
  • test_exited_declared_pause_is_bounded - the pre-existing dead-agent and captain-held recovery coverage, kept intact. Its third sub-case, which asserted that a live paused gate surfaces on first sight, is the behavior this PR deliberately changes; it is replaced by the two tests above.

Working-resume paths are already covered by test_nonterminal_paused_rechecks_authoritative_state, test_paused_authoritative_working_preserves_wedge_timer, and test_nonterminal_stale_pause_transitions_reclassify_unchanged_hash, all still green.

The regression test was verified to bite: run against the pre-fix bin/fm-watch.sh it fails with exactly the reported symptom.

not ok - watcher surfaced a live-agent declared pause instead of absorbing it: stale: test:fm-livehold

… cadence

A crew that appended `paused:` (or carried a durable `captain-held` hold) and
then idled with its agent still up was forced to the `none` absorb class on
every poll: pause_state_class read backend agent liveness and vetoed the
authoritative `paused` verdict unless the agent was confidently dead. `none`
surfaced a bare `stale: <window>` wake and re-seeded the pause markers, and any
pane churn (a ticking clock, a token counter) cleared the pause cadence
entirely, so the next stable hash surfaced bare again - one firstmate turn per
escalation window, forever. Observed live at ~60-180s intervals against a worker
that was deliberately idling to answer review comments.

Split the two directions that liveness read was conflating. Agent liveness may
only RECOVER a paused classification the authoritative state no longer reports;
it may never veto one that it does:

- `working` (active run step or busy pane) still wins over the declared wait, so
  a crew that resumed is handled normally;
- `paused` straight from fm-crew-state is now honored whether the agent is live
  or exited, because an idle-but-live agent is the normal shape of a declared
  wait - the worker stays up to answer when it clears;
- anything else means fm-crew-state has degraded to stopped or unknown, and only
  a confidently dead ordinary crew may still be read as the declared hold. A
  live or unreadable agent under that degraded state stays surfaced.

Secondmate handling is unchanged: the parent keeps the authoritative verdict and
never reads its endpoint's liveness. The event fast-path and away-mode daemon
already absorbed a declared pause with no liveness gate, so this puts all three
supervision paths on one rule.

Dropping the veto also removes a per-poll backend read from a deliberately idle
pane: the recheck window now short-circuits before both the state and backend
reads, and a full recheck still runs every FM_STALE_ESCALATE_SECS.
@ecloin

ecloin commented Aug 11, 2026

Copy link
Copy Markdown
Owner Author

CI note: two red checks, neither from this change.

  • PR must be raised via no-mistakes fails by design - this PR was raised on the direct-PR path, so it carries no no-mistakes signature.
  • Behavior tests (Herdr) fails on tests/fm-backend-herdr-presentation-e2e.test.sh: not ok - fm-hibit-resume-r1 restart fixture unexpectedly retained a registered agent. The same job fails with the identical assertion on PR fix(fleet-view): clip sidebar entries to a glanceable headline #4's run (job 93817494104), which touched no watcher or Herdr code, so it is pre-existing and unrelated to the pause-cadence fix.

Every lane that covers this change is green: Lint shell scripts, Test coverage guard, Behavior portable parallel 1-2, Behavior portable serial 1-4, Stock macOS Bash snapshot compatibility, and Repo invariants.

@ecloin
ecloin merged commit 335abe0 into main Aug 11, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant