feat: make ship refill event-driven with a concurrency floor - #9
Merged
Conversation
Completion (merge-poll and teardown) now enqueues one durable refill wake per task instead of relying on a printed reminder the supervisor can miss mid-turn. An optional config/concurrency-floor target independently surfaces a top-up when live ships drop below the target. Both paths only signal claim-and-dispatch; they never spawn and dedupe repeated completion events so re-fires cannot double-dispatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Make work refill EVENT-DRIVEN instead of a rule an agent must remember mid-turn, and add a concurrency floor as an independent safety net. This is firstmate's own shared tracked material.
Problem (observed twice 2026-08-03): continuous refill (claim the next ticket the moment one lands) was a RULE the supervising agent had to execute. It failed twice when the agent entered a long single turn (2h08m, then 2h32m), landed work inside that turn, and never reached the claim-next step, starving ~60 ready tickets. Completion events already existed (fm-pr-check merge poll wake, and fm-teardown's printed backlog-refresh reminder) but neither triggered refill.
Build:
Stay inside completion signalling and the floor. Do not redesign the watcher, the wake queue, or the dispatch procedure itself.
Acceptance: double completion delivery yields exactly one refill signal; signal durable across restart; floor emits when below target and none when at/above; absent config safe and documented; existing teardown/merge-poll behaviour unchanged apart from the added signal.
What Changed
Risk Assessment
✅ Low: Captain, the latest fixes correctly limit merge-triggered refill to ship and scout completions and acknowledge the expected refill before the retirement regression’s second watcher cycle; the full source review found no remaining material risk.
Testing
Baseline identity and clean-tree checks, focused refill and teardown suites, three merge-poll recovery cases, and an end-to-end CLI restart/floor/hazard probe all passed; the transcript provides reviewer-visible evidence, while no screenshot was applicable because this is a shell/watcher workflow with no rendered UI.
Evidence: Completion-refill end-to-end transcript
Shows duplicate completion dedupe, restart-safe drain, claim-cycle acknowledgement, below/at-target floor behavior, and parked-only safety hold.Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 5 issues found → auto-fixed (4) ✅
bin/fm-teardown.sh:1558- Intent requires completion refill to be “actionable without agent memory.” This call only appends a queue record after deleting the task metadata. If the last live task is cleaned up during the same long turn, the home becomes idle; existing supervision starts only for task metadata or X mode, and the watcher does not surface pre-existing queue rows. The turn can therefore end without another notification, leaving the original starvation reachable until restart or manual drain. Connect the refill record to the existing supervision notification boundary, including when no task metadata remains.bin/fm-watch.sh:1044- Intent requires the floor to emit “when live count drops below it,” but autonomous evaluation occurs only inside a watcher heartbeat. A configured home starting with zero ships has no task metadata, so normal supervision is not armed and this check never runs. Evaluate the floor at a supported startup/turn-end boundary or make an enabled below-target floor an explicit supervision need.bin/fm-refill-lib.sh:92- Intent specifies concurrent “live ships,” while this increments for every ship metadata file without checking lifecycle or endpoint state. A merged, dead, or parked task whose durable metadata remains is counted as live, so the independent floor can stay silent while real capacity is below target—the same starvation safety net it is meant to provide. Derive the count from the existing authoritative lifecycle/current-state boundary rather than metadata presence alone.bin/fm-refill-lib.sh:168- Intent requires a restart-durable signal and exactly-once behavior, but the durable dedupe marker is created before the queue append. A crash between the claim and append leaves a marker with no signal, and restart delivery is then suppressed; additionally, cleanup ignores enqueue failure after removing the task state. Commit the signal and receipt through one crash-consistent boundary and distinguish “already emitted” from an actual write failure.bin/fm-refill-lib.sh:119- Intent forbids refill dispatch into a pool while parked work has unpushed commits. This probe treats a failed Git read as clear by continuing, so an unreadable parked worktree can suppress the HOLD and permit normal dispatch despite unknown unpushed state. Coordinate with the lease/dispatch guard at the shared dispatch boundary, or conservatively treat unreadable state as a hold.🔧 Fix: Captain, harden refill supervision and crash recovery
4 errors still open:
bin/fm-watch.sh:717- The required completion signal is still not actionable when teardown fires after the watcher has already started. This check runs only before the watcher loop; teardown can queue the final task’s refill, the healthy watcher never examines that queue or marker, and the turn-end guard permits stopping because the watcher is healthy. The original zero-metadata starvation remains reachable. Check the shared refill predicate inside the watcher cycle or force the existing cycle to rewake at turn end.bin/fm-refill-lib.sh:77- The accepted live-count contract explicitly excludes dead endpoints, but everystate: workingresult is counted.fm-crew-state.shdeliberately reports an attributed active run as working without checking endpoint liveness, so a dead worker with an active pipeline still satisfies the floor and can suppress top-up. Combine authoritative lifecycle state with the existing backend endpoint-liveness boundary.bin/fm-refill-lib.sh:99- Intent requires holding refill when a parked task has unpushed commits, but this explicitly skips everylocal-onlyship. Local-only ships use isolated pooled worktrees and can park awaiting merge approval with necessarily unpushed commits, so refill can advertise normal pool dispatch during exactly the forbidden hazard. Include parked local-only worktrees in the shared safety proof.bin/fm-spawn.sh:1727- A handled refill can become an endless signal loop when normal pickup finds no eligible work: the completion-needed marker is cleared only after a successful ship spawn, while watcher startup re-enqueuesrefill-pendingwhenever that marker remains. This violates the accepted requirement to clear after the claim-and-dispatch cycle and can repeatedly wake or dispatch from one completion. Add a supported completion boundary for both successful spawn and handled no-ready/no-eligible outcomes.🔧 Fix: Captain, close residual refill supervision gaps
2 errors still open:
bin/fm-refill-lib.sh:78- The accepted criterion says “A dead worker with a lingering pipeline must NOT count,” but this checks onlyfm_backend_target_exists. That helper explicitly treats a surviving tmux pane/bare shell as present even when its agent is dead;fm-crew-statecan then report the lingering run as working, so the dead ship still satisfies the floor. Use the existing recovery-grade agent-state boundary to reject confirmed dead/missing agents before counting the working lifecycle result.bin/fm-refill-lib.sh:16- The new unconditionalfm-backend.shdependency is absent from the copied-bin fixtures intests/fm-turnend-guard.test.shandtests/fm-claude-stop-autoarm.test.sh, which currently install only refill and wake libraries. Their hooks now source a nonexistent sibling before exercising supervision. Addfm-backend.shand required adapter fixtures to both dependency bundles, or isolate the lightweight endpoint probe without adding this transitive source.🔧 Fix: Captain, harden refill liveness and fixture isolation
2 errors still open:
bin/fm-watch.sh:796- Intent limits refill to “on task completion,” but this unconditionalfm_refill_emit_completion "$id"also runs forkind=secondmate. Secondmates remain persistent after PR merges, so a routine merge incorrectly queues parent-home claim-next work. Apply the same secondmate exclusion used by teardown.tests/fm-pr-check-security.test.sh:2963- The first watcher now leaves the expected refill wake queued, so this second watcher surfaces that refill at startup instead of reachingz-stop; the existing retirement regression test deterministically fails. Drain or acknowledge the expected refill before asserting the subsequent poll cycle.🔧 Fix: Captain, scope merge refill and stabilize retirement
✅ Re-checked - no issues remain.
🔧 **Test** - 1 issue found → auto-fixed (2) ✅
bin/fm-refill-lib.sh:111-fm_refill_has_parked_unpushedexamines every ship record without checking whether its lifecycle is parked. The end-to-end probe configured a floor of 2 with one authoritative live, working ship; that ship's ordinary unpushed commit caused the refill payload to addHOLD ... do not pool-dispatch. This prevents the floor from topping up during normal active development and broadens the required parked-work safeguard. Restrict the hold to parked ship worktrees, preserve fail-closed behavior for unreadable parked state, and add a regression proving a working unpushed ship does not hold while a parked unpushed ship does.git status --short --branch; git rev-parse HEAD; git diff --name-status 2ef842cb530967264ba0f4835249f247b2c2fa40..3e56c8435492539ea4b5e22bf57e998143a2c406bash tests/fm-refill.test.shSelectedtest_merged_poll_retires_onceandtest_persistent_secondmate_retirement_is_poll_onlyfromtests/fm-pr-check-security.test.shSelectedtest_teardown_prompts_tasks_axi_done_when_compatibleandtest_teardown_manual_backend_prompts_hand_edit_even_when_tasks_axi_presentfromtests/fm-teardown.test.shExercised realfm-refill-lib.shandfm-wake-drain.shcalls for absent configuration, below-target floor emission, triple completion delivery, fresh-process replay, durable drain, and parked-unpushed safety; capturedrefill-cli-transcript.txtProbed a floor target of 2 with one authoritative working ship carrying an unpushed commit; capturedworking-ship-overhold.txtsed -n '16,34p' docs/configuration.mdcaptured asconfiguration-excerpt.txt🔧 Fix: Captain, restrict refill holds to parked ships
1 warning still open:
docs/configuration.md:40- The documentation says every refill payload holds unless every ship worktree is free of unpushed commits. The repaired implementation and acceptance boundary hold only for parked ships, while unreadable lifecycle state remains conservative. Rewrite this paragraph to describe parked ship worktrees specifically.bash tests/fm-refill.test.shbash tests/fm-teardown.test.shbash tests/fm-pr-check-security.test.shManual isolated-home probe across fresh shell processes usingfm_refill_emit_completion,fm_refill_emit_floor_if_needed, andbin/fm-wake-drain.shReviewed theShip concurrency floorsection indocs/configuration.mdgit status --shortandgit rev-parse HEAD🔧 Fix: Captain, document parked-ship refill holds precisely
✅ Re-checked - no issues remain.
git status --short --branch && git rev-parse HEAD && git rev-parse 2ef842cb530967264ba0f4835249f247b2c2fa40tests/fm-refill.test.shtests/fm-teardown.test.shFocusedtests/fm-pr-check-security.test.shselectors:test_merged_poll_retires_once,test_persistent_secondmate_retirement_is_poll_only, andtest_retirement_crash_recoveryInline end-to-end shell harness exercising two independent completion deliveries, restart drain,fm-refill-complete.sh no-ready, floor below/at target, absent configuration, and working-versus-parked unpushed ships✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.