Skip to content

feat: make ship refill event-driven with a concurrency floor - #9

Merged
sparkus merged 9 commits into
mainfrom
fm/fm-completion-refill-trigger
Aug 4, 2026
Merged

feat: make ship refill event-driven with a concurrency floor#9
sparkus merged 9 commits into
mainfrom
fm/fm-completion-refill-trigger

Conversation

@sparkus

@sparkus sparkus commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Intent

Make work refill EVENT-DRIVEN instead of a rule an agent must remember mid-turn, and add a concurrency floor as an independent safety net. This is firstmate's own shared tracked material.

Problem (observed twice 2026-08-03): continuous refill (claim the next ticket the moment one lands) was a RULE the supervising agent had to execute. It failed twice when the agent entered a long single turn (2h08m, then 2h32m), landed work inside that turn, and never reached the claim-next step, starving ~60 ready tickets. Completion events already existed (fm-pr-check merge poll wake, and fm-teardown's printed backlog-refresh reminder) but neither triggered refill.

Build:

  1. Completion-triggered refill: on task completion (merge-poll wake and/or teardown), emit a durable explicit refill signal for that home (actionable without agent memory, survives restart like other wake records).
  2. Concurrency floor independent of the event: configurable target of concurrent live ships per home; when live count drops below it, surface a top-up signal. Follow config/ conventions (LOCAL, gitignored, absent = sensible default/feature-off); document in docs/configuration.md.
  3. Idempotence critical: completion events re-fire (afk re-escalated same done: line thrice). Dedupe per task per completion so repeated events cannot cause a second dispatch.
  4. Must NOT blind-spawn: signal triggers normal claim-and-dispatch (verify-at-pickup, atomic claim, exclusions, held and parked). Replaces "remembering to start", not the checks.
  5. Respect parked-work hazard until leases protect ship worktrees (fix-lease-ship-worktrees): refill must not trigger dispatch into a pool while a parked task holds unpushed commits. Coordinate rather than re-implement lease guard.

Stay inside completion signalling and the floor. Do not redesign the watcher, the wake queue, or the dispatch procedure itself.

Acceptance: double completion delivery yields exactly one refill signal; signal durable across restart; floor emits when below target and none when at/above; absent config safe and documented; existing teardown/merge-poll behaviour unchanged apart from the added signal.

What Changed

  • Emit durable, per-task-deduplicated refill wakes from merge-poll and teardown completion paths, with restart-safe pending supervision and explicit handled-cycle acknowledgement.
  • Add an optional per-home concurrency floor that counts authoritative live ship workers and surfaces top-up signals through the existing claim-and-dispatch flow.
  • Hold refill dispatch for unreadable or unpushed parked ship worktrees, and document and test the completion, floor, supervision, and recovery behavior.

Risk Assessment

✅ Low: Captain, the latest fixes correctly limit merge-triggered refill to ship and scout completions and acknowledge the expected refill before the retirement regression’s second watcher cycle; the full source review found no remaining material risk.

Testing

Baseline identity and clean-tree checks, focused refill and teardown suites, three merge-poll recovery cases, and an end-to-end CLI restart/floor/hazard probe all passed; the transcript provides reviewer-visible evidence, while no screenshot was applicable because this is a shell/watcher workflow with no rendered UI.

Evidence: Completion-refill end-to-end transcript

Shows duplicate completion dedupe, restart-safe drain, claim-cycle acknowledgement, below/at-target floor behavior, and parked-only safety hold.

Firstmate completion-refill end-to-end evidence
target_commit=c027394f34aef44b50f45a142ff5e6ec8901be93

COMPLETION EVENT (two independent process deliveries)
delivery_1_exit=0 delivery_2_exit=1 durable_rows=1 receipt="committed 1785793312"
queued_payload=check: refill completion task-demo: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn; after the claim-and-dispatch attempt, a successful ship spawn clears any completion need automatically; otherwise run bin/fm-refill-complete.sh <no-ready|no-eligible|held-only>
after_restart_drain_payload=check: refill completion task-demo: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn; after the claim-and-dispatch attempt, a successful ship spawn clears any completion need automatically; otherwise run bin/fm-refill-complete.sh <no-ready|no-eligible|held-only>
queue_rows_after_drain=0 completion_need_after_drain=yes
ordinary_claim_cycle_ack=refill claim-and-dispatch cycle completed: no-ready
completion_need_after_ack=no

CONCURRENCY FLOOR (one live ship, target two)
live_count=1 target=2 durable_rows=1
queued_payload=check: refill floor: live ships 1 below target 2; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn; after the claim-and-dispatch attempt, a successful ship spawn clears any completion need automatically; otherwise run bin/fm-refill-complete.sh <no-ready|no-eligible|held-only>
working_unpushed_ship_hold=no

PARKED-WORK HAZARD (same unpushed ship now parked)
queued_payload=check: refill floor: live ships 0 below target 2; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn; after the claim-and-dispatch attempt, a successful ship spawn clears any completion need automatically; otherwise run bin/fm-refill-complete.sh <no-ready|no-eligible|held-only>; HOLD: parked ship worktree safety is not proven (unpushed commits in ship-a) - do not pool-dispatch until leases protect ship worktrees or the state is proven clear
parked_unpushed_ship_hold=yes

CONCURRENCY FLOOR (two live ships, target two)
live_count=2 target=2 emit_exit=1 durable_rows=0

ABSENT CONFIG
parsed_floor=0 emit_exit=1 durable_rows=0
- Outcome: 🔧 1 issue found → auto-fixed (2) ✅ across 3 runs (21m58s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 5 issues found → auto-fixed (4) ✅
  • 🚨 bin/fm-teardown.sh:1558 - Intent requires completion refill to be “actionable without agent memory.” This call only appends a queue record after deleting the task metadata. If the last live task is cleaned up during the same long turn, the home becomes idle; existing supervision starts only for task metadata or X mode, and the watcher does not surface pre-existing queue rows. The turn can therefore end without another notification, leaving the original starvation reachable until restart or manual drain. Connect the refill record to the existing supervision notification boundary, including when no task metadata remains.
  • 🚨 bin/fm-watch.sh:1044 - Intent requires the floor to emit “when live count drops below it,” but autonomous evaluation occurs only inside a watcher heartbeat. A configured home starting with zero ships has no task metadata, so normal supervision is not armed and this check never runs. Evaluate the floor at a supported startup/turn-end boundary or make an enabled below-target floor an explicit supervision need.
  • 🚨 bin/fm-refill-lib.sh:92 - Intent specifies concurrent “live ships,” while this increments for every ship metadata file without checking lifecycle or endpoint state. A merged, dead, or parked task whose durable metadata remains is counted as live, so the independent floor can stay silent while real capacity is below target—the same starvation safety net it is meant to provide. Derive the count from the existing authoritative lifecycle/current-state boundary rather than metadata presence alone.
  • 🚨 bin/fm-refill-lib.sh:168 - Intent requires a restart-durable signal and exactly-once behavior, but the durable dedupe marker is created before the queue append. A crash between the claim and append leaves a marker with no signal, and restart delivery is then suppressed; additionally, cleanup ignores enqueue failure after removing the task state. Commit the signal and receipt through one crash-consistent boundary and distinguish “already emitted” from an actual write failure.
  • 🚨 bin/fm-refill-lib.sh:119 - Intent forbids refill dispatch into a pool while parked work has unpushed commits. This probe treats a failed Git read as clear by continuing, so an unreadable parked worktree can suppress the HOLD and permit normal dispatch despite unknown unpushed state. Coordinate with the lease/dispatch guard at the shared dispatch boundary, or conservatively treat unreadable state as a hold.

🔧 Fix: Captain, harden refill supervision and crash recovery
4 errors still open:

  • 🚨 bin/fm-watch.sh:717 - The required completion signal is still not actionable when teardown fires after the watcher has already started. This check runs only before the watcher loop; teardown can queue the final task’s refill, the healthy watcher never examines that queue or marker, and the turn-end guard permits stopping because the watcher is healthy. The original zero-metadata starvation remains reachable. Check the shared refill predicate inside the watcher cycle or force the existing cycle to rewake at turn end.
  • 🚨 bin/fm-refill-lib.sh:77 - The accepted live-count contract explicitly excludes dead endpoints, but every state: working result is counted. fm-crew-state.sh deliberately reports an attributed active run as working without checking endpoint liveness, so a dead worker with an active pipeline still satisfies the floor and can suppress top-up. Combine authoritative lifecycle state with the existing backend endpoint-liveness boundary.
  • 🚨 bin/fm-refill-lib.sh:99 - Intent requires holding refill when a parked task has unpushed commits, but this explicitly skips every local-only ship. Local-only ships use isolated pooled worktrees and can park awaiting merge approval with necessarily unpushed commits, so refill can advertise normal pool dispatch during exactly the forbidden hazard. Include parked local-only worktrees in the shared safety proof.
  • 🚨 bin/fm-spawn.sh:1727 - A handled refill can become an endless signal loop when normal pickup finds no eligible work: the completion-needed marker is cleared only after a successful ship spawn, while watcher startup re-enqueues refill-pending whenever that marker remains. This violates the accepted requirement to clear after the claim-and-dispatch cycle and can repeatedly wake or dispatch from one completion. Add a supported completion boundary for both successful spawn and handled no-ready/no-eligible outcomes.

🔧 Fix: Captain, close residual refill supervision gaps
2 errors still open:

  • 🚨 bin/fm-refill-lib.sh:78 - The accepted criterion says “A dead worker with a lingering pipeline must NOT count,” but this checks only fm_backend_target_exists. That helper explicitly treats a surviving tmux pane/bare shell as present even when its agent is dead; fm-crew-state can then report the lingering run as working, so the dead ship still satisfies the floor. Use the existing recovery-grade agent-state boundary to reject confirmed dead/missing agents before counting the working lifecycle result.
  • 🚨 bin/fm-refill-lib.sh:16 - The new unconditional fm-backend.sh dependency is absent from the copied-bin fixtures in tests/fm-turnend-guard.test.sh and tests/fm-claude-stop-autoarm.test.sh, which currently install only refill and wake libraries. Their hooks now source a nonexistent sibling before exercising supervision. Add fm-backend.sh and required adapter fixtures to both dependency bundles, or isolate the lightweight endpoint probe without adding this transitive source.

🔧 Fix: Captain, harden refill liveness and fixture isolation
2 errors still open:

  • 🚨 bin/fm-watch.sh:796 - Intent limits refill to “on task completion,” but this unconditional fm_refill_emit_completion &#34;$id&#34; also runs for kind=secondmate. Secondmates remain persistent after PR merges, so a routine merge incorrectly queues parent-home claim-next work. Apply the same secondmate exclusion used by teardown.
  • 🚨 tests/fm-pr-check-security.test.sh:2963 - The first watcher now leaves the expected refill wake queued, so this second watcher surfaces that refill at startup instead of reaching z-stop; the existing retirement regression test deterministically fails. Drain or acknowledge the expected refill before asserting the subsequent poll cycle.

🔧 Fix: Captain, scope merge refill and stabilize retirement
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed (2) ✅
  • 🚨 bin/fm-refill-lib.sh:111 - fm_refill_has_parked_unpushed examines every ship record without checking whether its lifecycle is parked. The end-to-end probe configured a floor of 2 with one authoritative live, working ship; that ship's ordinary unpushed commit caused the refill payload to add HOLD ... do not pool-dispatch. This prevents the floor from topping up during normal active development and broadens the required parked-work safeguard. Restrict the hold to parked ship worktrees, preserve fail-closed behavior for unreadable parked state, and add a regression proving a working unpushed ship does not hold while a parked unpushed ship does.
  • git status --short --branch; git rev-parse HEAD; git diff --name-status 2ef842cb530967264ba0f4835249f247b2c2fa40..3e56c8435492539ea4b5e22bf57e998143a2c406
  • bash tests/fm-refill.test.sh
  • Selected test_merged_poll_retires_once and test_persistent_secondmate_retirement_is_poll_only from tests/fm-pr-check-security.test.sh
  • Selected test_teardown_prompts_tasks_axi_done_when_compatible and test_teardown_manual_backend_prompts_hand_edit_even_when_tasks_axi_present from tests/fm-teardown.test.sh
  • Exercised real fm-refill-lib.sh and fm-wake-drain.sh calls for absent configuration, below-target floor emission, triple completion delivery, fresh-process replay, durable drain, and parked-unpushed safety; captured refill-cli-transcript.txt
  • Probed a floor target of 2 with one authoritative working ship carrying an unpushed commit; captured working-ship-overhold.txt
  • sed -n &#39;16,34p&#39; docs/configuration.md captured as configuration-excerpt.txt

🔧 Fix: Captain, restrict refill holds to parked ships
1 warning still open:

  • ⚠️ docs/configuration.md:40 - The documentation says every refill payload holds unless every ship worktree is free of unpushed commits. The repaired implementation and acceptance boundary hold only for parked ships, while unreadable lifecycle state remains conservative. Rewrite this paragraph to describe parked ship worktrees specifically.
  • bash tests/fm-refill.test.sh
  • bash tests/fm-teardown.test.sh
  • bash tests/fm-pr-check-security.test.sh
  • Manual isolated-home probe across fresh shell processes using fm_refill_emit_completion, fm_refill_emit_floor_if_needed, and bin/fm-wake-drain.sh
  • Reviewed the Ship concurrency floor section in docs/configuration.md
  • git status --short and git rev-parse HEAD

🔧 Fix: Captain, document parked-ship refill holds precisely
✅ Re-checked - no issues remain.

  • git status --short --branch && git rev-parse HEAD && git rev-parse 2ef842cb530967264ba0f4835249f247b2c2fa40
  • tests/fm-refill.test.sh
  • tests/fm-teardown.test.sh
  • Focused tests/fm-pr-check-security.test.sh selectors: test_merged_poll_retires_once, test_persistent_secondmate_retirement_is_poll_only, and test_retirement_crash_recovery
  • Inline end-to-end shell harness exercising two independent completion deliveries, restart drain, fm-refill-complete.sh no-ready, floor below/at target, absent configuration, and working-versus-parked unpushed ships
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

sparkus added 9 commits August 3, 2026 14:18
Completion (merge-poll and teardown) now enqueues one durable refill wake
per task instead of relying on a printed reminder the supervisor can miss
mid-turn. An optional config/concurrency-floor target independently
surfaces a top-up when live ships drop below the target. Both paths only
signal claim-and-dispatch; they never spawn and dedupe repeated completion
events so re-fires cannot double-dispatch.
@sparkus
sparkus merged commit c91deff into main Aug 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant