feat(goal-conductor): durable inbox and dispatch record - #8489
feat(goal-conductor): durable inbox and dispatch record#8489iamwhatever wants to merge 1 commit into
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound durability mechanics, but the dispatch record has no lifecycle: no round scoping and no removal verb, so it eventually mis-replays or wedges. Watch
Suggestions
[DESIGN-REVIEWED] 908384b |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've validated all four candidates against the code and looked for grounded defects of my own. Candidate 1 (dispatch MAX_ITEMS never pruned): The cap is 200; a round is 2–3 items per the skill. Reaching a lifetime 200 live dispatch records requires assuming goals that never close out — the candidate itself cannot establish (a) that this occurs in practice. Capacity design limitation, not a concrete defect. Killed. Candidate 2 (full inbox counts claimed entries): Requires 200 residual claimed-but-never- Candidate 3 (same-second sidecar collision): Only affects forensic recovery of already-corrupt bytes, and the normal path (quarantine → write fresh valid file) makes back-to-back same-second corruption unnecessary to reach. No observable wrong outcome for live data. Killed. Candidate 4 (small Step 2: I examined the lock steal/fence logic (TOCTOU on steal is made non-destructive by the token fence and double-check in No findings. [OPUS-REVIEWED] 908384b Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/builtin_skills/goal-conductor/SKILL.md:108 -- Dispatch IDs cannot recover the created session (origin: validation) BLOCKING -- src/kiro_crew/builtin_skills/goal-conductor/scripts/dispatch_queue.py:248 -- Malformed mapping entries still crash queue operations (origin: validation) BLOCKING -- src/kiro_crew/builtin_skills/goal-conductor/scripts/dispatch_queue.py:496 -- Stale-lock takeover can unlink a new owner's lock (origin: validation) BLOCKING -- src/kiro_crew/builtin_skills/goal-conductor/scripts/dispatch_queue.py:202 -- Quarantine names can overwrite recoverable state FINDING -- src/kiro_crew/builtin_skills/goal-conductor/scripts/dispatch_queue.py:687 -- FINDING -- src/kiro_crew/builtin_skills/goal-conductor/scripts/dispatch_queue.py:44 -- [BLOCK-MERGE] 908384b |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All checks are done. Producing the review. First-Principles-Verdict: CONCERNS Both durable records answer a named, real loss and declare their costs; the one straggler is a What this change shipsIntent: stop a compaction or lost turn from silently discarding a user's mid-flight steering message or double-dispatching a work item — an ADDITION (a durable store) motivated by two named defects.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 908384b |
49a60bf to
1add78a
Compare
Both questions answered yes: the finding holds against the code, and the fix is
Coverage: Note: the script was renamed |
Both questions answered yes:
A new Fixing this also closed a latent bug the finding did not name: the lock's own Coverage: class |
Both questions answered yes. The fix does exactly the two things the finding
Verify the owner is dead. A steal past Fence lock ownership. Each holder writes a Coverage: |
Both questions answered yes, and the fix is literally the one the finding named —
Every The same pass added |
Rule 3 of That call site is already gated the way the check's own error message prescribes:
Routing through So the line is correct as written and there is no in-scope change that makes the scan pass honestly. Per the check's own remedy — "if this code path is intentionally POSIX-only and documented as such -- add the |
1add78a to
b74424c
Compare
Confirmed, and worse than advisory: the loop never terminates. The Fix: only a successful unlink retries. if self._may_steal(age):
try:
self.path.unlink()
except OSError:
# The steal was decided but could not be carried out ...
# An un-removable lock must converge on `locked`, not pin a core forever.
pass
else:
continue
if _now() >= deadline:
raise TimeoutError(...)
time.sleep(0.05)Covered by The test was verified against the bug before being trusted: with the unconditional |
The conductor held two things only in its own context: a user message that arrived mid-round, and whether an item had already been dispatched. Both are lost on compaction or a dropped turn — the message silently, and the dispatch as a second session for one item. `scripts/queue.py` moves both onto disk, one state file per goal, behind an `O_EXCL` lock with staleness steal so a crashed turn cannot wedge the goal. A claim does not delete, a full inbox refuses rather than evicting, and `dispatch_begin` returns the SAME id for an item already begun so a retry converges on one session instead of opening a second.
b74424c to
908384b
Compare
Confirmed, and it defeats the quarantine's whole purpose. The shape check went one
Fix: the entries are checked too, not just the shape holding them. if (
not isinstance(data, dict)
or not isinstance(data.get("inbox"), list)
or not isinstance(data.get("dispatch"), dict)
or not all(isinstance(entry, dict) for entry in data["inbox"])
or not all(isinstance(rec, dict) for rec in data["dispatch"].values())
):
_quarantine(path)
return _blank(goal)Coverage: three nested bodies added to Verified against the bug: with the nested checks reverted, the parametrized cases |
Confirmed. The write path already did the hard part — loop
Fix: a os.replace(str(tmp), str(path))
_fsync_dir(path.parent)Deliberately best-effort, and the reasoning is in the docstring: a directory is not Coverage: Verified against the bug: with |
Confirmed, and this one hollows out the fence rather than merely narrowing it.
The single check sat on entry to Fix: the load-bearing check is the second one, immediately before the rename. os.close(fd)
if fence is not None and not fence.still_held():
try:
tmp.unlink()
except OSError:
pass
raise _LockLost(f"{fence.path} was taken while this write was being flushed")
os.replace(str(tmp), str(path))The temp file is unlinked on the refusal path, so a lost fence does not leave Coverage: Verified against the bug: with the second check reverted, the test fails |
Confirmed.
Fix: path = _state_path(goal)
with _Lock(path):
data = _read(path, goal)
exists = path.exists()Checked deadlock-free before applying: every Coverage: new Verified against the bug: with the |
|
I don't think this is the right way to coordinate agents, closing it. |
Problem
The
goal-conductorskill holds two facts nowhere but in the conductor's ownprompt and context:
mid-flight" section told the conductor to remember the message and apply it at
the next round boundary. Remembering is context, and context is what
compaction drops. The user typed a steering instruction and gets no
acknowledgement that it was lost.
session_createthensession_send— ordered only by prose ("send the seedBEFORE recording the ledger row"). A turn lost between them leaves a session
with no seed, and the next patrol cycle cannot tell that from a session that
is merely quiet. The recovery a conductor actually reaches for is a second
session_create, which is how one item becomes two sessions.Why it matters
Both failures are silent and both are user-visible. The first discards
something a human typed. The second doubles work under a lost turn — the exact
condition a long-running conductor hits most, since it deliberately ends its turn
and comes back through
monitor_start.Fix
Symptom: a mid-flight message and a half-finished dispatch both live only in
context.
Root cause: the skill has no durable store.
accept_eval.pyholds theacceptance verdict and
ledger_entry.pyholds the entry format, but nothingholds the queue, so the prose had to carry it.
Change: a third bundled script,
scripts/dispatch_queue.py, owning one statefile per goal at
<data_home>/conductor/<goal>/queue.json. Seven modes:enqueue,claim,done,release,dispatch_begin,dispatch_sent,status. Same invocation contract as its siblings — argv mode, one JSON on stdin,one JSON on stdout, exit 0 for "ran" and 2 for a malformed call.
The design choices that carry the properties:
claiming leaves the message on disk, and a later
claimpast the stalenesswindow re-serves it with a
reclaimedcount. Only an explicitdoneremovesit.
MAX_INBOXthe call returnsinbox_fullratherthan evicting the oldest — every entry is a message a human typed, so a
backlog is the better failure.
MAX_TEXTrefuses rather than truncating, forthe same reason: half a steering instruction can invert its meaning.
dispatch_beginis idempotent by id. A second call for the same itemreturns the SAME dispatch id and
replay: true, plus a warning naming theconsequence when no seed was recorded yet. A fresh id per attempt is precisely
what opens a second session.
would be written back by the next mode, destroying the parked messages the
script exists to keep.
to
queue.corrupt-<ts>.json, so the run continues and the bytes stayinspectable. Valid JSON with the wrong shape is quarantined on the same path
rather than being substituted with a blank record. The shape check goes one
level deep, not just the containers:
{"inbox": [null]}has a list where alist belongs, so a container-only check accepts it and every mode then calls
entry.get(...)on a non-mapping and dies with an uncaughtAttributeError—the exact crash the quarantine exists to prevent.
os.writeislooped until every byte lands and
os.replaceis skipped entirely after ashort write, so a full disk produces a structured refusal and leaves the
previous record byte-identical rather than publishing truncated JSON. After the
replace, the parent directory is fsynced —
fsyncon the file commits thebytes, not the rename that points at them, so power loss between the two loses
an enqueue the script already reported as succeeded. Best-effort by design: a
directory is not openable on Windows and some filesystems refuse to sync one,
neither of which is a reason to fail a write whose data is already committed.
O_EXCLlockfile with a liveness-gated, fenced steal, notfcntl.flock—the backend test suite runs on Windows, and
flockis not there. A stale lockis stolen only once its holder is proven dead; a holder that is still running
is never stolen from, and one that cannot be judged waits for a 900s backstop.
Ownership is fenced by a token, so a writer whose lock was stolen refuses at
publish time instead of clobbering the newer state. The fence is checked
twice, and the second check is the load-bearing one: checking only on entry
leaves the whole
write+fsyncwindow — the slowest part, and exactly wherea writer stalls long enough to look dead — unguarded, so a writer that passed
the entry check could still resume after the steal and
os.replaceits stalerecord over the newer one. The re-check sits immediately before the rename,
unlinks the temp file, and refuses with
lock_lost.status.statuswrites nothing, but
_readcan quarantine — so an unlocked read of oldcorruption, racing an
enqueuethat publishes valid state, renames that newstate aside. A mode that only means to look still needs the lock, because its
read is not side-effect-free.
cannot be judged times out with
lockedrather than spinning, and so does onethe steal decides on but cannot carry out — a lock in a directory this
process may not write to (a goal dir owned by another uid, a read-only mount)
still stats fine and still looks steal-worthy, so retrying on a failed
unlinkre-decides the same steal every pass with no sleep and no deadlinecheck. Only a successful unlink retries; a failure falls through to the
deadline.
execute_bashgetsno verifiable identity from the gateway, so anything it read would be an
assertion. Keeping it a plain local state machine is what makes it safe as a
script rather than an MCP tool, and a source ratchet in the tests pins that.
Goal ids are validated against
^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$, notsanitized: a goal id names a directory, and sanitizing two different ids into one
merges two goals' queues.
Supporting edits, all queue-only:
SKILL.md—dispatch_beginbefore the dispatch steps anddispatch_sentright after the seed; the
enqueue/claim/done/releasecycle in "Goalchanges mid-flight"; the added approval cost stated in "Known limits". The
dispatch list still OPENS with
session_create, so the atomic-filing propertysession_create should accept a folder so filing is atomic with creation #6118 established is untouched.
agent.py— the conductor prompt names three bundled scripts instead of two.Tests
test/test_conductor_queue.py, 50 cases, one class per load-bearing property:TestInvocationContract— unknown mode and a JSON array on stdin are exit 2 onstdout, not a traceback; every mode reports
state_path.TestGoalIdIsValidatedNotSanitized— seven rejected shapes, and../../etcnever reaches a path.
TestAParkedMessageSurvives— the dead-turn case asserted directly: claim,die, re-claim, get the text back. Plus idempotent
done,release, therefusal at the cap with the oldest still present, oversize refusal, and the
corrupt-file sidecar.
TestARetriedDispatchConverges— same id andreplay: trueon the secondbegin, the warning naming "no seed", no warning once sent,
unsent_itemsinstatus, and a
dispatch_sentwith no begin recorded-and-flagged rather thanrefused (the send already happened; refusing would make the record less true).
TestItTouchesNoIdentity— source ratchet on named session-key and credentialsymbols, and on
subprocess/urllib/socket.TestUnreadableIsNotAbsent— the unreadable file refuses and the bytes arebyte-identical afterwards; absent still reads as blank.
TestTheLockAlwaysConvergesOnAnAnswer— a live lock times out, a stat thatalways fails still honours the deadline without stealing, a stale lock whose
unlinkkeeps failing times out instead of spinning, a stale lock from aproven-dead holder is stolen, one whose holder still runs is not, and a
holder that cannot be judged is stolen only past the 900s backstop. The
liveness cases are POSIX-gated and use a genuinely reaped pid, because
os.kill(pid, 0)is only a non-destructive probe on POSIX. The un-removablecase asserts both halves of the bug it pins: the wall-clock stays inside the
wait budget and the retry count stays bounded, since the failure mode is an
unbounded retry loop rather than a slow one.
TestAPartialWriteIsNeverPublished— a short write refuses and leaves theprevious record byte-identical; a simulated full disk is reported as a
structured refusal rather than a crash. A stolen lock makes the in-flight write
refuse instead of clobbering the newer state. The fence re-check is driven by
stealing the lock from inside a patched
os.fsync, so the entry check hasalready passed when the theft lands — the only arrangement that can distinguish
one fence check from two; it asserts
lock_lost, that the newer recordsurvives, and that no
queue.tmp-*is left behind. Directory durability isasserted by recording fsyncs whose fd
fstats asS_IFDIR, because a plaincall-count assertion passes against the bug, and a companion case parametrized
over "the directory will not open" and "it opens but will not sync" pins the
best-effort half.
TestStatusNeverDestroysWhatItOnlyMeantToRead— a live foreign lock makesstatusreturn the structuredlockedrefusal, and the quarantine caseasserts the invariant rather than a timing coincidence:
_quarantinerecordswhether
queue.lockexists and carries this pid at the moment of the rename.test_structurally_invalid_state_is_quarantined_not_silently_dropped—parametrized over six malformed bodies, three of them nested (
inbox: [null],inbox: ["a string"],dispatch: {"item-1": "abc"}); asserts the.corrupt-<epoch>sidecar's bytes equal the original. A sibling case drivesstatusover a malformed record through the real subprocess, so a tracebackfails the test rather than being interpreted as output.
The module runs on Windows rather than being collect-ignored, because
portability is the reason
_LockusesO_EXCLat all — excluding it would leavethat claim untested. Only the cases that need a POSIX-only primitive to mean what
they test are skipped there:
0o000to MEAN unreadable (a mode-0 file staysreadable by its owner on Windows), and the non-destructive liveness probe (on
Windows
os.killterminates the target, so the probe returns "cannot bejudged" and the backstop path covers it instead).
Manual verification
Why the script is named
dispatch_queue.py, notqueue.pyThe obvious name shadows the stdlib
queuemodule. Because the script livesinside the package tree,
mypyresolvedimport queuein six unrelated files(
embeddings.py,cli.pyand others) to this file instead of the stdlib,producing 21 errors across the repo. The rename is what makes
mypy src/kiro_crew/green; nothing else about the script changed with it.
Relation to #6237
This is the queue half of #6237, rebased onto current
mainand split out on itsown. #6237 also collapsed the eight session-control verb tools into two
op-shaped tools, which #6109 has since made expensive: per-verb auto-approval now
matches on tool NAME, so collapsing the names re-opens a security decision that
is out of scope for a durable queue. That half is being abandoned; this half is
independent of it and does not touch
mcp_dashboard.py,channel.pyorvalidation.py.no linked issue: this is a split-out half of #6237 (a PR, not a tracked issue), and the durable-queue gap it closes was found while reviewing that PR rather than filed separately.