feat(pipeline-conductor): probe signals for no-progress, finished, and undelivered workers - #8035
Conversation
e252bc6 to
7f9d6bd
Compare
7f9d6bd to
dc42ce9
Compare
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of All checks done. The base already carried the contract (SKILL.md documents Design-Verdict: CONCERNS Sound, measured signal fixes — but three protocol semantics now ride special-cased carry-forwards on a one-entry state record the PR itself calls the wrong shape. Watch
[DESIGN-REVIEWED] 256b20a |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/builtin_skills/pipeline-conductor/scripts/fleet_probe.py:459 -- False positive or not applicable? A repository writer can comment: |
dc42ce9 to
f745244
Compare
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of Verified the counts I'll cite: First-Principles-Verdict: CONCERNS NOPROGRESS is a second spelling of the no-progress test SKILL.md already assigns to the conductor, and the consumer's action table has no row for it. What this change shipsIntent: make the fleet probe report worker progress and delivery truthfully instead of loudness — a FIX for six measured misreadings, carrying extensions.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 256b20a |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've reviewed the full source diff (the fleet_probe.py changes end around diff line 1168; everything after is test code) and the candidate. Falsifying CANDIDATE 1: The candidate claims the module docstring (diff lines 99–105) and the But this is a comment/documentation inaccuracy, not a defect in what the code does when executed:
Scored below 80; dropped. Step 2: No new grounded defect surfaced while falsifying. The classification logic ( No findings. [OPUS-REVIEWED] 256b20a Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
f745244 to
65d2158
Compare
|
GPT's blocking finding is correct and accepted, fixed in Reproduced before changing anything. The branch was gated on The prescribed fix is also the one the design called for, so it is applied verbatim: the The clock is deliberately left governing the other case: a 75 tests pass, and per-file coverage on the script holds at 95%. |
65d2158 to
e0909f5
Compare
|
Dispositioning the three lanes on The finding is correct: the new signals are not documented hereOpus names it most concretely (SKILL.md:121,128 still says fired lines carry "metadata That is accurate, and it is deliberate. This PR is one of three landing together against Adding even a one-line mention from here is the merge conflict the split exists to Declining GPT's prescribed fix, with the reasonGPT's remedy is "revert these signals until the consumer contract is updated". Declined: "
|
e0909f5 to
d36d56b
Compare
d36d56b to
f6cc39a
Compare
f6cc39a to
1ca65b6
Compare
|
Dispositioning Design Review and First Principles on Acted onThe index was advanced by the conductor's own nudge (Design, Watch 1, second half). The delivery counters never aged out (Design, Watch 2). Correct, and it inverted the Pin the defaults against the real constants (Design, Suggestion 1). Done, in the Stat the uid for an unreadable cwd (Design, Suggestion 2). Done. A different owner Derive Declined, with reasonsDrop the persisted Collapse Raised rather than decidedSurfacing |
a1a836f to
d63b28e
Compare
buluoray
left a comment
There was a problem hiding this comment.
Review — d63b28e
Source-verified from a fresh clone at the current head. I read fleet_probe.py in full, ran both test files, and mutation-tested the TERMINAL predicate. The only source delta since the first reviewed head (1ca65b6) is the accepted GPT fix guarding os.getuid() with hasattr(...) so a Windows AttributeError cannot escape the OSError handler and kill the whole scan — a real crash fix, and it has coverage (test_a_platform_without_getuid_still_scans).
Signal predicates and failure directions
(a) NO-PROGRESS — the i=<index> field. Predicate: _tail_entries sets last_index = produced - 1, where produced is a byte-level count of the session's OWN rows (_OWN_ROW_NEEDLES = assistant + the three tool roles), counted over the WHOLE file, never the parse window. The probe does not emit a "no-progress" tag; it emits a monotonic position and the conductor is meant to diff it across two cycles. Correctly reasoned edge cases, both source-confirmed: an inbound nudge/inject/user row does NOT advance the index (so the conductor's own nudge cannot fake progress), and the count is file-relative so it does not saturate past tail_bytes. A slow-but-working worker inside a single long build/model turn produces no new own-rows, so its index legitimately holds still — this is why the discriminator is paired with the WORKING heartbeat and the window is the conductor's judgment, not the probe's. Failure direction: an unreadable/empty transcript yields index=None → prints i=?, which cannot read as "unchanged number" — safe.
(b) FINISHED — the TERMINAL tag. Predicate: tag == "-" (unprefixed tail) AND _recorded_proto(handled, key) in TERMINAL_TAGS (a STANDDOWN/PROPOSAL the conductor already dispositioned, stored in its own proto field). Distinguished from idle-between-turns (no recorded terminal proto → falls through to age > idle_secs → IDLE), from crashed (ERR takes precedence), and — critically — from a re-seeded worker: the tag == "-" guard means a WORKING: tail is never read as finished, because WORKING is also non-firing but is live work. Failure direction: TERMINAL requires an explicit recorded terminal disposition; an unreadable worker is GONE or silent, never fabricated-finished. I mutation-tested this: inverting the predicate to not in TERMINAL_TAGS reddens exactly the 5 TERMINAL/IDLE tests and nothing else — the predicate is load-bearing and the tests are substantive.
(c) UNDELIVERED — the deliver init-timeout <a>, watchdog <b> OK-line counters. Predicate (_tail_matches): walk the window NEWEST-first; return True only if an init-timeout/watchdog pattern is reached BEFORE any protocol report. A worker that filed a report after the notice reads as delivered (returns False). So it cannot fire for a message that was delivered and reported on. Failure direction is over-admission (safe): load/mem can read healthy while this still flags a fleet that cannot deliver.
(d) UNREADABLE STATE. Missing/symlinked-out transcript → GONE (explicit, drives reclaim). Permission error on read → ([], None) → i=?, tag -, silent unless already past idle. Malformed/partial JSON → unparseable lines skipped per-row (except: continue); the index still counts via byte needles, so a torn last line does not corrupt it. It never emits FINISHED or a fake unchanged-index on an unreadable worker. Explicit-unknown / silence direction confirmed — the safe one.
(e) Names / prose / contract. The emitted names and docstring prose are accurate and internally consistent. The one gap (see non-blocking below) is that the consumer doc was not updated in the same commit.
(f) Test substance. 69 tests pass on the current head. They drive real writers (ConversationLog, _history_key_for, the token recorder) in the round-trip file so a format drift reds a test, and the agent-file signal tests assert directional behavior (index moves only on own-rows; TERMINAL vs IDLE vs WORKING; counter clears only after a report). The mutation confirmed they are not restatements.
Non-blocking
-
SKILL.md(the conductor's tag-keyed action table) was not updated in this commit, though the probe's output contract changed: there is noTERMINALaction row, no instruction to diffi=across cycles (so the headline no-progress signal has no documented consumer), and the "metadata only (key, age, tag, digest)" line is now stale (lines also carryi=).deliver/foreignOK-line fields are likewise undocumented. AGENTS.md asks for the doc update in the same commit; the advisory Design Review flagged the same thing. Recommend folding the action-table rows + probe-cycle update into this PR or a fast follow. Not blocking: the script itself is correct and safe, and doc-sync is outside the blocking set. -
Minor:
_tail_matchesclears the undelivered counter only on a protocol-prefixed report. A worker that recovers and resumes plain (non-protocol) output keeps counting as undelivered until the notice scrolls out of the 200 KB window. This is the safe (over-admission) direction and arguably by-design given the protocol requires status reports, so noting only.
Verdict: approve. Blocking: 0. Non-blocking: 2.
Ran both test files (69 passed) and mutation-tested the TERMINAL predicate (reddened exactly its 5 tests). Could not verify: bot lanes have not re-run on the current head d63b28ec (their comments cite older SHAs), and I did not run the full repo suite — only the two probe test files.
e007baa to
e55bdb6
Compare
|
Accepted. Fixed forward in The defect is real and the anchor is right
Why not the revertThe prescription is to revert configurable-root ownership "until roots come from trusted 1. Dangerous roots are refused at load time, exit 2 with a message -- the same discipline as 2. Ownership now fails toward the NON-stopping class. Every uncertain answer in One thing the mutation test showed that I would otherwise have overstatedRemoving the filesystem-root guard did NOT turn the 117 tests pass, script at 93%. Both new tests failed against the previous implementation. The advisory finding
What has changed is that the bound is now written where the next owner will hit it, as the |
e55bdb6 to
91c6183
Compare
|
All three accepted and fixed in Blocking: a symlinked root smuggled the wider scope backValidation compared the literal path only. The classifier compares Both spellings are now judged against both rules, and a root that cannot be resolved is refused I surfaced a ruling the conductor could not dismissLast round's fix made a suppressed That is worse than the bug it fixed. A signal that cannot be marked is not a signal, it is noise This is also the fourth time in this PR that a fix has left a neighbouring instance standing, The docstring had drifted from the code on three countsAll three were true and all three were mine: Worth stating plainly: this PR's whole subject is a probe that reported loudly instead of 119 tests pass, script at 93%. Both new behavioural tests failed against the previous head. |
The conductor's claim predicate was one prose line, `gh pr list --search`, and it was blind in three directions at once. Each blind spot cost a whole worker dispatch to discover the work did not exist: an item already fixed by a MERGED PR (an `--state open` query structurally cannot see one), four items that each had an OPEN PR carrying `Fixes #N` behind a single field that answered empty, and three items that declared ownership in PROSE. claim_preflight.py asks all five questions in one call and returns one verdict on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The verdict is a pure function of a checks dict, so every precedence branch is a unit test with no forge access, and an unanswerable question yields UNKNOWN, never CLAIM. Six rules earn their own mention because measurement produced each one, not reasoning: - The prose scan reads what an author SAYS, not what they QUOTE: the item specifying this script quotes the closure phrases it detects, and a raw scan returned CLOSE on live work. - The newest human comment is chosen by timestamp, never by position: the comments endpoint ignores `sort`/`direction` and answers oldest-first, so asking for `direction=desc` read the OLDEST of twelve comments on a real item. - A merged PR is coverage only if it CLAIMS to close the item. A bare mention is not closure. Measured on a real item: 7597 has TWO landed merged PRs that merely reference it, one titled "docs: investigation for #7597", and treating either as coverage would have closed an item still being fixed. - A closure request needs standing (the reporter or a repository insider), since CLOSE acts on live work. - A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any commenter can cast is a denial-of-work channel, so an unauthorized claim annotates `risk=high` and takes the live recheck. - An absent symbol vetoes only when the item's metadata corroborates bug-class: a feature request names the symbol it PROPOSES to add, so an unconditional veto parked that whole class permanently. `closedByPullRequestsReferences` is not consulted at all. It measured `[]` on two items that were closed by merged PRs, and a per-candidate forge call that cannot change the verdict is pure cost against a shared rate limit. Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true (that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035, 8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low. Refs #8029
The conductor's claim predicate was one prose line, `gh pr list --search`, and it was blind in three directions at once. Each blind spot cost a whole worker dispatch to discover the work did not exist: an item already fixed by a MERGED PR (an `--state open` query structurally cannot see one), four items that each had an OPEN PR carrying `Fixes #N` behind a single field that answered empty, and three items that declared ownership in PROSE. claim_preflight.py asks all five questions in one call and returns one verdict on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The verdict is a pure function of a checks dict, so every precedence branch is a unit test with no forge access, and an unanswerable question yields UNKNOWN, never CLAIM. Six rules earn their own mention because measurement produced each one, not reasoning: - The prose scan reads what an author SAYS, not what they QUOTE: the item specifying this script quotes the closure phrases it detects, and a raw scan returned CLOSE on live work. - The newest human comment is chosen by timestamp, never by position: the comments endpoint ignores `sort`/`direction` and answers oldest-first, so asking for `direction=desc` read the OLDEST of twelve comments on a real item. - A merged PR is coverage only if it CLAIMS to close the item. A bare mention is not closure. Measured on a real item: 7597 has TWO landed merged PRs that merely reference it, one titled "docs: investigation for #7597", and treating either as coverage would have closed an item still being fixed. - A closure request needs standing (the reporter or a repository insider), since CLOSE acts on live work. - A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any commenter can cast is a denial-of-work channel, so an unauthorized claim annotates `risk=high` and takes the live recheck. - An absent symbol vetoes only when the item's metadata corroborates bug-class: a feature request names the symbol it PROPOSES to add, so an unconditional veto parked that whole class permanently. `closedByPullRequestsReferences` is not consulted at all. It measured `[]` on two items that were closed by merged PRs, and a per-candidate forge call that cannot change the verdict is pure cost against a shared rate limit. Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true (that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035, 8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low. Refs #8029
91c6183 to
3b3d4a2
Compare
|
Rebased onto merged main ( Agrees
One real gap:
|
The conductor's claim predicate was one prose line, `gh pr list --search`, and it was blind in three directions at once. Each blind spot cost a whole worker dispatch to discover the work did not exist: an item already fixed by a MERGED PR (an `--state open` query structurally cannot see one), four items that each had an OPEN PR carrying `Fixes #N` behind a single field that answered empty, and three items that declared ownership in PROSE. claim_preflight.py asks all five questions in one call and returns one verdict on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The verdict is a pure function of a checks dict, so every precedence branch is a unit test with no forge access, and an unanswerable question yields UNKNOWN, never CLAIM. Six rules earn their own mention because measurement produced each one, not reasoning: - The prose scan reads what an author SAYS, not what they QUOTE: the item specifying this script quotes the closure phrases it detects, and a raw scan returned CLOSE on live work. - The newest human comment is chosen by timestamp, never by position: the comments endpoint ignores `sort`/`direction` and answers oldest-first, so asking for `direction=desc` read the OLDEST of twelve comments on a real item. - A merged PR is coverage only if it CLAIMS to close the item. A bare mention is not closure. Measured on a real item: 7597 has TWO landed merged PRs that merely reference it, one titled "docs: investigation for #7597", and treating either as coverage would have closed an item still being fixed. - A closure request needs standing (the reporter or a repository insider), since CLOSE acts on live work. - A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any commenter can cast is a denial-of-work channel, so an unauthorized claim annotates `risk=high` and takes the live recheck. - An open fork PR still SKIPs, but not silently. Opening a fork PR needs no permission, so rule 2 is a suppression channel anybody can use. Refusing to trust fork PRs is the wrong trade -- 192 of this repo's 301 open PRs come from forks, so that reinstates the duplicate-dispatch class this script was built from -- and the objection was never to the detection but to a response that was unconditional AND silent. So the verdict is unchanged and an unvouched fork's SKIP carries `untrusted-fork=true risk=high`. Standing is an insider association or the item's own reporter, since fixing your own bug from a fork is the ordinary case. The consumer is the conductor's review of untrusted-fork suppressions. - An absent symbol vetoes only when the item's metadata corroborates bug-class: a feature request names the symbol it PROPOSES to add, so an unconditional veto parked that whole class permanently. `closedByPullRequestsReferences` is not consulted at all. It measured `[]` on two items that were closed by merged PRs, and a per-candidate forge call that cannot change the verdict is pure cost against a shared rate limit. Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true (that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035, 8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low. The fork marker is live too: 6799 and 6509 answer SKIP untrusted-fork=true risk=high behind first-time-contributor fork PRs, while 8071 stays unmarked because its fork PR author reported the item. Refs #8029
…nished, and undelivered The probe told the conductor how loud a session was, not whether it was making progress, and three of its signals were measurably wrong. Classification (2e). A protocol word only counts in its protocol form, "<WORD>:". Matching a bare word boundary read prose as a report: over the 60 most recent transcripts on the development host, 20 of the 94 matching assistant rows were not reports, 13 of them a bare "PR #<n>". Tool rows are now dropped before classifying. Role is the transcript's own discriminator, since the presentation class is never persisted, and the error half used to read the last row of ANY role, so an error phrase quoted inside a tool title raised ERR on a healthy worker (a tool row is last on 6 of those 60 transcripts). No error signal is lost: every timeout, stall-watchdog and throttle line in that sample landed on an error/assistant/inject/user/nudge row and none on a tool row. Tail index (2a). Every fired line now carries i=<n> before d=, and the handled entry records it. An unchanged index across two probes is no progress whether or not a turn is open, which is what a self-deadlocked worker cannot fake. It counts from the start of the file rather than the start of the parse window: a window-relative count freezes once a transcript passes tail_bytes and then reads, at the sizes real worker sessions reach, as the deadlock it exists to detect. It is free, because the read already loads the whole file and keeps only the window. TERMINAL (2b). A worker that filed STANDDOWN or PROPOSAL and then wrote one unprefixed line is finished, not wedged. The handled set keeps one entry per key, so that terminal disposition was overwritten by the next tag and the finished worker aged into IDLE, which calls for the opposite action. The last dispositioned protocol tag now survives in its own field. Delivery counters (2c). "deliver init-timeout <a>, watchdog <b>" on the OK line, counted for every watched session. Load and memory can both read healthy while the fleet cannot deliver. Both pattern sets are config (init_timeout_res, watchdog_res) defaulted from the emitters, and validated like the existing ones. Banned scan (2d). A banned command shape is only a banned operation when the fleet owns it. /proc/<pid>/cwd is compared against a new fleet_worktrees config key; only fleet-owned or unreadable matches are printed, with the class on the line, and unrelated matches are summarised as "foreign <n>". An undeclared scope reports everything, because scoping against an empty set would mute the signal invisibly. The argv still never appears. Leading decoration is normalised away (2e, extended). The tag match is anchored at position zero, so "**BLOCKED:**" puts an asterisk where the tag has to be: the match fails, the line reads as no-prefix, and on a fresh transcript IDLE does not fire either, so the report is silent rather than delayed. Emphasis, strikethrough, code ticks, heading hashes, blockquote arrows and list markers are stripped from the front before matching. Two boundaries are explicit because both are observable: the normalisation applies to the MATCHED text only, so the digest still covers what the worker wrote, and the anchor survives, so a bolded protocol word mid-sentence is not a report. BLOCKED is sticky (2b, extended). The tag is now the newest REPORT, not the newest message. A probe samples rather than subscribes, and the protocol requires a blocked worker to keep reporting status, so the worker's own next message displaced the one place the probe looks. Nothing was marked and nothing was suppressed: the BLOCKED was never observed, so the obligation existed on both sides and was visible to neither. A heartbeat and unprefixed text now leave a sticky report standing; any other report supersedes it. The digest is keyed on the sticky report's own text, so it fires once, the ruling quiets it, and only a new blocker re-fires. TERMINAL and sticky BLOCKED stay separate tags: one says close me, the other says a ruling is owed. Banned-ops premise (2d). The pytest rule's sense is unchanged, but its comment claimed the repo's "-n auto" addopts forks one worker per core. setup.cfg documents the opposite: the rootdir conftest's pytest_xdist_auto_num_workers hook (conftest.py:856) sizes the pool by available memory and by what concurrent runs already hold, and "an explicit -n <N> bypasses the budget". So on this repo the explicit spelling is the one that can outgrow the host. The comment now states what the rule actually catches, a run whose worker count nobody chose, and says plainly what that costs. -n0 is asserted bounded rather than assumed, alongside -n 0 and --numprocesses=0, because -n0 is the form the fleet is required to use and a rule that flagged it would stop every worker obeying it. Coverage (2f) is DEFERRED, not delivered. The tests here take the script to 95% when coverage is sourced at the tree the tests run against, but CI records 47/331 = 14.2%, the import-only footprint, so the covering tests are not attributed there and no test can move the gated number. The existing baseline exemption for this file is therefore left exactly as it stands on the default branch. Refs #7597, where the sharper diagnosis belongs: the sibling script in the same directory, loaded through the same helper, is ABSENT from the coverage report entirely (0 occurrences among 1,233 measured classes), so it is never judged rather than judged and passing, and any analysis resting on it as a control case is reasoning from a case it never measured. --config and --mark-handled KEY TAG DIGEST keep their exact signatures, and a state file written before the new fields still suppresses: index and proto are additive metadata outside the digest. Refs #8029 Refs #7597
3b3d4a2 to
256b20a
Compare
|
Accepted, correct as traced, and fixed by the prescribed remedy in The defectA worker files That is the harm Why my tests missed it, which is the part worth keeping
I have hit a version of this before in this PR: the missing The fixThe conversion is now refused when the recorded disposition is terminal, on both paths rather |
|
Both watch items are already-recorded material and neither asks for a change here. One factual The The governance ask is already in place, and it is the right one. "Hold the author to landing The rest stands as disclosed. The one-entry-per-key residual and the On the "why did the conductor miss my BLOCKED" note -- that is exactly the right thing to |
What is the problem?
The pipeline conductor's
fleet_probe.pyreports how loud a worker session is, notwhether it is making progress, and three of its signals were measurably wrong when
checked against real transcripts.
matched a bare word boundary,
^<WORD>\b. Measured over the 60 most recent sessiontranscripts on the development host, 20 of the 94 matching assistant rows were
not reports at all, 13 of them opening with a bare
PR #<n>.row on 6 of those 60 transcripts, so an error phrase quoted inside a tool card's
title raised
ERRon a healthy worker.per key, so a
STANDDOWN/PROPOSALdisposition was overwritten by the next tag, andthe session then aged into
IDLE.MESSAGE, and the protocol requires a blocked worker to keep reporting status, so the
worker's own next
WORKING:displaced theBLOCKEDbefore the next sample. Unlike thesuppression case, nothing was marked and nothing was suppressed: the signal was never
observed, so the obligation existed on both sides and was visible to neither.
zero, so
**BLOCKED:**puts an asterisk where the tag has to be: the match fails, theline reads as no-prefix, and on a fresh transcript
IDLEdoes not fire either. Thereport is not delayed, it is silent -- and a worker writing emphasis is following
ordinary formatting habit, not breaking protocol.
touched", which anything touching the file resets. Nothing answered "has this session
actually said anything new".
printed, "unchanged across two probes" is a comparison delegated to whoever reads the
output, documented nowhere and enforced by nothing, so it may simply never happen.
bannedcounted the whole host. Any process matching a banned command shape wasreported, whatever directory it ran in.
sessions are dying on an initialize timeout or having turns ended by the stall
watchdog.
fleet_probe.pyis recorded in.github/coverage-baselines/backend.txtat 14.8%(34/229) after landing below the per-file floor with no tests (main is red: fleet_probe.py at 14.8% fails the per-file coverage floor on every rebased PR #7597).
Why this issue matters to the user
The conductor decides, per cycle, whether to nudge a worker, reclaim its item, or close
it out. Every defect above pushes it toward the wrong one of those:
PRtag on a prose line spends a round reading a PR that does not exist.ERRon a healthy worker interrupts a turn that was fine.IDLEon a worker that already stood down asks it to keep going, or reclaims andre-dispatches an item that is already settled. That is the duplicate-dispatch failure
the probe exists to prevent.
one, because both hold an open turn and both keep their transcript warm.
that was not the offender.
How our fix solves it
Each change starts from the measurement, so the discriminator is the one the transcript
actually carries rather than a guess.
Classification reads what the session said (2e).
roleis the transcript's owndiscriminator: the writers tag a tool card with its role and the presentation class is
never persisted, so nothing else separates the two.
tool/tool_call/tool_resultrows are dropped before either half classifies, and a protocol word now only counts in
its protocol form,
<WORD>:. Excluding tool rows costs no error signal, and the samesample proves it: every
initialize timed out, stall-watchdog and throttle line landedon an
error,assistant,inject,userornudgerow, and not one on a tool row.A monotonic tail index (2a). Every fired line carries
i=<n>befored=, and thehandled entry records it. An unchanged index across two probes is no progress, whether
or not a turn is open, which is the one thing a self-deadlocked worker cannot fake. It
counts lines from the START of the file rather than from the start of the parse window,
because a window-relative count saturates the moment a transcript passes
tail_bytes(200 KB) and then holds still while the session talks, reading as exactly the deadlock
it exists to detect. It costs nothing: the read already loads the whole file and keeps
only the window, so the prefix is in hand. It is a line position, so it carries no
transcript content.
TERMINAL(2b). The last dispositioned protocol tag is stored in its own field, soa later
IDLEorGONEdisposition cannot erase it. When that tag isSTANDDOWNorPROPOSALand the current tail has no protocol prefix, the probe firesTERMINALinstead of ageing into
IDLE, then suppresses by digest like any other tag. Thetag == "-"guard is load-bearing: the non-firing set holds both-andWORKING, sowithout it a worker that stood down, was re-seeded, and is now reporting
WORKING:wouldread as finished and have its live work closed.
Leading decoration is normalised away (2e, extended). Emphasis and strikethrough
markers, code ticks, heading hashes, blockquote arrows and list markers are stripped from
the front of a candidate before the tag is matched, in any combination, so
**BLOCKED:**,> BLOCKED:,- **GREEN:**,### PROPOSAL:and1. STANDDOWN:all classify as theirtag. Two boundaries are explicit because both are observable. The normalisation applies to
the MATCHED text only -- the digest is still computed over what the worker wrote, so a
decorated report keeps a stable, distinct identity and
--mark-handledround-trips on it.And the anchor survives: decoration comes off the FRONT, so a bolded protocol word
mid-sentence is a worker talking about a report rather than filing one. Making this a
substring search would tag every message that mentioned a tag.
NOPROGRESSanswers the question instead of posing it (2a, extended). The probe nowcompares the current tail index against the one recorded at the last
--mark-handledandfires
NOPROGRESSitself, rather than printing two numbers and hoping someone diffs them.This needs no new write, so the one-writer rule holds: the only write is still the mark.
The claim is correspondingly precise -- not "quiet for a while" but "has emitted no message
and run no tool since you last acted on this session".
Three properties are deliberate and each was found by testing rather than reasoning. The
disposition must be at least one idle budget old, or the tag fires on the cycle right after
every mark, since a session that just filed a report has trivially produced nothing in the
seconds since.
IDLEoutranks it, because a cold transcript is already fully described byIDLEand its nudge ladder is the right action -- whatIDLEcannot see is the session heldWARM by traffic it never answers, which is the case this tag exists for. And it reaches
sessions whose named tag is SUPPRESSED, which is its most useful instance: the ruling was
delivered, the tag went quiet, and nothing has come out since. It expires like
IDLE,because a stall is a continuing condition rather than a payload filed once.
BLOCKEDis sticky (2b, extended). The tag is now the newest REPORT rather than thenewest message. A probe samples; it does not subscribe, so it can only see a session's
latest message -- and the protocol requires a blocked worker to keep reporting status,
which means the worker's own next message displaces the one place the probe looks. The
result is not a suppressed signal or a deferred one: the
BLOCKEDis never observed atall, and the debt is then invisible from both ends, with the worker holding position for a
ruling and the conductor never learning it owes one. A heartbeat (
WORKING) and unprefixedtext therefore leave a sticky report standing, and any other report supersedes it, because
a worker that has since filed
PR,GREEN,STANDDOWNorPROPOSALhas moved on. Thedigest is keyed on the sticky report's own text, so sticky does not mean noisy: it fires
once, the ruling quiets it, and only a genuinely new blocker re-fires.
TERMINALandsticky
BLOCKEDare deliberately separate tags, since one says close me and the other saysa ruling is owed.
Delivery counters (2c).
deliver init-timeout <a>, watchdog <b>on theOKline,counted for every watched session whether or not it fires, because an undelivered
session is a fleet fact rather than a per-tag one. Both pattern sets are config keys,
init_timeout_resandwatchdog_res, defaulted from the emitters themselves(
dashboard.state.TOOL_STALL_RECOVERY_PREFIX/STALE_RECOVERY_PREFIX,acp.types.STOP_REASON_TOOL_STALL,mcp_gateway.backend's initialize timeout) andvalidated exactly like
err_res: a bad regex is malformed config, exit 2, never a crashmid-cycle.
Ownership-scoped banned scan (2d). A banned command SHAPE is only a banned OPERATION
when the fleet owns it, so the process is attributed against a new
fleet_worktreesconfigkey before the line is printed. Only fleet-owned or unattributable matches print, with the
class on the line (
cwd=fleet|unknown); matches provably belonging to somebody else aresummarised as
foreign <n>.The reason this resolves the class DURING the scan rather than emitting a bare pid is
measured, not argued from principle. Over real conductor patrols, five
BANNEDlines firedand attribution was attempted within seconds of each probe returning. Three processes
were already gone --
/proc/<pid>/cwdunreadable,/proc/<pid>/cmdlineabsent,psempty. These are short-lived targeted runs and the probe-to-action gap is reliably longer
than the process lives, so a bare
BANNED pid=N rule=Xcannot tell a fleet workerviolating the directive from unrelated activity on the same host. The only safe response to
a line like that is to ignore it, which teaches the operator to ignore the whole class.
Resolving the class while the evidence still exists is what makes the line actionable.
The other two were attributable, and they changed the design. In BOTH,
/proc/<pid>/cwdwas unreadable while
/proc/<pid>/cmdlineread fine -- the cwd andexesymlinks need theaccess a debugger would have, and the cmdline does not, so it is the only one of the three
that survives another user's process. A cwd-only classifier would have returned
unknownfor two processes that could be PROVEN not to be the fleet's, and an unknown match makes
the conductor act. So the program path is consulted when the cwd cannot be read.
The two signals do NOT carry the same authority, and that asymmetry is the load-bearing
part:
fleetforeignunknownTreating any non-match as
foreignlooks symmetric and would mute the signal this scanexists to produce: measured on the host this runs on, a fleet worktree has no
.venvand its workers invoke a global
python3shim, so a real banned run INSIDE the fleet has aprogram path outside every fleet worktree. Calling that
foreigndrops it silently. Thesymmetric form was implemented and run against the suite rather than reasoned about: it
broke
test_an_unreadable_cwd_is_unknown_and_still_reported, which shipped with theoriginal 2d, so the property was already pinned before this refinement touched it. "Is it a
venv" is decided by the
pyvenv.cfgmarker beside the interpreter, which sits in the sameplace for POSIX
bin/pythonand WindowsScripts/python.exe, rather than by a pathheuristic.
Reading argv is not the same as printing it. The command line is still never emitted,
because a secret can ride in an argument -- but the program PATH is structural, so it can be
compared for a decision and dropped. That distinction is stated at the call site, since the
next reader would otherwise conclude argv was excluded from being READ.
An unattributable match is still
unknown, still PRINTED and still counted as banned. Anunknown is not a foreign. Dropping it would be the one outcome worse than a noisy line -- a
real banned run inside the fleet, silently unseen.
A relative
fleet_worktreesentry can never match an absolute cwd, so it is rejected asmalformed config rather than silently muting the scan, and an undeclared or empty
fleet_worktreesreports everything asunknown, because scoping against an empty setwould classify every match as foreign and mute the signal invisibly. Path comparison is
normalised in one place: the Windows lane caught a literal string compare misfiling every
match, because
os.readlinkthere can answer an extended-length\\?\D:\...path that noconfigured root will ever spell, and case and separators do not compare byte-wise. The
classifier also gets a second chance through
realpath, so a symlinked worktree root or ashort (8.3) name still matches. The argv is still never echoed.
Coverage (2f) is deferred, not delivered. The tests here take the script to 95% when
coverage is sourced at the tree the tests run against, but that is not the number the gate
consumes: CI records
47/331 = 14.2%, the import-only footprint, so the covering testsare not attributed there and no test can move the gated figure. The file's existing
baseline exemption is therefore left exactly as it stands on the default branch, and this
PR makes no claim to have lifted it above the floor. The cause and a sharper diagnosis are
recorded on #7597.
Banned-ops premise (2d). The pytest rule's sense is unchanged, but its comment
claimed the repo's
-n autoaddopts forks one worker per core.setup.cfgdocuments theopposite, and the comment now states what the rule actually catches: a run whose worker
count nobody CHOSE.
-n0is asserted bounded rather than assumed.Backward compatibility is explicit and tested:
--configand--mark-handled KEY TAG DIGESTkeep their exact signatures, and a state file writtenbefore this change still suppresses, because
indexandprotoare additive metadataoutside the digest.
What tests we did
All in
test/test_pipeline_conductor_agent.py, single-process (-n0), plus theexisting round-trip contracts in
test_pipeline_conductor_probe_roundtrip.py:74 passed.
Every new behaviour has a test that fails against the previous implementation. Stashing
only the script and re-running the class reds 17 of them, and the ones that stay
green are the deliberate no-regression guards (a spoken report still fires with a tool
row after it; a non-terminal report still ages into
IDLE).Named coverage of the contract's required cases:
row;
i=precedesd=on the fired line;TERMINAL, notIDLE, and a laternon-protocol disposition does not erase it; a re-seeded worker reporting
WORKING:isNOT terminal, while a
WORKINGtail that goes silent still ages intoIDLE;BLOCKED:followed by twoWORKING:messages still classifies asBLOCKED, andstill does on a second probe that did not mark it handled; unprefixed text does not
clear it and does not let it age into
IDLE; its digest is unchanged by new heartbeatsso it fires once; the ruling quiets it and a genuinely new blocker re-fires;
PR,GREENandSTANDDOWNeach clear it; and it outranks a recorded terminal disposition;**BLOCKED:**,> BLOCKED:,- **GREEN:**,### PROPOSAL:,1. STANDDOWN:,__PR:__and a code-ticked prefix all classify as their tag; a decoratedBLOCKEDsurvives two following heartbeats (the two rules composing); a bolded protocol word
mid-sentence still does NOT fire; a tool row carrying
**PR:**still does not classify;and the digest still differs between a decorated and an undecorated report, with
--mark-handledround-tripping on the decorated one;STANDDOWN,PR:and an error phrase classifies as no-prefix,and does not feed the delivery counters either;
PR #6580does not fire;fleet_worktreesisforeign, notbanned; a subdirectory ofa worktree is fleet-owned; a sibling named
wt-a-oldis not swallowed bywt-a; aworktree reached through a symlink still matches; an extended-length
\\?\pathnormalises to the root it names; an unreadable cwd is
unknownand still reported; theargv still never appears;
even when the cwd is unreadable, leaves a system/shim interpreter
unknownand stillreports it, and decides on the path while a secret in an ARGUMENT never reaches the
output;
-n0,-n 0and--numprocesses=0are each present as their own fixture in thebounded-spellings test (verified by name, not inferred from the
\dbranch), alongsidethe existing
-n 4/-n=4/-n4/--numprocesses=4cases, while-n autoand abare pytest still fire;
NOPROGRESSfires on a session held warm by nudges it never answers, and NOT on thecycle right after a disposition; one produced row (a message or a single tool call)
clears it; a live
BLOCKEDoutranks it; it is suppressible and expires likeIDLE; anda session the conductor has never acted on cannot be stalled yet;
nudge,injectoruserrow does not advance the index while a tool rowdoes, and a row QUOTING transcript JSON does not advance it either -- so the
line-anchored needle is pinned independently of
json.dumpsescaping;separate assertions pin that it does not: a condition mark preserves an
answered payload, so does any other later mark, the legacy state shape carries
both halves of it across an upgrade, and an answered
ERRdoes not bury aruling nobody has answered;
OKline, are configurable, and a badinit_timeout_resorwatchdog_resregex is malformed config (exit 2), not a crash;fleet_worktreesentry is malformed config;config with none of the new keys still produces a full
OKline.Repo gates run on the two changed files:
black,isort,flake8,mypyclean, pluscheck_brand_name,check_builtin_skill_scope,check_loop_bound_locks,check_black_formatting,check_testpaths_coverageandcheck_harness_parity.Any other suggestions on the work
The coverage half of the original scope is deferred, and the baseline is untouched.
.github/coverage-baselines/backend.txtrecordsfleet_probe.py 14.8 # 34/229, and itis left exactly as the default branch has it. The tests in this PR do cover the script
(95% when coverage is sourced at the tree the tests run against), but that measurement is
not what the gate reads: CI's own artifact records
47/331 = 14.2%, which is theimport-only footprint. Removing the exemption on the strength of a local number makes the
file a
new_offenderat 14.2% and reds a required lane for every open PR, so the exemptionstays and 2f is honestly unfinished rather than quietly claimed.
A sharper diagnosis for #7597 than the one currently recorded there. The comment above
that baseline entry, and the repo's investigation of the issue, both rest on a control case:
that a sibling script loaded through the same helper is unbaselined and passes the gate,
which would mean the loader attributes coverage correctly and the problem is specific to
this file. CI's coverage artifact does not support it.
credit_spend.pyis absent fromthe report entirely -- 0 occurrences among 1,233 measured classes -- so it is never judged
rather than judged and passing.
fleet_probe.pyis the only file from that skill directorypresent at all, and it enters import-only. Whatever the root cause is, it is not the loader,
and an analysis resting on that control case is reasoning from a case it never measured.
An explicit
-n <N>is the LESS safe pytest spelling on this repo, which is why thebanned rule's old comment was misleading.
setup.cfgdocuments that-n autodoes notmean one worker per core here: the rootdir conftest's
pytest_xdist_auto_num_workershook(
conftest.py:856) sizes the pool by available memory and by what concurrent runs on thehost already hold, and "an explicit
-n <N>bypasses the budget". So guidance of the form"use a bounded
-n 2" hands a fleet the one spelling that can outgrow the host, while-n0-- the repo's own documented override -- is genuinely in-process and costs nothing.The rule's SENSE is deliberately unchanged, because which shapes it flags decides what the
conductor stops mid-turn across a whole fleet; only its stated premise is corrected, and
-n0is now asserted bounded rather than assumed.Two observations for whoever owns the probe next:
tail_bytescaps how much of a transcript is PARSED, not how much is READ. Theimplementation is
path.read_bytes()[-max_bytes:], so the whole file is loadedeither way. That is what makes the monotonic index free, and it is worth knowing
before anyone reads the setting as an I/O bound on a large session.
suppressed signals whose classification moved will re-fire a single time on the first
cycle after deployment. That is the documented degradation (a handled signal re-fires
once, never a crashed patrol), not a new failure mode.
Accepting a finding is not the same as agreeing with its reasoning. The line-anchored
index needle was taken from a review suggestion whose stated case cannot actually arise:
json.dumpsescapes inner quotes, so a row quoting{"role": "assistant"is stored as{\"role\": ...and never matched the needle. The fix stands on its own -- anchoring iscorrect without depending on that argument being sound, and it also covers a torn line at
the window edge -- so it was accepted for better reasons rather than rebutted.
Residual: one entry per key. The handled set stores one record per session, and that
single premise carries three jobs -- suppression, terminality, and progress. Terminality and
stickiness are therefore SPECIAL CASES layered on top of the record rather than properties of
the data model: a terminal disposition survives a later mark only because the payload tag and
digest are carried forward explicitly, and a sticky
BLOCKEDreaches past an answeredERRonly because the firing path looks for it. Both work and both are asserted, but neither falls
out of the shape.
Stickiness also has a bound the state file cannot close. It reaches only as far back as the
parse window, so a blocked worker that heartbeats long enough pushes its own report past
tail_bytesand the debt goes invisible. Sourcing it from the handled set looks like the fixand is not: the recorded tag exists exactly when the ruling was already delivered and the
signal is correctly suppressed anyway, and it is absent in the case that actually bites -- an
undispositioned report ageing out. Reading it would be a no-op or a machine for re-firing
answered rulings, so the transcript stays the source of truth.
A per-tag handled map makes both properties structural instead of special-cased, and it changes
the persisted state schema, so it wants its own change with its own migration test. A follow-up
carries it.
An advisory finding needs the same verification as a blocking one. One suggestion here
-- resolve an unreadable
/proc/<pid>/cwdby owner, so another user's process reads asforeignrather thanunknown-- was cosmetic, and taking it introduced a real portabilitydefect:
os.getuidis POSIX-only, its absence raisesAttributeErrorrather thanOSError, and the Windows shards run this scan against a fake/proc. The next lane caughtit in one push; the test written to cover it then depended on the same primitive it was
testing the absence of. It is withdrawn, with the reasoning left at the call site: this file
imports nothing from the package, so it cannot route through
platform_compat, and anunreadable cwd already reports as
unknown, which is the fail-open reading that matters.The two findings treated as defects rather than suggestions -- the index counting inbound
rows, and the delivery counters never ageing out -- were both real and both worth it.
Refs #8029
Refs #7597