Skip to content

feat(pipeline-conductor): probe signals for no-progress, finished, and undelivered workers - #8035

Merged
buluoray merged 1 commit into
mainfrom
feat/conductor-probe-signals-8029
Sep 3, 2026
Merged

buluoray merged 1 commit into
mainfrom
feat/conductor-probe-signals-8029

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What is the problem?

The pipeline conductor's fleet_probe.py reports how loud a worker session is, not
whether it is making progress, and three of its signals were measurably wrong when
checked against real transcripts.

  • A protocol word anywhere at the start of a line counted as a report. The regex
    matched a bare word boundary, ^<WORD>\b. Measured over the 60 most recent session
    transcripts on the development host, 20 of the 94 matching assistant rows were
    not reports at all, 13 of them opening with a bare PR #<n>.
  • The error half read the last transcript row of any role. A tool row is the last
    row on 6 of those 60 transcripts, so an error phrase quoted inside a tool card's
    title raised ERR on a healthy worker.
  • A finished worker read exactly like a wedged one. The handled set keeps one entry
    per key, so a STANDDOWN/PROPOSAL disposition was overwritten by the next tag, and
    the session then aged into IDLE.
  • A ruling owed could be lost to sampling entirely. The probe classified the newest
    MESSAGE, and the protocol requires a blocked worker to keep reporting status, so the
    worker's own next WORKING: displaced the BLOCKED before the next sample. Unlike the
    suppression case, nothing was marked and nothing was suppressed: the signal was never
    observed, so the obligation existed on both sides and was visible to neither.
  • Any leading decoration defeated the tag entirely. The match is anchored at position
    zero, so **BLOCKED:** puts an asterisk where the tag has to be: the match fails, the
    line reads as no-prefix, and on a fresh transcript IDLE does not fire either. The
    report is not delayed, it is silent -- and a worker writing emphasis is following
    ordinary formatting habit, not breaking protocol.
  • There was no progress signal at all. Age answers "how long since this file was
    touched", which anything touching the file resets. Nothing answered "has this session
    actually said anything new".
  • And the no-progress question was posed rather than answered. Even with an index
    printed, "unchanged across two probes" is a comparison delegated to whoever reads the
    output, documented nowhere and enforced by nothing, so it may simply never happen.
  • banned counted the whole host. Any process matching a banned command shape was
    reported, whatever directory it ran in.
  • Nothing reported undelivered work. Load and memory can both read healthy while
    sessions are dying on an initialize timeout or having turns ended by the stall
    watchdog.
  • fleet_probe.py is recorded in .github/coverage-baselines/backend.txt at 14.8%
    (34/229) after landing below the per-file floor with no tests (main is red: fleet_probe.py at 14.8% fails the per-file coverage floor on every rebased PR #7597).

Why this issue matters to the user

The conductor decides, per cycle, whether to nudge a worker, reclaim its item, or close
it out. Every defect above pushes it toward the wrong one of those:

  • A fabricated PR tag on a prose line spends a round reading a PR that does not exist.
  • ERR on a healthy worker interrupts a turn that was fine.
  • IDLE on a worker that already stood down asks it to keep going, or reclaims and
    re-dispatches an item that is already settled. That is the duplicate-dispatch failure
    the probe exists to prevent.
  • With no progress signal, a self-deadlocked worker is indistinguishable from a busy
    one, because both hold an open turn and both keep their transcript warm.
  • A banned-shape process in an unrelated checkout on the same host got a worker stopped
    that was not the offender.
  • A fleet that cannot deliver reads as a healthy fleet doing nothing.

How our fix solves it

Each change starts from the measurement, so the discriminator is the one the transcript
actually carries rather than a guess.

Classification reads what the session said (2e). role is the transcript's own
discriminator: the writers tag a tool card with its role and the presentation class is
never persisted, so nothing else separates the two. tool/tool_call/tool_result
rows are dropped before either half classifies, and a protocol word now only counts in
its protocol form, <WORD>:. Excluding tool rows costs no error signal, and the same
sample proves it: every initialize timed out, stall-watchdog and throttle line landed
on an error, assistant, inject, user or nudge row, and not one on a tool row.

A monotonic tail index (2a). Every fired line carries i=<n> before d=, and the
handled entry records it. An unchanged index across two probes is no progress, whether
or not a turn is open, which is the one thing a self-deadlocked worker cannot fake. It
counts lines from the START of the file rather than from the start of the parse window,
because a window-relative count saturates the moment a transcript passes tail_bytes
(200 KB) and then holds still while the session talks, reading as exactly the deadlock
it exists to detect. It costs nothing: the read already loads the whole file and keeps
only the window, so the prefix is in hand. It is a line position, so it carries no
transcript content.

TERMINAL (2b). The last dispositioned protocol tag is stored in its own field, so
a later IDLE or GONE disposition cannot erase it. When that tag is STANDDOWN or
PROPOSAL and the current tail has no protocol prefix, the probe fires TERMINAL
instead of ageing into IDLE, then suppresses by digest like any other tag. The
tag == "-" guard is load-bearing: the non-firing set holds both - and WORKING, so
without it a worker that stood down, was re-seeded, and is now reporting WORKING: would
read as finished and have its live work closed.

Leading decoration is normalised away (2e, extended). Emphasis and strikethrough
markers, code ticks, heading hashes, blockquote arrows and list markers are stripped from
the front of a candidate before the tag is matched, in any combination, so **BLOCKED:**,
> BLOCKED:, - **GREEN:**, ### PROPOSAL: and 1. STANDDOWN: all classify as their
tag. Two boundaries are explicit because both are observable. The normalisation applies to
the MATCHED text only -- the digest is still computed over what the worker wrote, so a
decorated report keeps a stable, distinct identity and --mark-handled round-trips on it.
And the anchor survives: decoration comes off the FRONT, so a bolded protocol word
mid-sentence is a worker talking about a report rather than filing one. Making this a
substring search would tag every message that mentioned a tag.

NOPROGRESS answers the question instead of posing it (2a, extended). The probe now
compares the current tail index against the one recorded at the last --mark-handled and
fires NOPROGRESS itself, rather than printing two numbers and hoping someone diffs them.
This needs no new write, so the one-writer rule holds: the only write is still the mark.
The claim is correspondingly precise -- not "quiet for a while" but "has emitted no message
and run no tool since you last acted on this session".

Three properties are deliberate and each was found by testing rather than reasoning. The
disposition must be at least one idle budget old, or the tag fires on the cycle right after
every mark, since a session that just filed a report has trivially produced nothing in the
seconds since. IDLE outranks it, because a cold transcript is already fully described by
IDLE and its nudge ladder is the right action -- what IDLE cannot see is the session held
WARM by traffic it never answers, which is the case this tag exists for. And it reaches
sessions whose named tag is SUPPRESSED, which is its most useful instance: the ruling was
delivered, the tag went quiet, and nothing has come out since. It expires like IDLE,
because a stall is a continuing condition rather than a payload filed once.

BLOCKED is sticky (2b, extended). The tag is now the newest REPORT rather than the
newest message. A probe samples; it does not subscribe, so it can only see a session's
latest message -- and the protocol requires a blocked worker to keep reporting status,
which means the worker's own next message displaces the one place the probe looks. The
result is not a suppressed signal or a deferred one: the BLOCKED is never observed at
all, and the debt is then invisible from both ends, with the worker holding position for a
ruling and the conductor never learning it owes one. A heartbeat (WORKING) and unprefixed
text therefore leave a sticky report standing, and any other report supersedes it, because
a worker that has since filed PR, GREEN, STANDDOWN or PROPOSAL has moved on. The
digest is keyed on the sticky report's own text, so sticky does not mean noisy: it fires
once, the ruling quiets it, and only a genuinely new blocker re-fires. TERMINAL and
sticky BLOCKED are deliberately separate tags, since one says close me and the other says
a ruling is owed.

Delivery counters (2c). deliver init-timeout <a>, watchdog <b> on the OK line,
counted for every watched session whether or not it fires, because an undelivered
session is a fleet fact rather than a per-tag one. Both pattern sets are config keys,
init_timeout_res and watchdog_res, defaulted from the emitters themselves
(dashboard.state.TOOL_STALL_RECOVERY_PREFIX / STALE_RECOVERY_PREFIX,
acp.types.STOP_REASON_TOOL_STALL, mcp_gateway.backend's initialize timeout) and
validated exactly like err_res: a bad regex is malformed config, exit 2, never a crash
mid-cycle.

Ownership-scoped banned scan (2d). A banned command SHAPE is only a banned OPERATION
when the fleet owns it, so the process is attributed against a new fleet_worktrees config
key before the line is printed. Only fleet-owned or unattributable matches print, with the
class on the line (cwd=fleet|unknown); matches provably belonging to somebody else are
summarised as foreign <n>.

The reason this resolves the class DURING the scan rather than emitting a bare pid is
measured, not argued from principle. Over real conductor patrols, five BANNED lines fired
and attribution was attempted within seconds of each probe returning. Three processes
were already gone
-- /proc/<pid>/cwd unreadable, /proc/<pid>/cmdline absent, ps
empty. These are short-lived targeted runs and the probe-to-action gap is reliably longer
than the process lives, so a bare BANNED pid=N rule=X cannot tell a fleet worker
violating the directive from unrelated activity on the same host. The only safe response to
a line like that is to ignore it, which teaches the operator to ignore the whole class.
Resolving the class while the evidence still exists is what makes the line actionable.

The other two were attributable, and they changed the design. In BOTH, /proc/<pid>/cwd
was unreadable while /proc/<pid>/cmdline read fine -- the cwd and exe symlinks need the
access a debugger would have, and the cmdline does not, so it is the only one of the three
that survives another user's process. A cwd-only classifier would have returned unknown
for two processes that could be PROVEN not to be the fleet's, and an unknown match makes
the conductor act. So the program path is consulted when the cwd cannot be read.

The two signals do NOT carry the same authority, and that asymmetry is the load-bearing
part:

program path class why
under a fleet worktree fleet conclusive; nothing outside that checkout runs its interpreter
a venv interpreter elsewhere foreign a venv belongs to the checkout that created it
a system or shim interpreter unknown every checkout shares it, so it attributes nothing

Treating any non-match as foreign looks symmetric and would mute the signal this scan
exists to produce: measured on the host this runs on, a fleet worktree has no .venv
and its workers invoke a global python3 shim, so a real banned run INSIDE the fleet has a
program path outside every fleet worktree. Calling that foreign drops it silently. The
symmetric form was implemented and run against the suite rather than reasoned about: it
broke test_an_unreadable_cwd_is_unknown_and_still_reported, which shipped with the
original 2d, so the property was already pinned before this refinement touched it. "Is it a
venv" is decided by the pyvenv.cfg marker beside the interpreter, which sits in the same
place for POSIX bin/python and Windows Scripts/python.exe, rather than by a path
heuristic.

Reading argv is not the same as printing it. The command line is still never emitted,
because a secret can ride in an argument -- but the program PATH is structural, so it can be
compared for a decision and dropped. That distinction is stated at the call site, since the
next reader would otherwise conclude argv was excluded from being READ.

An unattributable match is still unknown, still PRINTED and still counted as banned. An
unknown is not a foreign. Dropping it would be the one outcome worse than a noisy line -- a
real banned run inside the fleet, silently unseen.

A relative fleet_worktrees entry can never match an absolute cwd, so it is rejected as
malformed config rather than silently muting the scan, and an undeclared or empty
fleet_worktrees reports everything as unknown, because scoping against an empty set
would classify every match as foreign and mute the signal invisibly. Path comparison is
normalised in one place: the Windows lane caught a literal string compare misfiling every
match, because os.readlink there can answer an extended-length \\?\D:\... path that no
configured root will ever spell, and case and separators do not compare byte-wise. The
classifier also gets a second chance through realpath, so a symlinked worktree root or a
short (8.3) name still matches. The argv is still never echoed.

Coverage (2f) is deferred, not delivered. The tests here take the script to 95% when
coverage is sourced at the tree the tests run against, but that is not the number the gate
consumes: CI records 47/331 = 14.2%, the import-only footprint, so the covering tests
are not attributed there and no test can move the gated figure. The file's existing
baseline exemption is therefore left exactly as it stands on the default branch, and this
PR makes no claim to have lifted it above the floor. The cause and a sharper diagnosis are
recorded on #7597.

Banned-ops premise (2d). The pytest rule's sense is unchanged, but its comment
claimed the repo's -n auto addopts forks one worker per core. setup.cfg documents the
opposite, and the comment now states what the rule actually catches: a run whose worker
count nobody CHOSE. -n0 is asserted bounded rather than assumed.

Backward compatibility is explicit and tested: --config and
--mark-handled KEY TAG DIGEST keep their exact signatures, and a state file written
before this change still suppresses, because index and proto are additive metadata
outside the digest.

What tests we did

All in test/test_pipeline_conductor_agent.py, single-process (-n0), plus the
existing round-trip contracts in test_pipeline_conductor_probe_roundtrip.py:
74 passed.

Every new behaviour has a test that fails against the previous implementation. Stashing
only the script and re-running the class reds 17 of them, and the ones that stay
green are the deliberate no-regression guards (a spoken report still fires with a tool
row after it; a non-terminal report still ages into IDLE).

Named coverage of the contract's required cases:

  • unchanged tail index across two probes, and it moves by exactly one on one appended
    row;
  • the index stays monotonic on a transcript far larger than the parse cap;
  • i= precedes d= on the fired line;
  • a terminal report followed by unprefixed text is TERMINAL, not IDLE, and a later
    non-protocol disposition does not erase it; a re-seeded worker reporting WORKING: is
    NOT terminal, while a WORKING tail that goes silent still ages into IDLE;
  • a BLOCKED: followed by two WORKING: messages still classifies as BLOCKED, and
    still does on a second probe that did not mark it handled; unprefixed text does not
    clear it and does not let it age into IDLE; its digest is unchanged by new heartbeats
    so it fires once; the ruling quiets it and a genuinely new blocker re-fires; PR,
    GREEN and STANDDOWN each clear it; and it outranks a recorded terminal disposition;
  • **BLOCKED:**, > BLOCKED:, - **GREEN:**, ### PROPOSAL:, 1. STANDDOWN:,
    __PR:__ and a code-ticked prefix all classify as their tag; a decorated BLOCKED
    survives two following heartbeats (the two rules composing); a bolded protocol word
    mid-sentence still does NOT fire; a tool row carrying **PR:** still does not classify;
    and the digest still differs between a decorated and an undecorated report, with
    --mark-handled round-tripping on the decorated one;
  • a tool line carrying STANDDOWN, PR: and an error phrase classifies as no-prefix,
    and does not feed the delivery counters either;
  • prose opening with PR #6580 does not fire;
  • a banned match outside fleet_worktrees is foreign, not banned; a subdirectory of
    a worktree is fleet-owned; a sibling named wt-a-old is not swallowed by wt-a; a
    worktree reached through a symlink still matches; an extended-length \\?\ path
    normalises to the root it names; an unreadable cwd is unknown and still reported; the
    argv still never appears;
  • the program-path fallback attributes a foreign venv, treats a fleet interpreter as owned
    even when the cwd is unreadable, leaves a system/shim interpreter unknown and still
    reports it, and decides on the path while a secret in an ARGUMENT never reaches the
    output;
  • -n0, -n 0 and --numprocesses=0 are each present as their own fixture in the
    bounded-spellings test (verified by name, not inferred from the \d branch), alongside
    the existing -n 4 / -n=4 / -n4 / --numprocesses=4 cases, while -n auto and a
    bare pytest still fire;
  • NOPROGRESS fires on a session held warm by nudges it never answers, and NOT on the
    cycle right after a disposition; one produced row (a message or a single tool call)
    clears it; a live BLOCKED outranks it; it is suppressible and expires like IDLE; and
    a session the conductor has never acted on cannot be stalled yet;
  • an inbound nudge, inject or user row does not advance the index while a tool row
    does, and a row QUOTING transcript JSON does not advance it either -- so the
    line-anchored needle is pinned independently of json.dumps escaping;
  • one entry per key means a later mark can erase an earlier fact, so four
    separate assertions pin that it does not: a condition mark preserves an
    answered payload, so does any other later mark, the legacy state shape carries
    both halves of it across an upgrade, and an answered ERR does not bury a
    ruling nobody has answered;
  • the delivery counters appear on the OK line, are configurable, and a bad
    init_timeout_res or watchdog_res regex is malformed config (exit 2), not a crash;
  • a relative fleet_worktrees entry is malformed config;
  • a state file written by the old version still loads and still suppresses, and an old
    config with none of the new keys still produces a full OK line.

Repo gates run on the two changed files: black, isort, flake8, mypy clean, plus
check_brand_name, check_builtin_skill_scope, check_loop_bound_locks,
check_black_formatting, check_testpaths_coverage and check_harness_parity.

Any other suggestions on the work

The coverage half of the original scope is deferred, and the baseline is untouched.
.github/coverage-baselines/backend.txt records fleet_probe.py 14.8 # 34/229, and it
is left exactly as the default branch has it. The tests in this PR do cover the script
(95% when coverage is sourced at the tree the tests run against), but that measurement is
not what the gate reads: CI's own artifact records 47/331 = 14.2%, which is the
import-only footprint. Removing the exemption on the strength of a local number makes the
file a new_offender at 14.2% and reds a required lane for every open PR, so the exemption
stays and 2f is honestly unfinished rather than quietly claimed.

A sharper diagnosis for #7597 than the one currently recorded there. The comment above
that baseline entry, and the repo's investigation of the issue, both rest on a control case:
that a sibling script loaded through the same helper is unbaselined and passes the gate,
which would mean the loader attributes coverage correctly and the problem is specific to
this file. CI's coverage artifact does not support it. credit_spend.py is absent from
the report entirely
-- 0 occurrences among 1,233 measured classes -- so it is never judged
rather than judged and passing. fleet_probe.py is the only file from that skill directory
present at all, and it enters import-only. Whatever the root cause is, it is not the loader,
and an analysis resting on that control case is reasoning from a case it never measured.

An explicit -n <N> is the LESS safe pytest spelling on this repo, which is why the
banned rule's old comment was misleading.
setup.cfg documents that -n auto does not
mean one worker per core here: the rootdir conftest's pytest_xdist_auto_num_workers hook
(conftest.py:856) sizes the pool by available memory and by what concurrent runs on the
host already hold, and "an explicit -n <N> bypasses the budget". So guidance of the form
"use a bounded -n 2" hands a fleet the one spelling that can outgrow the host, while
-n0 -- the repo's own documented override -- is genuinely in-process and costs nothing.
The rule's SENSE is deliberately unchanged, because which shapes it flags decides what the
conductor stops mid-turn across a whole fleet; only its stated premise is corrected, and
-n0 is now asserted bounded rather than assumed.

Two observations for whoever owns the probe next:

  1. tail_bytes caps how much of a transcript is PARSED, not how much is READ. The
    implementation is path.read_bytes()[-max_bytes:], so the whole file is loaded
    either way. That is what makes the monotonic index free, and it is worth knowing
    before anyone reads the setting as an I/O bound on a large session.
  2. The digest is keyed on the classified tail text, so this change rotates digests once:
    suppressed signals whose classification moved will re-fire a single time on the first
    cycle after deployment. That is the documented degradation (a handled signal re-fires
    once, never a crashed patrol), not a new failure mode.

Accepting a finding is not the same as agreeing with its reasoning. The line-anchored
index needle was taken from a review suggestion whose stated case cannot actually arise:
json.dumps escapes inner quotes, so a row quoting {"role": "assistant" is stored as
{\"role\": ... and never matched the needle. The fix stands on its own -- anchoring is
correct without depending on that argument being sound, and it also covers a torn line at
the window edge -- so it was accepted for better reasons rather than rebutted.

Residual: one entry per key. The handled set stores one record per session, and that
single premise carries three jobs -- suppression, terminality, and progress. Terminality and
stickiness are therefore SPECIAL CASES layered on top of the record rather than properties of
the data model: a terminal disposition survives a later mark only because the payload tag and
digest are carried forward explicitly, and a sticky BLOCKED reaches past an answered ERR
only because the firing path looks for it. Both work and both are asserted, but neither falls
out of the shape.

Stickiness also has a bound the state file cannot close. It reaches only as far back as the
parse window, so a blocked worker that heartbeats long enough pushes its own report past
tail_bytes and the debt goes invisible. Sourcing it from the handled set looks like the fix
and is not: the recorded tag exists exactly when the ruling was already delivered and the
signal is correctly suppressed anyway, and it is absent in the case that actually bites -- an
undispositioned report ageing out. Reading it would be a no-op or a machine for re-firing
answered rulings, so the transcript stays the source of truth.

A per-tag handled map makes both properties structural instead of special-cased, and it changes
the persisted state schema, so it wants its own change with its own migration test. A follow-up
carries it.

An advisory finding needs the same verification as a blocking one. One suggestion here
-- resolve an unreadable /proc/<pid>/cwd by owner, so another user's process reads as
foreign rather than unknown -- was cosmetic, and taking it introduced a real portability
defect: os.getuid is POSIX-only, its absence raises AttributeError rather than
OSError, and the Windows shards run this scan against a fake /proc. The next lane caught
it in one push; the test written to cover it then depended on the same primitive it was
testing the absence of. It is withdrawn, with the reasoning left at the call site: this file
imports nothing from the package, so it cannot route through platform_compat, and an
unreadable cwd already reports as unknown, which is the fail-open reading that matters.
The two findings treated as defects rather than suggestions -- the index counting inbound
rows, and the delivery counters never ageing out -- were both real and both worth it.

Refs #8029
Refs #7597

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 3, 2026 00:26
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from e252bc6 to 7f9d6bd Compare September 3, 2026 00:36
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from 7f9d6bd to dc42ce9 Compare September 3, 2026 00:51
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 256b20a2a51541bb7b570f787a4a33a7695ffcd4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All checks done. The base already carried the contract (SKILL.md documents TERMINAL, NOPROGRESS, i=, the delivery counters, and the cwd= class — landed in #8034), so this PR is the implementation converging on an already-merged spec; the emitter-literal and writer-format couplings are pinned by roundtrip tests against the real constants and the real ConversationLog writer; the state file and CLI compat are explicit and tested; the agent-authored fleet_worktrees key is validated against the widening direction (filesystem root, session store, symlink-to-root). The one design residual is author-acknowledged.

Design-Verdict: CONCERNS

Sound, measured signal fixes — but three protocol semantics now ride special-cased carry-forwards on a one-entry state record the PR itself calls the wrong shape.

Watch

  • The handled map's single entry per key now carries suppression, terminality, and stickiness via explicit carry-forward (settled, legacy proto recovery, the sticky-past-suppressed-ERR reach, NOPROGRESS decided on two separate paths in run_probe). The PR's own history shows the failure mode recurring one door down (the two-field version re-presented answered rulings after an ERR), so the next edit to marking or suppression that misses one carry re-opens exactly these bugs, with only the pinning tests in the way. The per-tag handled map is deferred to a follow-up — hold the author to landing it before any further probe feature stacks on this record.
  • Sticky BLOCKED reaches only as far as tail_bytes, so a blocked worker that heartbeats long enough loses its ruling debt again — the loss class is narrowed, not closed. Disclosed and reasoned; worth remembering when a "why did the conductor miss my BLOCKED" report arrives.

[DESIGN-REVIEWED] 256b20a

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 256b20a2a51541bb7b570f787a4a33a7695ffcd4 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/builtin_skills/pipeline-conductor/scripts/fleet_probe.py:459 -- produced = _count_own_rows(raw) resets after transcript rotation, so the retained-row index can equal the handled index and falsely emit NOPROGRESS -> Fix: store and compare the existing rotation_generation with the index.
[GPT-REVIEWED] 256b20a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 256b20a2a51541bb7b570f787a4a33a7695ffcd4: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 256b20a2a51541bb7b570f787a4a33a7695ffcd4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Verified the counts I'll cite: NOPROGRESS occurs only in fleet_probe.py (defining) and its tests — 0 occurrences in the consumer's contract, SKILL.md, whose tag-keyed action table (lines 397–408) has no row for it and whose lines 75/95/380 instead assign the same comparison to the conductor. init_timeout_res|watchdog_res|fleet_worktrees have 0 setters outside the probe and tests. Everything else in the diff matches a contract already documented in SKILL.md on main. Final review:

First-Principles-Verdict: CONCERNS

NOPROGRESS is a second spelling of the no-progress test SKILL.md already assigns to the conductor, and the consumer's action table has no row for it.

What this change ships

Intent: make the fleet probe report worker progress and delivery truthfully instead of loudness — a FIX for six measured misreadings, carrying extensions.

  1. A protocol word in prose no longer fires a tag (<WORD>: required) — justified, measured 20/94.
  2. Tool-card text can no longer raise a tag or ERR — justified, measured 6/60.
  3. **BLOCKED:**-style decoration no longer silences a report — justified, documented contract (SKILL.md:361).
  4. Every fired line carries i=<index>, monotonic past the window — justified, documented contract (SKILL.md:370).
  5. New TERMINAL tag + persisted settled record: finished workers stop being nudged — justified, documented contract (SKILL.md:404).
  6. BLOCKED survives heartbeats and a handled ERR — justified, documented contract (SKILL.md:114).
  7. New NOPROGRESS fired tag + index in state — duplicate of the conductor-side diff at SKILL.md:75/95/380; zero consumers in its action table.
  8. Banned scan scoped by new fleet_worktrees key; foreign matches hidden — justified (SKILL.md:572–582), but the key has no documented author.
  9. OK line gains foreign and deliver counters — justified, documented admission instrument (SKILL.md:550).
  10. New init_timeout_res/watchdog_res override keys — zero consumers, inherited symmetry with err_res.

Watch

  • The stated cause for NOPROGRESS — "'unchanged across two probes' is … documented nowhere and enforced by nothing" — is contradicted by the repo: SKILL.md:380 documents exactly that rule and SKILL.md:95 gives the conductor a last_index field to enforce it. Moving the comparison into the probe is defensible mechanism-level work, but shipping the tag while the doc still mandates the conductor-side diff leaves two divergent spellings (since-last-mark vs since-last-cycle), and a fired tag the action table doesn't know.
  • fleet_worktrees is the only thing that can ever produce cwd=fleet — the one stopping class — yet SKILL.md:345 tells the conductor to maintain only sessions in probe-config.json; 0 setters exist. The scoping ships defaulted-off with nobody assigned to turn it on.

Subtractions

  • Drop the init_timeout_res and watchdog_res config keys (0 setters outside tests; grepped repo-wide) — keep the DEFAULT_* tuples, which the roundtrip test pins to the real emitter constants; a config override is precisely what that pinning cannot protect.
  • Delete one of the two no-progress spellings: either the conductor-side diff instruction (SKILL.md:75/380, the spec-sync invariant makes that edit belong to this same commit) or the NOPROGRESS tag itself.

[FIRST-PRINCIPLES-REVIEWED] 256b20a

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 256b20a2a51541bb7b570f787a4a33a7695ffcd4 — this comment is updated in place on each push.

Review details

I've reviewed the full source diff (the fleet_probe.py changes end around diff line 1168; everything after is test code) and the candidate.

Falsifying CANDIDATE 1: The candidate claims the module docstring (diff lines 99–105) and the STICKY_TAGS comment reference a legacy proto field that no code reads or writes. I verified this against the code: _recorded_proto() reads entry.get("settled") then falls back to entry.get("tag") — never a proto key — and mark_handled() writes tag, digest, ts, index, settled — never proto. So the comments are indeed imprecise (they call the tag-fallback recovery path a "legacy proto field").

But this is a comment/documentation inaccuracy, not a defect in what the code does when executed:

  • (c) observable wrong outcome fails. The candidate's own harm is explicitly contingent — "If a proto-writing version was ever shipped, the promised recovery does not happen" — and it self-scores medium, admitting it "could not verify whether a proto-writing build ever shipped." No shipped state shape contains a proto key (no version in this code writes one; the "earlier version" named in mark_handled's comment is a pre-merge iteration, not a released one), so _recorded_proto correctly recovers from the entry's own tag for the real {tag, digest, ts} shape. There is no input on which the executed code misbehaves.
  • The instructions are explicit: a finding must be "grounded in what the code DOES when executed, never in what text in the diff says about it," and a hedged "if a caller were to / if X was ever shipped" outcome is a drop. A stale comment naming a nonexistent field executes nothing.

Scored below 80; dropped.

Step 2: No new grounded defect surfaced while falsifying. The classification logic (_classify, sticky/terminal ladder, _owner_class fail-toward-unknown, config validation refusing relative/NUL/root worktrees) is internally consistent and matched by the added tests. Nothing meets the (a)+(b)+(c) bar at 80+.

No findings.

[OPUS-REVIEWED] 256b20a

Verdict parsed from the review's SHA-scoped output markers for commit 256b20a2a51541bb7b570f787a4a33a7695ffcd4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 256b20a2a51541bb7b570f787a4a33a7695ffcd4: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from f745244 to 65d2158 Compare September 3, 2026 01:11
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

GPT's blocking finding is correct and accepted, fixed in 65d2158c4.

Reproduced before changing anything. The branch was gated on tag not in _FIRING, and
that set holds BOTH - and WORKING, so the two shared a branch. A worker that filed
STANDDOWN:, had it dispositioned, was re-seeded, and then reported WORKING: came out
as TERMINAL. A regression test asserting that reds against the previous commit with
assert 'TERMINAL' not in ..., which is the inversion GPT described: TERMINAL tells the
conductor to close the item, so live work would have been abandoned.

The prescribed fix is also the one the design called for, so it is applied verbatim: the
terminal fallback now requires tag == "-". Only an unprefixed tail can inherit a
terminal disposition, because WORKING: is a protocol message and means active work.

The clock is deliberately left governing the other case: a WORKING tail that then goes
silent past the threshold still raises IDLE, which is the correct nudge rather than a
closure. The test pins both directions, and test_a_non_terminal_report_still_ages_into_idle
already covered the second one.

75 tests pass, and per-file coverage on the script holds at 95%.

@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from 65d2158 to e0909f5 Compare September 3, 2026 01:38
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositioning the three lanes on e0909f530bc84049a9603119ba5bb4ca8e1245a7. GPT and Opus
are green; First Principles is CONCERNS (advisory). All three land on the same point, so
here is one answer for it.

The finding is correct: the new signals are not documented here

Opus names it most concretely (SKILL.md:121,128 still says fired lines carry "metadata
only (key, age, tag, digest)" and its action table has no TERMINAL row), First
Principles calls SKILL.md the probe's declared interface, and GPT reaches it from the
consumer side (i=, TERMINAL and deliver have no consumable output path).

That is accurate, and it is deliberate. This PR is one of three landing together against
one interface contract, with exclusive file ownership per branch. The probe's declared
interface lives in three places, and all three belong to a sibling branch:
src/kiro_crew/builtin_skills/pipeline-conductor/SKILL.md,
docs/design/pipeline-conductor.md, and the pipeline-conductor prompt block in
src/kiro_crew/agent.py. That branch documents both scripts from the shared contract,
which is where TERMINAL, i=, the deliver counters and the new fleet_worktrees key
get their action-table rows and their config documentation.

Adding even a one-line mention from here is the merge conflict the split exists to
prevent, so the honest disposition is: correct observation, resolved by the sibling
branch, not by this diff.

Declining GPT's prescribed fix, with the reason

GPT's remedy is "revert these signals until the consumer contract is updated". Declined:
the consumer contract is being updated in the same landing, by the branch that owns it.
Reverting here would invert the problem rather than solve it, leaving the documentation
branch describing signals that no longer exist. The two halves are sequenced by ownership,
not by time.

"fleet_worktrees has no setter, so 2d ships inert" -- measured, and it does not

With no fleet_worktrees configured, the banned scan behaves exactly as it did before
2d: every match is reported, classified cwd=unknown, and counted in banned. That is
asserted, not asserted-at: test_an_undeclared_fleet_scope_reports_every_match.

The choice is deliberate and is stated in the code. Scoping against an empty set would
classify every match as foreign and mute the banned signal entirely, which is a failure
the conductor cannot see. So an undeclared scope degrades to the previous behaviour rather
than to silence, and the feature activates when the config carries the key. A relative
entry is rejected as malformed config (exit 2) for the same reason: it could never match
an absolute cwd, so it would silently mute the scan.

On the deleted baseline entry, since it is in the diff

This repo's own investigation of #7597 reached the same conclusion independently and
before this PR: docs/task-specs/2026/09/task-fleet-probe-coverage-investigation/analysis.md
says "Do not add fleet_probe.py to .github/coverage-baselines/backend.txt", on the
grounds that the file was already covered and a 34/229 reading is "the fingerprint of the
covering tests not landing in the coverage data that got combined".

Its control case verifies: credit_spend.py sits in the same scripts directory, is loaded
by the same load_skill_script in the same test file, measures 96%, and is not baselined.
So the loader attributes coverage normally, and the entry recorded a false rate for a
covered file. Deleting it is the correction; the new tests here cover genuinely new
behaviour rather than existing behaviour the gate had misread.

@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from e0909f5 to d36d56b Compare September 3, 2026 02:11
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from d36d56b to f6cc39a Compare September 3, 2026 02:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from f6cc39a to 1ca65b6 Compare September 3, 2026 03:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 3, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositioning Design Review and First Principles on
f6cc39a65afcf630a997bb99848f430c0d4178f7. Both are advisory, and four items were real
enough to act on. Fixed in 1ca65b6d2.

Acted on

The index was advanced by the conductor's own nudge (Design, Watch 1, second half).
This is the sharpest finding in either review and it defeated the feature's stated purpose:
the index counted rows of every role, so nudging a wedged worker advanced its index and the
next probe read that as progress. It now counts only rows the session PRODUCED -- its own
messages and its tool activity -- and never inbound nudge, inject, user or metadata
rows. Tool rows still count, deliberately: a session running tools is working even while it
is silent. test_an_inbound_nudge_does_not_advance_the_index pins both directions, and the
round-trip file now drives the real writer to pin the needle the count depends on.

The delivery counters never aged out (Design, Watch 2). Correct, and it inverted the
instrument: scanning the whole 200 KB window meant one healed init-timeout kept a session in
the undelivered column until it scrolled out, so a recovered fleet read as chronically
failing. The walk now goes newest-first and stops at the first protocol report, because a
session that has filed a report since the notice evidently got a turn through. Only a match
reached before any report is outstanding. Two tests, one per direction, since recovery is
judged by ORDER and not by presence.

Pin the defaults against the real constants (Design, Suggestion 1). Done, in the
round-trip file rather than the fixture file, which is where this repo already keeps
contracts against real writers. DEFAULT_WATCHDOG_RES is now matched against
dashboard.state.TOOL_STALL_RECOVERY_PREFIX, STALE_RECOVERY_PREFIX and
acp.types.STOP_REASON_TOOL_STALL, so an emitter rewording reds a test instead of silently
zeroing the counters.

Stat the uid for an unreadable cwd (Design, Suggestion 2). Done. A different owner
cannot be a fleet worker, so those become a summarised foreign instead of cwd=unknown
lines the conductor cannot act on. Our OWN unreadable entries still fail open to unknown,
and if the ownership question is itself unanswerable, unknown still wins -- the match is
reported rather than dropped.

Derive PROTO from PROTO_TAGS (First Principles, Subtraction 2). Done; the six words
are spelled once. Ordered longest-first so PR cannot be tried before PROPOSAL, making
the match independent of backtracking, with a test asserting every tag matches as itself.

Declined, with reasons

Drop the persisted entry["index"] (First Principles, Subtraction 1). The reviewer is
right that nothing in this diff reads it. It stays because the interface contract this PR
implements requires it: the handled-set entry gains "index": <n> alongside tag/digest/
ts. Its reader is the conductor comparing an index across cycles, and the action table
that teaches it to do so belongs to a sibling branch by file ownership. Removing a
contracted field because its documented consumer has not landed yet would make the two
halves fail to compose, which is the one outcome the split exists to prevent.

Collapse STICKY_TAGS and HEARTBEAT_TAGS to scalar comparisons (Subtraction 3). The
anti-speculative-generality instinct is right in general and I would normally take it. Here
the category is the contract's unit rather than an anticipated one: TERMINAL_TAGS in the
same file already has two members, and whether PR/GREEN should also be sticky is an open
question I raised with the reviewer of this work rather than answered myself. A set whose
second member is under active discussion is a named concept, not unused generality. If that
question closes as "no", collapsing both to tag == "BLOCKED" / tag == "WORKING" is a
two-line change and I will take it then.

Raised rather than decided

Surfacing i= for non-firing sessions, or a NOPROGRESS tag (Design, Watch 1, first
half).
The observation is correct: a warm-mtime session tagged - or WORKING never
fires, so its index is never printed, and the discriminator is absent for exactly the
sessions it was built for. I have not done it, because printing a line for every watched
session changes the probe's output contract -- today a quiet session prints nothing, which
is what makes a patrol cycle cheap -- and a new tag changes the action table. Both are
contract-level calls rather than implementation ones, so they are with the owner of that
contract. Fixing the nudge-provenance half was not: that one made the index lie about the
sessions it DOES print.

@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch 2 times, most recently from a1a836f to d63b28e Compare September 3, 2026 03:53
buluoray
buluoray previously approved these changes Sep 3, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — d63b28e

Source-verified from a fresh clone at the current head. I read fleet_probe.py in full, ran both test files, and mutation-tested the TERMINAL predicate. The only source delta since the first reviewed head (1ca65b6) is the accepted GPT fix guarding os.getuid() with hasattr(...) so a Windows AttributeError cannot escape the OSError handler and kill the whole scan — a real crash fix, and it has coverage (test_a_platform_without_getuid_still_scans).

Signal predicates and failure directions

(a) NO-PROGRESS — the i=<index> field. Predicate: _tail_entries sets last_index = produced - 1, where produced is a byte-level count of the session's OWN rows (_OWN_ROW_NEEDLES = assistant + the three tool roles), counted over the WHOLE file, never the parse window. The probe does not emit a "no-progress" tag; it emits a monotonic position and the conductor is meant to diff it across two cycles. Correctly reasoned edge cases, both source-confirmed: an inbound nudge/inject/user row does NOT advance the index (so the conductor's own nudge cannot fake progress), and the count is file-relative so it does not saturate past tail_bytes. A slow-but-working worker inside a single long build/model turn produces no new own-rows, so its index legitimately holds still — this is why the discriminator is paired with the WORKING heartbeat and the window is the conductor's judgment, not the probe's. Failure direction: an unreadable/empty transcript yields index=None → prints i=?, which cannot read as "unchanged number" — safe.

(b) FINISHED — the TERMINAL tag. Predicate: tag == "-" (unprefixed tail) AND _recorded_proto(handled, key) in TERMINAL_TAGS (a STANDDOWN/PROPOSAL the conductor already dispositioned, stored in its own proto field). Distinguished from idle-between-turns (no recorded terminal proto → falls through to age > idle_secs → IDLE), from crashed (ERR takes precedence), and — critically — from a re-seeded worker: the tag == "-" guard means a WORKING: tail is never read as finished, because WORKING is also non-firing but is live work. Failure direction: TERMINAL requires an explicit recorded terminal disposition; an unreadable worker is GONE or silent, never fabricated-finished. I mutation-tested this: inverting the predicate to not in TERMINAL_TAGS reddens exactly the 5 TERMINAL/IDLE tests and nothing else — the predicate is load-bearing and the tests are substantive.

(c) UNDELIVERED — the deliver init-timeout <a>, watchdog <b> OK-line counters. Predicate (_tail_matches): walk the window NEWEST-first; return True only if an init-timeout/watchdog pattern is reached BEFORE any protocol report. A worker that filed a report after the notice reads as delivered (returns False). So it cannot fire for a message that was delivered and reported on. Failure direction is over-admission (safe): load/mem can read healthy while this still flags a fleet that cannot deliver.

(d) UNREADABLE STATE. Missing/symlinked-out transcript → GONE (explicit, drives reclaim). Permission error on read → ([], None)i=?, tag -, silent unless already past idle. Malformed/partial JSON → unparseable lines skipped per-row (except: continue); the index still counts via byte needles, so a torn last line does not corrupt it. It never emits FINISHED or a fake unchanged-index on an unreadable worker. Explicit-unknown / silence direction confirmed — the safe one.

(e) Names / prose / contract. The emitted names and docstring prose are accurate and internally consistent. The one gap (see non-blocking below) is that the consumer doc was not updated in the same commit.

(f) Test substance. 69 tests pass on the current head. They drive real writers (ConversationLog, _history_key_for, the token recorder) in the round-trip file so a format drift reds a test, and the agent-file signal tests assert directional behavior (index moves only on own-rows; TERMINAL vs IDLE vs WORKING; counter clears only after a report). The mutation confirmed they are not restatements.

Non-blocking

  1. SKILL.md (the conductor's tag-keyed action table) was not updated in this commit, though the probe's output contract changed: there is no TERMINAL action row, no instruction to diff i= across cycles (so the headline no-progress signal has no documented consumer), and the "metadata only (key, age, tag, digest)" line is now stale (lines also carry i=). deliver/foreign OK-line fields are likewise undocumented. AGENTS.md asks for the doc update in the same commit; the advisory Design Review flagged the same thing. Recommend folding the action-table rows + probe-cycle update into this PR or a fast follow. Not blocking: the script itself is correct and safe, and doc-sync is outside the blocking set.

  2. Minor: _tail_matches clears the undelivered counter only on a protocol-prefixed report. A worker that recovers and resumes plain (non-protocol) output keeps counting as undelivered until the notice scrolls out of the 200 KB window. This is the safe (over-admission) direction and arguably by-design given the protocol requires status reports, so noting only.

Verdict: approve. Blocking: 0. Non-blocking: 2.
Ran both test files (69 passed) and mutation-tested the TERMINAL predicate (reddened exactly its 5 tests). Could not verify: bot lanes have not re-run on the current head d63b28ec (their comments cite older SHAs), and I did not run the full repo suite — only the two probe test files.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from e007baa to e55bdb6 Compare September 3, 2026 09:03
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Accepted. Fixed forward in e55bdb642 rather than by the prescribed revert, on the reasoning
the finding's own sentence supplies.

The defect is real and the anchor is right

fleet_worktrees is read from a config an agent authors, and cwd=fleet is the one ownership
class that STOPS a session. A root of / therefore makes every process on the host read as
fleet-owned and turns the ownership guard into a false-stop generator against unrelated
activity -- the precise harm that scoping this scan was introduced to remove, reintroduced
through the config surface. Trusting an agent-written file to decide who may be stopped is a
trust boundary, not a validation nicety.

Why not the revert

The prescription is to revert configurable-root ownership "until roots come from trusted
pipeline state". Reverting also removes the capability a field measurement justified: in both
attributable BANNED lines observed on this host, /proc/<pid>/cwd was unreadable while the
cmdline read fine, so cwd alone is not a sufficient ownership signal. Two narrower changes close
the widening without giving that up:

1. Dangerous roots are refused at load time, exit 2 with a message -- the same discipline as
the NUL-byte check, and for the same reason: an entry that cannot mean what it says is malformed
configuration, stated at load rather than acted on. Refused: a filesystem root, and any root
that CONTAINS the session store (the same widening one level up -- the store is the conductor's
own data directory, never a worktree, so a root above it makes the conductor and its siblings
read as stoppable workers).

2. Ownership now fails toward the NON-stopping class. Every uncertain answer in
_owner_class and _program_class is unknown, never fleet, including when a path call
raises. unknown re-injects the directive without stopping anyone, so a root that somehow
survives validation still cannot cause a false stop. Fail-safe here means fail-toward-not-
enforcing, because the cost of a wrong fleet is a stopped worker and the cost of a wrong
unknown is a redundant directive.

One thing the mutation test showed that I would otherwise have overstated

Removing the filesystem-root guard did NOT turn the / test red. On POSIX / is always an
ancestor of the session store, so the store rule catches it; C:\ is not absolute on POSIX, so
the absolute-path rule catches that. The root rule earns its place only on Windows, where the
store can sit on another drive and a drive root is both absolute and not an ancestor of it. The
test now asserts the outcome on every platform and says which rule fires where, rather than
implying it isolates that branch. The store-ancestor guard IS independently load-bearing --
disabling it turns its test red.

117 tests pass, script at 93%. Both new tests failed against the previous implementation.

The advisory finding

rotation_generation for the progress identity: still declined, and the reason is unchanged --
no writer in this runtime reuses a transcript path (they are append-only JSONL and the reader
takes a tail), so a rotation counter would be persisted state carried for a case that does not
occur, which is the state-shape growth this PR was told to stop adding.

What has changed is that the bound is now written where the next owner will hit it, as the
condition under which the identity breaks rather than a claim that it cannot: the identity is a
COUNT and is monotonic only while the transcript only grows, so the thing to watch for is the
arrival of a writer that truncates or reuses a transcript path -- that is when a position needs
a generation beside it. A documented bound is honest; a silent assumption rots.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from e55bdb6 to 91c6183 Compare September 3, 2026 09:13
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

All three accepted and fixed in 91c618325. The second one is the more serious of the set and
it was mine, introduced by the previous round's fix.

Blocking: a symlinked root smuggled the wider scope back

Validation compared the literal path only. The classifier compares realpath as well -- that
second chance exists so a symlinked worktree still matches its root -- so a root spelled as a
symlink to / passed validation and then matched every cwd on the host. The widening returned
through the exact door the convenience opened.

Both spellings are now judged against both rules, and a root that cannot be resolved is refused
rather than assumed benign. test_a_symlinked_fleet_root_cannot_smuggle_a_wider_scope covers a
link to a filesystem root and a link to the session store's parent; mutation-checked by
restoring the literal-only comparison.

I surfaced a ruling the conductor could not dismiss

Last round's fix made a suppressed ERR fall through to the sticky BLOCKED underneath it, and
printed a digest over the RULING. --mark-handled recomputes the payload from the transcript,
where the error row is still last, so it digested the ERR text, saw a mismatch, and refused the
mark with exit 3. The ruling fired every cycle and could not be dispositioned.

That is worse than the bug it fixed. A signal that cannot be marked is not a signal, it is noise
with a deadline, and the conductor's only escape would have been to ignore the line -- which
teaches ignoring the whole class. mark_handled now resolves the same payload the probe
surfaced, so the two halves of the protocol agree on what is being marked.

This is also the fourth time in this PR that a fix has left a neighbouring instance standing,
and this time the neighbour was created BY the fix -- I changed what the probe prints without
checking that the writer could still accept it. The pairing to check was "does every signal I
can emit have a working disposition path", and I checked only the emit half.

The docstring had drifted from the code on three counts

All three were true and all three were mine: i= was described as the index of the last LINE
when it counts only rows the session PRODUCED and counts from the start of the file;
TERMINAL_TAGS was listed as STANDDOWN/PROPOSAL after GREEN was added -- the very fix a
review lane surfaced two rounds ago; and the state fields were described as index plus proto
after proto and settled were collapsed into one settled record.

Worth stating plainly: this PR's whole subject is a probe that reported loudly instead of
truthfully, and its own module docstring was describing a contract the code no longer had.
The same rule I have been applying to the PR body -- the description must match the code, and an
edit that re-triggers review is worth it -- applies to the docstring, which is the thing the
next reader actually opens.

119 tests pass, script at 93%. Both new behavioural tests failed against the previous head.

chenmingwei23 added a commit that referenced this pull request Sep 3, 2026
The conductor's claim predicate was one prose line, `gh pr list --search`, and
it was blind in three directions at once. Each blind spot cost a whole worker
dispatch to discover the work did not exist: an item already fixed by a MERGED
PR (an `--state open` query structurally cannot see one), four items that each
had an OPEN PR carrying `Fixes #N` behind a single field that answered empty,
and three items that declared ownership in PROSE.

claim_preflight.py asks all five questions in one call and returns one verdict
on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The
verdict is a pure function of a checks dict, so every precedence branch is a
unit test with no forge access, and an unanswerable question yields UNKNOWN,
never CLAIM.

Six rules earn their own mention because measurement produced each one, not
reasoning:

- The prose scan reads what an author SAYS, not what they QUOTE: the item
  specifying this script quotes the closure phrases it detects, and a raw scan
  returned CLOSE on live work.
- The newest human comment is chosen by timestamp, never by position: the
  comments endpoint ignores `sort`/`direction` and answers oldest-first, so
  asking for `direction=desc` read the OLDEST of twelve comments on a real item.
- A merged PR is coverage only if it CLAIMS to close the item. A bare mention is
  not closure. Measured on a real item: 7597 has TWO landed merged PRs that
  merely reference it, one titled "docs: investigation for #7597", and treating
  either as coverage would have closed an item still being fixed.
- A closure request needs standing (the reporter or a repository insider), since
  CLOSE acts on live work.
- A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any
  commenter can cast is a denial-of-work channel, so an unauthorized claim
  annotates `risk=high` and takes the live recheck.
- An absent symbol vetoes only when the item's metadata corroborates bug-class:
  a feature request names the symbol it PROPOSES to add, so an unconditional
  veto parked that whole class permanently.

`closedByPullRequestsReferences` is not consulted at all. It measured `[]` on
two items that were closed by merged PRs, and a per-candidate forge call that
cannot change the verdict is pure cost against a shared rate limit.

Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true
(that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035,
8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low.

Refs #8029
chenmingwei23 added a commit that referenced this pull request Sep 3, 2026
The conductor's claim predicate was one prose line, `gh pr list --search`, and
it was blind in three directions at once. Each blind spot cost a whole worker
dispatch to discover the work did not exist: an item already fixed by a MERGED
PR (an `--state open` query structurally cannot see one), four items that each
had an OPEN PR carrying `Fixes #N` behind a single field that answered empty,
and three items that declared ownership in PROSE.

claim_preflight.py asks all five questions in one call and returns one verdict
on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The
verdict is a pure function of a checks dict, so every precedence branch is a
unit test with no forge access, and an unanswerable question yields UNKNOWN,
never CLAIM.

Six rules earn their own mention because measurement produced each one, not
reasoning:

- The prose scan reads what an author SAYS, not what they QUOTE: the item
  specifying this script quotes the closure phrases it detects, and a raw scan
  returned CLOSE on live work.
- The newest human comment is chosen by timestamp, never by position: the
  comments endpoint ignores `sort`/`direction` and answers oldest-first, so
  asking for `direction=desc` read the OLDEST of twelve comments on a real item.
- A merged PR is coverage only if it CLAIMS to close the item. A bare mention is
  not closure. Measured on a real item: 7597 has TWO landed merged PRs that
  merely reference it, one titled "docs: investigation for #7597", and treating
  either as coverage would have closed an item still being fixed.
- A closure request needs standing (the reporter or a repository insider), since
  CLOSE acts on live work.
- A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any
  commenter can cast is a denial-of-work channel, so an unauthorized claim
  annotates `risk=high` and takes the live recheck.
- An absent symbol vetoes only when the item's metadata corroborates bug-class:
  a feature request names the symbol it PROPOSES to add, so an unconditional
  veto parked that whole class permanently.

`closedByPullRequestsReferences` is not consulted at all. It measured `[]` on
two items that were closed by merged PRs, and a per-candidate forge call that
cannot change the verdict is pure cost against a shared rate limit.

Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true
(that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035,
8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low.

Refs #8029
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 3, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from 91c6183 to 3b3d4a2 Compare September 3, 2026 10:05
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebased onto merged main (a665d5f04, 41 commits) as one commit, still exactly three files.
119 tests pass, gates clean. Here is the comparison against the now-authoritative SKILL.md,
reported rather than conformed in either direction.

Agrees

  • The four-row ownership table matches the implementation, including the direction ruled
    late: the merged text says cwd=unknown is "NON-stopping ... Never a stop, never a silent
    drop", which is exactly what failing toward unknown guarantees -- an ownership answer that
    cannot be established never promotes a process into the one class that stops a session.
  • The output grammar matches, character for character: the doc's
    BANNED pid=<pid> rule=<regex> cwd=fleet|unknown against the emitter's
    f"BANNED pid={...} rule={matched} cwd={cwd_class}".
  • TERMINAL has a full action row and its description -- "the last dispositioned protocol
    report was terminal and the session has gone quiet since ... Do NOT nudge" -- is what the code
    implements, including that GREEN counts as terminal.

One real gap: NOPROGRESS

grep -c NOPROGRESS on the merged file returns 0. The probe can fire it, so it is a bell
line with no instructed action.

More than an omission, the merged text still describes the workflow this tag REPLACED. It
defines a workers.last_index field as "the previous cycle's probe i=, which is what makes
the no-progress test a comparison instead of something you have to remember", and states "An
unchanged index across two probes is no progress ... check the EFFECT and never liveness". So
the doc asks the conductor to persist last_index and diff it by hand, while the script now
answers the question itself. Both detect the same condition; only one of them is what the code
does, and the doc's version leaves a fired tag with no row.

I am not resolving this in either direction: removing the tag would discard authorized contract
work, and SKILL.md is not my file. Two facts for whoever writes the row: the tag means "this
session has emitted no message and run no tool since you last acted on it", and its action is
the one the doc already gives for an unchanged index -- check the effect, not liveness.

Worth noting the failure mode this closes, since it happened to me mid-PR: my own babysit loop
lapsed over a red PR, and what surfaced it was the conductor checking head SHA and lane state
rather than whether I looked alive. An unchanged digest plus an unchanged head was the reading.
That is this tag's case exactly.

Not a gap, recorded for accuracy

PROPOSAL has no table row either, but the doc discusses it in four places including the
disposition flow, so it is documented rather than missing. The table also carries a cwd=foreign
row while foreign matches are only counted and never printed as lines -- the response there is
"count only", so behaviour agrees and the row describes a line that cannot appear.

Also in this push

The legacy proto fallback is removed, on a First Principles subtraction that measured out:
merged main's mark_handled writes {tag, digest, ts} and no commit on main ever wrote a
proto field, so it only existed in intermediate commits of this branch. The path upgraded a
shape that never shipped, and its test asserted behaviour for an input that cannot occur.

Removing it exposed what it had been masking: the real upgrade path reported IDLE for a
delivered worker. The reader now recovers the terminal reading from the entry's own tag when that
tag is a payload -- a field that actually ships. That also turned an existing compatibility test
from asserting IDLE to asserting TERMINAL, which is the better reading and the harm the
field exists to prevent.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 3, 2026
chenmingwei23 added a commit that referenced this pull request Sep 3, 2026
The conductor's claim predicate was one prose line, `gh pr list --search`, and
it was blind in three directions at once. Each blind spot cost a whole worker
dispatch to discover the work did not exist: an item already fixed by a MERGED
PR (an `--state open` query structurally cannot see one), four items that each
had an OPEN PR carrying `Fixes #N` behind a single field that answered empty,
and three items that declared ownership in PROSE.

claim_preflight.py asks all five questions in one call and returns one verdict
on the exit code: 0 CLAIM, 10 SKIP, 11 CLOSE, 3 UNKNOWN, 2 malformed. The
verdict is a pure function of a checks dict, so every precedence branch is a
unit test with no forge access, and an unanswerable question yields UNKNOWN,
never CLAIM.

Six rules earn their own mention because measurement produced each one, not
reasoning:

- The prose scan reads what an author SAYS, not what they QUOTE: the item
  specifying this script quotes the closure phrases it detects, and a raw scan
  returned CLOSE on live work.
- The newest human comment is chosen by timestamp, never by position: the
  comments endpoint ignores `sort`/`direction` and answers oldest-first, so
  asking for `direction=desc` read the OLDEST of twelve comments on a real item.
- A merged PR is coverage only if it CLAIMS to close the item. A bare mention is
  not closure. Measured on a real item: 7597 has TWO landed merged PRs that
  merely reference it, one titled "docs: investigation for #7597", and treating
  either as coverage would have closed an item still being fixed.
- A closure request needs standing (the reporter or a repository insider), since
  CLOSE acts on live work.
- A self-claim needs standing too, but downgrades instead of vetoing: a SKIP any
  commenter can cast is a denial-of-work channel, so an unauthorized claim
  annotates `risk=high` and takes the live recheck.
- An open fork PR still SKIPs, but not silently. Opening a fork PR needs no
  permission, so rule 2 is a suppression channel anybody can use. Refusing to
  trust fork PRs is the wrong trade -- 192 of this repo's 301 open PRs come from
  forks, so that reinstates the duplicate-dispatch class this script was built
  from -- and the objection was never to the detection but to a response that
  was unconditional AND silent. So the verdict is unchanged and an unvouched
  fork's SKIP carries `untrusted-fork=true risk=high`. Standing is an insider
  association or the item's own reporter, since fixing your own bug from a fork
  is the ordinary case. The consumer is the conductor's review of untrusted-fork
  suppressions.
- An absent symbol vetoes only when the item's metadata corroborates bug-class:
  a feature request names the symbol it PROPOSES to add, so an unconditional
  veto parked that whole class permanently.

`closedByPullRequestsReferences` is not consulted at all. It measured `[]` on
two items that were closed by merged PRs, and a per-candidate forge call that
cannot change the verdict is pure cost against a shared rate limit.

Verified against live items: 8088 answers CLOSE merged-pr=#8092 landed=true
(that PR carries `Closes #8088`), 7597 and 8029 answer SKIP open-pr=#8035,
8031 answers CLAIM risk=high (no bug-class label), 8007 answers CLAIM risk=low.
The fork marker is live too: 6799 and 6509 answer SKIP untrusted-fork=true
risk=high behind first-time-contributor fork PRs, while 8071 stays unmarked
because its fork PR author reported the item.

Refs #8029
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 3, 2026
…nished, and undelivered

The probe told the conductor how loud a session was, not whether it was
making progress, and three of its signals were measurably wrong.

Classification (2e). A protocol word only counts in its protocol form,
"<WORD>:". Matching a bare word boundary read prose as a report: over the
60 most recent transcripts on the development host, 20 of the 94 matching
assistant rows were not reports, 13 of them a bare "PR #<n>". Tool rows
are now dropped before classifying. Role is the transcript's own
discriminator, since the presentation class is never persisted, and the
error half used to read the last row of ANY role, so an error phrase
quoted inside a tool title raised ERR on a healthy worker (a tool row is
last on 6 of those 60 transcripts). No error signal is lost: every
timeout, stall-watchdog and throttle line in that sample landed on an
error/assistant/inject/user/nudge row and none on a tool row.

Tail index (2a). Every fired line now carries i=<n> before d=, and the
handled entry records it. An unchanged index across two probes is no
progress whether or not a turn is open, which is what a self-deadlocked
worker cannot fake. It counts from the start of the file rather than the
start of the parse window: a window-relative count freezes once a
transcript passes tail_bytes and then reads, at the sizes real worker
sessions reach, as the deadlock it exists to detect. It is free, because
the read already loads the whole file and keeps only the window.

TERMINAL (2b). A worker that filed STANDDOWN or PROPOSAL and then wrote
one unprefixed line is finished, not wedged. The handled set keeps one
entry per key, so that terminal disposition was overwritten by the next
tag and the finished worker aged into IDLE, which calls for the opposite
action. The last dispositioned protocol tag now survives in its own
field.

Delivery counters (2c). "deliver init-timeout <a>, watchdog <b>" on the
OK line, counted for every watched session. Load and memory can both
read healthy while the fleet cannot deliver. Both pattern sets are
config (init_timeout_res, watchdog_res) defaulted from the emitters, and
validated like the existing ones.

Banned scan (2d). A banned command shape is only a banned operation when
the fleet owns it. /proc/<pid>/cwd is compared against a new
fleet_worktrees config key; only fleet-owned or unreadable matches are
printed, with the class on the line, and unrelated matches are summarised
as "foreign <n>". An undeclared scope reports everything, because scoping
against an empty set would mute the signal invisibly. The argv still
never appears.

Leading decoration is normalised away (2e, extended). The tag match is
anchored at position zero, so "**BLOCKED:**" puts an asterisk where the tag
has to be: the match fails, the line reads as no-prefix, and on a fresh
transcript IDLE does not fire either, so the report is silent rather than
delayed. Emphasis, strikethrough, code ticks, heading hashes, blockquote
arrows and list markers are stripped from the front before matching. Two
boundaries are explicit because both are observable: the normalisation
applies to the MATCHED text only, so the digest still covers what the
worker wrote, and the anchor survives, so a bolded protocol word
mid-sentence is not a report.

BLOCKED is sticky (2b, extended). The tag is now the newest REPORT, not the
newest message. A probe samples rather than subscribes, and the protocol
requires a blocked worker to keep reporting status, so the worker's own
next message displaced the one place the probe looks. Nothing was marked
and nothing was suppressed: the BLOCKED was never observed, so the
obligation existed on both sides and was visible to neither. A heartbeat
and unprefixed text now leave a sticky report standing; any other report
supersedes it. The digest is keyed on the sticky report's own text, so it
fires once, the ruling quiets it, and only a new blocker re-fires.
TERMINAL and sticky BLOCKED stay separate tags: one says close me, the
other says a ruling is owed.

Banned-ops premise (2d). The pytest rule's sense is unchanged, but its
comment claimed the repo's "-n auto" addopts forks one worker per core.
setup.cfg documents the opposite: the rootdir conftest's
pytest_xdist_auto_num_workers hook (conftest.py:856) sizes the pool by
available memory and by what concurrent runs already hold, and "an
explicit -n <N> bypasses the budget". So on this repo the explicit
spelling is the one that can outgrow the host. The comment now states
what the rule actually catches, a run whose worker count nobody chose,
and says plainly what that costs. -n0 is asserted bounded rather than
assumed, alongside -n 0 and --numprocesses=0, because -n0 is the form the
fleet is required to use and a rule that flagged it would stop every
worker obeying it.

Coverage (2f) is DEFERRED, not delivered. The tests here take the script
to 95% when coverage is sourced at the tree the tests run against, but CI
records 47/331 = 14.2%, the import-only footprint, so the covering tests
are not attributed there and no test can move the gated number. The
existing baseline exemption for this file is therefore left exactly as it
stands on the default branch. Refs #7597, where the sharper diagnosis
belongs: the sibling script in the same directory, loaded through the same
helper, is ABSENT from the coverage report entirely (0 occurrences among
1,233 measured classes), so it is never judged rather than judged and
passing, and any analysis resting on it as a control case is reasoning
from a case it never measured.

--config and --mark-handled KEY TAG DIGEST keep their exact signatures,
and a state file written before the new fields still suppresses: index and
proto are additive metadata outside the digest.

Refs #8029
Refs #7597
@chenmingwei23
chenmingwei23 force-pushed the feat/conductor-probe-signals-8029 branch from 3b3d4a2 to 256b20a Compare September 3, 2026 10:18
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Accepted, correct as traced, and fixed by the prescribed remedy in 256b20a2a. This is the
sharpest finding this PR has had, because it identifies a defect AND the reason my own tests
could not see it.

The defect

A worker files GREEN:, the conductor marks it, and the transcript then never changes again --
the common shape for a finished worker. The trace holds exactly as written: _classify returns
GREEN, which is a FIRING tag, so the terminal/idle ladder that would have said TERMINAL is
never reached. The report is suppressed by digest, so control lands in the no-progress branch.
The index is unchanged because nothing more was written, and the mark is old enough, so the tag
becomes NOPROGRESS -- and because that tag expires, it re-fires every cycle, sending the
conductor to nudge a session that has already delivered.

That is the harm TERMINAL_TAGS exists to remove, reopened for the ordinary in-window case by
the tag I added to detect stalls. A finished worker produces nothing BY DEFINITION, so absence
of output is not evidence of a stall -- the no-progress test is only meaningful for a session
that still owes work.

Why my tests missed it, which is the part worth keeping

test_a_delivered_worker_goes_quiet_rather_than_idle covers the same situation and passes,
because its fixture APPENDS one unprefixed line after the mark -- and that append moves the
index, which is precisely the condition that stops NOPROGRESS firing. The test escaped the bug
by accident of its setup, not by the code being right.

I have hit a version of this before in this PR: the missing GREEN in TERMINAL_TAGS survived
105 passing tests because I exercised the mechanism through PROPOSAL, the tag nobody uses.
Same failure mode one level down -- there the wrong TAG, here the wrong TRANSCRIPT SHAPE. A
fixture that adds a row to "make the scenario realistic" quietly tests a different branch than
the one being claimed. The new test changes nothing after the mark and only moves the clock, and
it is named for the property rather than the mechanism.

The fix

The conversion is now refused when the recorded disposition is terminal, on both paths rather
than only in the ladder. Verified by mutation: removing that clause turns
test_a_delivered_worker_is_not_reclassified_as_stalled red, and it was red before the fix.
120 tests pass, script at 93%, gates clean.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both watch items are already-recorded material and neither asks for a change here. One factual
correction and one confirmation.

The proto recovery is gone as of this reviewed head. grep -c '"proto"' at
256b20a2a returns 0. It was removed earlier in this same round on a First Principles
subtraction that measured out: merged main's mark_handled writes {tag, digest, ts} and no
commit on main ever wrote a proto field, so it only existed in intermediate commits of this
branch -- a compatibility path for a shape that never shipped. So the carry-forward count in
that item is one lower than stated: settled, the sticky-past-suppressed-ERR reach, and the
NOPROGRESS guard on two paths.

The governance ask is already in place, and it is the right one. "Hold the author to landing
the per-tag map before any further probe feature stacks on this record" matches how the deferral
was actually filed: it is a tracked follow-up node with an ordering edge that prevents it being
scheduled until this PR merges, and a second edge to the docs split. It is recorded as the
structural insight this batch surfaced rather than as a nice-to-have, so the next probe feature
meets it first rather than stacking on the record.

The rest stands as disclosed. The one-entry-per-key residual and the tail_bytes bound on
sticky BLOCKED are both stated in the PR body in present tense, including that terminality and
stickiness are special cases rather than properties of the data model. The reading that the
failure mode recurs "one door down" is accurate and is why it is disclosed rather than claimed
solved: four instances of that class were found and fixed during this PR, the last two by review
lanes rather than by me.

On the "why did the conductor miss my BLOCKED" note -- that is exactly the right thing to
remember, and the bound is stated at the definition in code as well as in the body, so the next
reader hits it where the decision lives rather than having to reconstruct it from a PR
description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants