fix(cron): retry ACP process death on the type, not the message wording - #7396
fix(cron): retry ACP process death on the type, not the message wording#7396SebastianYuSun wants to merge 1 commit into
Conversation
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/acp/client.py:7784 -- “never from |
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of All claims verified against the base tree. I have enough to render the design review. Design-Verdict: CONCERNS Sound root-cause fix — type over wording, fail-closed — but the retained substring arm still permits the exact mid-turn replay the new invariant forbids. WatchThe legacy substring arm remains a mid-turn replay channel that contradicts the PR's own invariant. Suggestions
[DESIGN-REVIEWED] 9f07f6d |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsBased on my analysis, I've independently traced the candidate. Let me confirm one final detail: that I've now verified:
No blocking issues; one advisory finding. FINDING — src/kiro_crew/slack/gateway.py:5261 — the retained [OPUS-REVIEWED] 9f07f6d |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All claims verified against the base tree. The counts in the description match my greps exactly: 3 First-Principles-Verdict: PASS Scoping the substring arm removes base's retry of mid-prompt "Process exited during prompt" deaths — cron jobs that self-healed those now take a hard failure; confirm that trade. What this change shipsInventory (5 items)Intent: stop a healthy cron job from being durably auto-paused when its ACP child dies before the prompt was ever delivered — a FIX.
The defect has checkable provenance: the base predicate at [FIRST-PRINCIPLES-REVIEWED] 9f07f6d |
92c3b5f to
2621b28
Compare
|
Rebased onto The finding was correct. The fix is not the prescribed revert. Reverting the type arm would also drop the three pipe-broken sites this PR exists to cover, and it would put the retry decision back on exception wording, which is the defect class itself. Instead the guard now excludes the one subclass that is unsafe to retry: (isinstance(exc, AcpProcessDied) and not isinstance(exc, AcpToolStalled))
The legacy substring arm is still there for the five plain- Tests: |
2621b28 to
8125735
Compare
8125735 to
20c259a
Compare
|
Round 4 on GPT 5.6 (blocking) — closedThe finding is right: Enumerating every raise site rather than reasoning from the taxonomy:
The last two are sites First-Principles named that I had missed, and The invariant is now about when the death was discovered, not which subsystem noticed. Only transport-write sites are retryable, which is exactly the class this PR exists to cover. Not the prescribed revert, for the third round: reverting the type arm drops the three pipe-broken sites this PR is for, puts the decision back on message wording (the defect class), and leaves the mid-turn hazard untouched at all four sites. First Principles (advisory) — subtraction taken
The undeclared-surface point is fixed: the description now lists all four items (predicate, attribute + kwarg, two The false docstring premise is rewritten to state the real invariant, and records that the earlier wording was false so it is not reintroduced. Deferred, with reasons
Tests
903 passed across the cron, ACP-client, ACP-error-surface and slack-gateway suites. All local gates green, including the repo's own 14 bare-invocation script gates ( |
20c259a to
cb85ad9
Compare
Round 5 — all three lanes named the same defect, and it is realOpus 4.8 (blocking), Design and First Principles independently converged on one Measured before changing anything. The shipped predicate, driven with every
Fix, at cause. The substring arm is now scoped so a typed death is decided The default is inverted, per @Design's argumentDesign and First Principles both showed the audit was incomplete — That is two consecutive rounds of an incomplete enumeration, and it is the
The three sites you named are now correct without being listed, which is the point. So the test changed direction too. Asserting "every in-flight site opts out" Subtractions taken
On the one point where the lanes disagreedAn earlier round had Opus calling the hazard pre-existing while another lane called Verification
Local gates on this head: black (repo wrapper), isort, flake8, mypy over One diff-scope note: formatting the touched test file graduated it out of The |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
cb85ad9 to
79ef804
Compare
|
Rebased onto main Clean rebase, no conflicts. The diff is unchanged (5 files, +520/-13); only the surrounding code moved, so the gateway cron hunk now lands at Gates run locally on the changed files only: Please review the rebase result. A maintainer push makes the maintainer the last pusher, so under the repo's last-push rule a second approver is needed. Reply if anything looks wrong. |
The cron failure handler has a retry branch built for ACP process death, but it
identified that death by string-matching the exception message for "not running"
or "process exited". The most common death is discovered as a broken pipe on the
next write and is worded "ACP process pipe broken: <cause>", which matches
neither substring -- so the branch built for process death was skipped for the
signature that produces it most often. A recoverable one-off child death was
recorded as a hard failure instead, marching consecutive_failures toward
auto-pause.
The generic transient ladder does not cover the gap either: AcpError.transient
defaults to None at that raise site and the message carries no throttle/5xx
marker, so acp_error_is_transient falls back to marker-matching and returns
False.
Test AcpProcessDied, with two deliberate narrowings.
AcpToolStalled, a new subclass, is raised only by the tool-stall detector and is
excluded from the retry. It is the one death where a tool HAS been dispatched, so
its side effects may already have completed and resubmitting the prompt can
repeat a mutation. A subclass, so every other `except AcpProcessDied` handler
keeps catching it.
The two legacy substrings stay as an explicit fallback arm rather than being
replaced: five sites spell process death as a plain AcpError ("ACP process not
running" for a missing stdin, "ACP process exited (code=...)"), so a bare type
swap would silently stop retrying those.
Five of the new tests fail on unfixed source; the parametrized legacy-wording
case and the raise-site anchors pass on both by design.
79ef804 to
9f07f6d
Compare
|
Thanks for the rebase @bolichen97 — all four of your factual claims verify, and one consequence needs flagging because it is not visible from the rebase itself. Your claims, checked against the code:
The consequence: the rebase carried forward a head that three lanes had already blocked.
That was fixed locally before your audit ran, but the fix had not been pushed — my mistake, and the reason your rebase picked up the stale head. Now pushed, rebased onto your What changed from
Why the safe claim is sound at One gate interaction worth your attention, unrelated to my diff.
But Local gates re-run on the new head, not inherited from the pre-rebase one. Separate finding for you, not caused by this diff:
|
|
Follow-up on the comment-history ratchet, now with CI's own verdict rather than my local measurement — and I've pinpointed the commit, so this should be a two-minute fix for whoever owns it. CI confirms the diff is not the cause
Where the 2 markers came from
Baseline still records 67; main measures 69. Consistent with that, Why this blocks any PR touching
|
Problem / Motivation
A cron job whose ACP child dies mid-turn is recorded as a hard failure rather than
retried, even though the failure handler contains a retry branch built for exactly
that case. The branch identifies process death by string-matching the exception
message for
"not running"or"process exited", but the most common death isdiscovered as a broken pipe on the next write and is worded
"ACP process pipe broken: <cause>"— which matches neither substring.So the retry that exists for this failure is skipped for the signature that
produces it most often. Observed shape on a real job:
last_status: "error",last_error: "ACP process pipe broken: Connection lost", on a job whose previousrun completed normally and whose host was not persistently broken.
Why it matters
consecutive_failuresauto-pauses a job at_AUTO_PAUSE_THRESHOLD, and a pausedjob never fires again. So repeated one-off child deaths durably disable a healthy
job instead of self-healing. The degradation is silent in the worst way: the retry
looks present in the code, so the behaviour reads as "ACP death is handled".
The generic transient ladder does not cover the gap either.
AcpError.__init__defaults
transient=Noneand the raise site passes notransient=kwarg, soacp_error_is_transientfalls back to marker-matching a message that carries nothrottle or 5xx marker, and returns
False. Both ladders miss, so control reachesrecord_failure().What changed (motivation → approach → change)
Symptom: a cron job accumulates hard failures for a recoverable transient.
Root cause: the retry predicate reads the exception's wording instead of its
type.
AcpProcessDiedis raised at every site that discovers a dead child, is asubclass of
AcpError, and is already imported ingateway.py— it was simply notconsulted.
Change: decide on a
resubmit_safeflag the ACP layer sets at the raise site,and keep the two legacy substrings as a fallback arm scoped to plain
AcpError.The full surface, declared.
slack/gateway.py— the cron retry predicate reads the exception's type plusresubmit_safeinstead of its message wording.acp/client.py— new publicresubmit_safeattribute +__init__kwarg onAcpProcessDied, mirroring the existingAcpError.transientprecedent.acp/client.py—_send_promptre-tags a caught death as resubmit-safe.acp/session_provider.py— the pre-conversation liveness check passesresubmit_safe=True.The invariant is about WHEN the death was discovered, not which subsystem
noticed it. With a turn in flight a tool may already have completed its side
effects, so resubmitting the prompt can repeat a mutation.
The default REFUSES, and only two sites claim otherwise — each one where the
turn provably had not started:
resubmit_safeclient.py_send_promptTrue(explicit)session_provider.pyruntime not aliveTrue(explicit)AcpProcessDiedconstructionFalse(default)The claim is made at
_send_prompt, not at the transport write, and that placementis the whole point. A transport write cannot know turn state: the shared
_send_requestalso carriessession/steerandcommands/execute, and its siblings_send_response/_send_errorexist ONLY to answer requests the child raisesmid-turn — tool-permission replies and unknown-method rejections. Tagging the shared
write safe therefore mislabels the majority of its callers, and the concrete harm is a
replay: a cron turn completes tool 1, the agent requests permission for tool 2, the
child dies, the approval write raises a safe-tagged death, and the resubmitted prompt
runs tool 1 again. An earlier revision of this PR did exactly that; reviewers caught
it, and the fix was to move the claim rather than to defend the tag.
Why a
drain()failure at_send_promptimplies nothing ran — this is notobvious and was measured rather than assumed.
write()never raises; the death alwayssurfaces at
drain(); anddrain()fails iff unflushed bytes remain (a child thatconsumed the whole payload drains clean even if it dies immediately after). Wire
messages are newline-delimited JSON-RPC, so bytes still queued mean the child never
received a complete
session/promptline, and it cannot dispatch a tool for a requestit has not finished reading. Partial consumption is reachable — a child that reads
a prefix then dies does raise here — but a prefix is not a parseable request. The code
carries this reasoning, including the reachable case, so it does not read as a
counterexample later.
Defaulting to
Trueand having each in-flight site opt out would cost about the samein kwargs, but it is only correct while the set of in-flight sites is completely
enumerated — and that set is not locally checkable. Reviewers of earlier revisions
found in-flight sites that explicit audits had missed, including ones reachable only
through
session_provider._translate_dead. Fail-closed removes the dependency on thatenumeration being exhaustive, so the tests pin the enumerable direction instead:
exactly which sites claim
True.The substring arm is scoped to plain
AcpError, and that scoping is load-bearing.Five raise sites spell process death as a plain
AcpError, notAcpProcessDied(
AcpError("ACP process not running")for a missing stdin, andAcpError(f"ACP process exited (code=...)")), so dropping the arm would silentlystop retrying all five and two existing tests pin that. But left unrestricted the arm
out-votes the type's own verdict:
AcpProcessDied("Process exited during prompt ...")is raised with a turn in flight, and its message CONTAINS
process exited, so thewording resurrected a death the classification had already refused. Restricting the
arm to non-
AcpProcessDiederrors is what makes a typed death decided solely by itsflag.
The narrower alternative — adding
"pipe broken"to the substring list — closesthis instance but leaves the retry decision coupled to exception wording, which is
the defect class rather than the instance.
Known residual, deliberately out of scope.
resubmit_safeis consulted at 1of 3 resubmit sites.
dashboard/chat_runner.pyandtask_executor.pyalsoresubmit on
AcpProcessDiedwithout reading it, so the mutation-repeat hazardpersists at both. That is pre-existing, in files this PR does not touch, and the
right non-resubmit behaviour differs for a foreground turn versus a cron wake — a
design question rather than a mechanical port. With the fail-closed default those
handlers now inherit a refusing verdict on any unclassified death rather than a
permissive one, so reading the flag is a strict improvement whenever they choose to.
Tests
test/test_cron_acp_retry.py. Verified load-bearing by revertingsrc/to theprevious published head and re-running: the three tests below fail, naming the site.
Load-bearing:
test_mid_prompt_death_is_not_retried_despite_legacy_wording— the death whosewording matched the legacy substring and out-voted its own
resubmit_safe=False.Asserts the stream is entered exactly once. This is the test for the defect a
reviewer found in the previous revision, and it fails on that revision.
test_only_the_audited_sites_claim_resubmit_safety— exactly 1 + 1 + 0 sites claimresubmit-safety across
client.py,session_provider.py,session_handle.py.Deliberately pins the OPT-IN direction: the in-flight set is not reliably
enumerable, so a new opt-in is what must justify itself.
test_resubmit_safe_states_the_verdict_without_exporting_the_hierarchy— thedefault refuses, so a death nobody classified cannot be resubmitted.
test_acp_pipe_broken_triggers_retry— the realpipe brokenwording resets thesession and retries once. Returning rather than raising is also what keeps the run
off the auto-pause ladder:
CronService._executecallsrecord_failure()only inits
exceptarm andrecord_success()(which zeroesconsecutive_failures) whenthe callback returns.
test_resubmit_safe_death_retries_regardless_of_wording[something nobody predicted]— shares no substring with the legacy guard, pinning the decision to the
classification rather than to today's vocabulary.
test_pipe_broken_retries_only_once— the_acp_retriedmarker bounds the newarm exactly as it bounds the substring arm.
Controls — these pass on fixed and unfixed source alike, and each exists because
without it the fix could silently become a no-op or over-reach:
test_resubmit_safe_death_retries_regardless_of_wording[a wording that happens to say process exited]— a safe death must retry BECAUSE it is classified safe, notbecause its wording matched.
test_plain_acp_error_still_matches_the_legacy_wording_arm— the five plainAcpErrordeath sites are notAcpProcessDied, so scoping the substring arm toplain errors leaves them matched. Without this, the scoping could narrow the guard
past the defect it repairs.
test_non_process_death_acp_error_still_not_retried— widening to the type did notwiden to every
AcpError.TestAcpProcessDiedRaiseSites(2 tests) — anchor the wording at the real_send_requestraise site and thetransient=Noneclassification, so a futurerewording cannot quietly invalidate the reason this fix exists.
The pre-existing tests in the file are unchanged and still pass.
Manual verification
N/A — unit coverage sufficient. The tests drive the real
GatewayOrchestrator._cron_callbackand the realAcpClient._send_request(with areal
BrokenPipeError) rather than a stand-in, so the wording and the retrydecision are both measured rather than assumed.
The predicate was additionally measured directly against every death shape the code
raises, which is how the previous revision's defect was located rather than argued:
of seven shapes, six agreed with the declared contract and one — the mid-prompt exit
— contradicted it. That one shape is now its own test.
The per-site classification is pinned at source level because reaching those sites
needs a child that dies mid-turn. The assertion is on which sites pass
resubmit_safe=True, not on which take the refusing default: the opt-ins are few and checkable,whereas asserting the in-flight set is complete would restate the assumption that
failed twice.
Screenshots / video
N/A — no user-visible UI change. The diff touches the cron failure handler, the
ACP exception classification, and tests; there is no panel, component, layout or
theme surface involved.
Related Issues
Fixes #7395
Pattern harvest
Rule candidate: review-prompt
Pattern: a retry/classification decision string-matches an exception message when a
typed exception carrying the same meaning is already in scope at the decision site.
The message is presentation; the type is the contract. This generalizes — the same
shape appears wherever recovery logic greps
str(exc)instead of testingisinstance, and each instance fails silently the first time somebody rewords amessage.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement