fix(sage): carry a run's failure cause as a token, not only as prose - #8186
Conversation
UX Review (Fable 5) — ✅ PASSUX-level review of UX-Verdict: PASS Failure cards now translate by a stable cause token instead of prose-matching, so reworded backend messages can no longer silently regress to raw English. The screenshots confirm every cause renders as an assertive what-happened + what-to-do sentence (e.g. "Reviewer never started — no kiro-cli executable was found on this host. Install it, or add it to PATH."), the token fallback resolves to existing translated catalog keys in all locales, actionable prose (the [UX-REVIEWED] ca42552 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Additive machine-readable cause token beside prose, with prose-first precedence and permanent legacy-run compatibility — the right shape, alternatives explicitly weighed and pinned by tests. [DESIGN-REVIEWED] ca42552 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsThe candidate list contains no candidates, and my independent falsification of the diff confirms the change is sound: No findings. [OPUS-REVIEWED] ca42552 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/apps/builtins/code_review_sage/backend/routes.py:527 -- " False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All verification is done. The fix is genuine (I confirmed the on-main drift: none of the eight prose branches matches the run-level First-Principles-Verdict: CONCERNS Cause-level fix that earns its surface — but its premise "the dashboard never sees the token" is false: What this change shipsIntent: stop failed-run cards from silently reverting to untranslated English whenever a backend failure message is reworded — a FIX (drift verified on main).
Watch
[FIRST-PRINCIPLES-REVIEWED] ca42552 |
A Sage run's failure cause exists as a stable token in the backend -- `skipped_reason` is one of `no_review_recorded`, `review_record_incomplete`, `runtime_unavailable`, `review_failed` -- but nothing in the payload the dashboard receives named it. `routes.py::_first_change_error` mapped the token to an English sentence before the run was serialized, and each per-change `progress` entry carried prose while the token sat on the driver-internal record beside it. So the dashboard's translator had to recognize causes by their wording, and rewording any backend message silently reverted its card to untranslated pass-through with no test going red -- the frontend's fixtures are its own copies of the backend's strings. That drift is not hypothetical. It has already happened and is on main now: `_first_change_error` maps `no_review_recorded` to "the reviewer finished but wrote no findings record", and NONE of the translator's eight prose branches matches it -- the no-record branch keys on "no result record", the driver's per-change wording for the same cause. The sibling branch for `review_record_incomplete` deliberately covers both of ITS wordings, so this is an asymmetry rather than a decision. That card renders raw backend English today; with the token it explains the cause. The token is now carried BESIDE the sentence, never instead of it: - `_first_change_failure` returns `(sentence, token)` from ONE pass over `per_change`. One pass rather than two functions because the sentence is often built from a record's `deep_error` while the token is that record's `skipped_reason`; an independent scan for the token would return a LATER record's token and label this failure with an unrelated cause. `_first_change_error` keeps its `-> str` signature as a thin wrapper, so every existing caller and its eight assertions across two test files stay untouched -- the change is additive, not a signature migration. - The run record gains `reason` on both error branches. Absent, never blank, when no record carried a token, so a reader cannot mistake one for the other. - Each of the driver's five failure paths puts its token on the `progress` entry it already wrote prose to, agreeing with the `skipped_reason` it sets on the same path. - `failureReason` resolves the token through the same per-change-then-run precedence it already uses for the prose, and consults it exactly where the verbatim pass-through used to be -- which is where a reword lands. The prose path is PERMANENT, not transitional: `progress` lives inside a persisted run record, so every run already on disk has no token and never will. Prose matching is the compatibility path for those runs for good. Two deliberate narrowings, both with a test pinning them: The token is consulted AFTER the prose branches, not before. For one cause the prose is strictly more specific than the token: all three runtime-preflight messages carry the single token `runtime_unavailable`, so a token-first lookup would replace "no kiro-cli executable was found" and "the ACP runtime is not importable" -- one of which carries the exact repair command -- with the generic sentence. The translator's own comment on those branches already makes this argument about a bare /kiro-cli/ pattern; keying on the token first would reintroduce the loss it warns about by another route. `review_failed` is emitted but is NOT in the translator's token table. The other three label prose the backend wrote, a closed set of fixed sentences the token faithfully stands in for. `review_failed` labels whatever the failed dispatch returned, so its prose is arbitrary and usually far more specific: the missing-agent-spec message arrives on that path carrying its own `kirocrew setup --agent-only --clean` repair command. Adding it to the table reddens the verbatim guard, which is how that was caught. The screenshot harness grew the reword case, and an assertion that made it observable: its per-card check finds a card BY its expected sentence, so it could not see a regression on a cause whose sentence another fixture also renders -- the `find` returned the healthy sibling. It now also asserts no cause's raw backend wording appears anywhere in the list. The two committed frames are regenerated from it; `0-before-*` is left alone, being the historical pre-fix frame from #7686. Refs #7688
3502c6e to
ca42552
Compare
|
First Principles is right and I have corrected the description rather than The premise was false. I checked the three claims independently:
So On whether the backend field is preferred over consuming the existing one --
Net: the added surface is one One thing the review got exactly right that is worth echoing for the record: the |
bolichen97
left a comment
There was a problem hiding this comment.
Approving on the strength of a full readiness audit of every open PR against main, not a
line-by-line reading of this diff — recording that plainly so the next reader knows what this
stamp does and does not cover.
Verified against this exact head SHA:
readiness: passedpresent, andPR Readiness— the one required status context onmain
(rulesetprotected-branches) — issuccesson this head.- No check run on this head is
failure,cancelled,timed_outor still in flight. Skipped
jobs are path-filtered conditionals, none of them required. mergeable: true, and the head is not far enough behindmainfor its green CI to describe a
base that no longer exists.- No surviving reviewer
CHANGES_REQUESTED: any such review is on an older commit and therefore
already dismissed bydismiss_stale_reviews_on_push. - Every issue comment, inline review comment and review thread was read and classified. Nothing
left is an unresolved human change request — the remainder is bot review-lane output, resolved
or outdated threads, explicitly non-blocking suggestions, and author status notes.
Auto-merge (squash) is armed, so this lands once every other ruleset requirement is met.
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
Diff shape, up front: 10 files -- 8 source/test, and 2 regenerated screenshots
under
temp-screenshots/sage-cause-translator/(the two frames this PR's ownharness produces;
0-before-*is left untouched, being the historical pre-fixframe from #7686). No other generated assets.
Problem / Motivation
A Sage run's failure cause exists as a stable token in the backend --
skipped_reasonis one ofno_review_recorded,review_record_incomplete,runtime_unavailable,review_failed-- but nothing in the payload thedashboard receives named it.
routes.py::_first_change_errormaps the token to an English sentence beforethe run is serialized, so
run["error"]is prose only.progressentry carries prose and no token, so the entry thedashboard reads per change cannot name its own cause.
Correction, from First Principles review on this PR: an earlier version of
this description said the token "sat on the driver-internal
per_changerecord... which the dashboard never sees". That was false and is worth stating plainly
rather than quietly editing.
run["summary"] = summary(routes.py:491) includesper_change, and_handle_runs(routes.py:914) returns run dicts whole, sosummary.per_change[].skipped_reasonIS already in the/runspayload forcompleted runs. The frontend even declares the field (
per_change?: unknown[],types.ts:72) with zero readers. So a frontend-only fix consuming that existing
field was available for completed runs, and the reviewer was right that it was
never weighed.
It is still not the shape this PR takes, for three reasons that hold
independently of the correction:
summaryat all.run["summary"]is assignedonly after
run_reviewreturns; a run interrupted by a gateway restart ispromoted from
runningat routes.py:215 having never got there.failureReasonhandles
status === 'interrupted', so for those runsprogress[cid].reasonisthe ONLY carrier a token can arrive on.
progressis keyed by change id;per_changeis an unordered list. Thetranslator resolves the per-change cause first precisely because a multi-PR
run's run-level error may belong to a different change. Reading
per_changemeans re-deriving that mapping in the frontend.
built from a record's
deep_errorwhile the token is that record'sskipped_reason;_first_change_failureguarantees the pairing in one pass andhas a test for the mis-pairing. A frontend consumer would have to re-implement
that, against a field currently typed
unknown[].So the backend field is one
stringwith a stated contract, rather than thefrontend taking a dependency on the driver's internal record shape.
So
failureReasonhad to recognize causes by their wording, and now carrieseight prose-keyed regexes for that.
Why it matters
Rewording a backend message silently reverted its card to verbatim
pass-through: the regex stops matching, the reader gets untranslated English
again, and no test goes red -- the frontend's fixtures are its own copies of the
backend's strings. That is the defect #7242 fixed from the outside, and the next
reword re-creates it.
And the drift has already happened -- it is on
mainright now._first_change_errormapsno_review_recordedto "the reviewer finished butwrote no findings record", and NONE of the eight prose branches matches it: the
no-record branch keys on "no result record", which is the driver's per-change
wording for the same cause. I checked all eight against that sentence on main's
own
format.ts; zero match, so that card renders raw backend English today.The sibling branch for
review_record_incompletedeliberately covers both ofits wordings (per-change and run-level, see the comment on it), so the gap is an
asymmetry rather than a decision. This PR closes it for new runs via the token.
What changed (motivation -> approach -> change)
The token is carried BESIDE the sentence, never instead of it, so nothing that
reads the payload today changes.
_first_change_failure(summary) -> (sentence, token)does it in ONE pass overper_change. One pass rather than two functions because the sentence is oftenbuilt from a record's
deep_errorwhile the token is that record'sskipped_reason; an independent scan would return a LATER record's token andlabel this failure with an unrelated cause. There is a test for exactly that.
_first_change_errorkeeps its-> strsignature, as a thin wrapper. Itis pinned by eight assertions across two files (
tests/test_backend_routes.pyand
test/test_sage_backend_routes_coverage.py, including a table test overthe reason-to-sentence mapping).
tests/test_backend_routes.pyis untouched bythis diff, and the change to the coverage file is a pure insertion, so those
pins are byte-identical and still passing -- the change is additive, not a
signature migration.
reasonon both error branches. Absent, never blank,when no record carried a token, so a reader cannot mistake "no token" for one.
progressentryit already wrote prose to, agreeing with the
skipped_reasonon that path.failureReasonresolves the token through the same per-change-then-runprecedence it already uses for the prose, and consults it exactly where the
verbatim pass-through used to be -- which is where a reword lands.
The prose path is permanent, not transitional.
progresslives inside apersisted run record (
runs.json), so every run already on disk has no tokenand never will. Prose matching stays as the compatibility path for those runs
for good; it is not scaffolding to remove in a follow-up. The acceptance test
for this change is a token-less record still rendering the correct sentence.
Two deliberate narrowings of "let the translator key on the token", each with a
test pinning it:
runtime-preflight messages carry the single token
runtime_unavailable, so atoken-first lookup would replace "no kiro-cli executable was found" and "the
ACP runtime is not importable" -- one of which carries the exact repair
command -- with the generic sentence. The translator's own comment on those
branches already makes this argument about a bare
/kiro-cli/pattern;keying on the token first reintroduces the loss it warns about by another
route. Prose-first changes behaviour ONLY on the pass-through path.
review_failedis emitted but is NOT in the translator's token table.The other three label a closed set of backend-authored sentences the token
faithfully stands in for.
review_failedlabels whatever the failed dispatchreturned, so its prose is arbitrary and usually far more specific -- the
missing-agent-spec message arrives on that path carrying its own
kirocrew setup --agent-only --cleancommand. It is still emitted, becausethe field's contract is "the cause token" and omitting one value would make a
review_failedentry read as having no token at all.Tests
Backend,
test/test_sage_backend_routes_coverage.py(119 passed):TestFirstChangeFailureToken-- the token accompanies the humanized sentence,comes from the SAME record as the sentence, and is empty when that record kept
none.
TestRunPayloadCarriesReasonToken-- drives the real_run_review_bgwith thedriver, pool and persistence stubbed, asserting the token on the record the job
leaves behind, on both error branches, absent on success.
Driver,
tests/test_review_driver.py(59 passed) -- one test per failure pathasserting the
progressentry's token AND that it agrees with the record'sskipped_reason, including every change of a preflight-failed run.tests/test_backend_routes.py(222 passed) -- untouched, and passing, which isthe evidence the string function's contract is intact.
Frontend,
src/test/CodeReviewSageRunList.test.tsx(48 passed) -- thetoken-less acceptance table over nine real backend strings (asserting the
fixture carries no token, then that each still translates); the already-drifted
run-level no-record sentence, asserted both before and after; a reworded message
translating by its token; the token read from the named change; a matching prose
branch winning over the token; the agent-spec message staying verbatim despite
carrying a token; an unknown token falling back to prose.
Mutation-verified, 14 mutations, each reddening exactly its guarding test on an
AssertionError(not an exception), all Python runs withPYTHONPATHpinned tothe worktree so the shared venv's editable install could not silently test
main:run-level assignment deleted.
review_failedadded to the table;per-change token read dropped; unknown token no longer falling back;
no_review_recordeddropped from the table; and -- for the persisted-recordproperty -- token-less records bypassing the prose chain, which reddens the
acceptance table plus nine pre-existing prose tests, so old runs cannot
degrade without a wide red.
Two of those enforce the narrowings above rather than leaving them as comments:
adding
review_failedto the table reddens the verbatim guard, which is howthat regression was caught before it shipped.
Screenshots / video
The list, one card per cause. The 6th card (59m ago) is the reword case: its
backend sentence matches no prose branch, and it renders the translated
explanation via the token. The 5th is the missing-agent-spec message, still
verbatim with its repair command intact.
The detail notice, showing the explanation with the driver's raw wording kept
beneath it.
Manual verification
Both frames come from
website/scripts/capture-sage-cause-translator.mjsrunagainst the real built SPA, and are the files it writes, so a later re-run lands
on the reviewed images.
That harness grew the reword fixture -- and an assertion that made it
observable. Its per-card check finds a card BY its expected sentence, so it could
not see a regression on a cause whose sentence another fixture also renders: the
findreturned the healthy sibling's card and the broken one was neverexamined. The first run passed with the token lookup deleted, which is how the
hole surfaced. It now also asserts that no cause's raw backend wording appears
anywhere in the list, and with that it fails under the same mutation, naming the
untranslated text.
Operator-facing path, for the record:
failureReasonfeedsFailureNotice.tsxand
RunCard.tsx, both of which render itstext. This change surfaces thetoken through that existing channel and adds no new one.
Related Issues
Refs #7688
A bare
Refsrather than a closing keyword, and not for the usual reason. Theissue's title is the payload dropping the token, and narrowing 2 above leaves
review_failedkeyed on prose deliberately -- so "the token keys thetranslator" is by design only partially true, and
Closeswould misreport thatdecision as a complete fix.
Both remaining prose-keyed cases are now filed as #8198, which is also where the
reasoning for keeping
review_failedprose-keyed is recorded -- the table readsas though an entry were forgotten, and the obvious symmetry "fix" reddens the
verbatim guard and drops the missing-agent-spec repair command:
Refs: a failed dispatch reportingok=Falsewith an EMPTY error string renders the backend's own "the review turn failed",
which is reword-fragile but arrives on the path that must stay prose-keyed. It
is untranslated on
maintoday as well, so this change neither closes norworsens it.
no_review_recordedsentence matches no prose branch on
maintoday. This PR fixes it for new runsvia the token; records already on disk still need the branch widened, the way
the incomplete-record branch already covers both of its wordings.
#7688 should stay open for those.
Housekeeping for a maintainer: #7688 carries a
blockedlabel that has beenstale since 2026-09-02T04:35:55Z. It was applied for a stated prerequisite,
#7686, which merged at that timestamp (
df653d1e7, an ancestor ofmain), sothe label can be dropped.
Pattern harvest
Rule candidate: review-prompt
Pattern: a discriminating token collapsed into human prose at a serialization
boundary, leaving a downstream consumer to recover it by pattern-matching the
prose. The prose is the presentation and the token is the contract; when only
the prose crosses, every reword is a silent breaking change with no failing
test, because the consumer's fixtures are its own copies of the producer's
strings. This PR carries its own proof that the failure mode is real and not
theoretical -- one backend sentence already has no matching branch on
main.Worth flagging wherever a producer maps an enum to a sentence and the consumer
regexes it back.
Second, narrower candidate for the same prompt: an evidence harness that locates
its subject BY the subject's expected output cannot observe a regression in a
subject whose expected output a sibling fixture also produces -- the lookup
silently returns the healthy one. Assert the complement (the bad output appears
nowhere) rather than the per-subject form.