refactor(prepare-pr): split rationale out and fix six self-contradictions - #6436
Conversation
|
Both findings from the local
Local gates, the 751 pinning tests, and both |
|
The local
Its two "token dropped" observations were checked and are correctly non-guards: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Real harm (contradictory per-invocation instructions), fixed at root cause via the test-endorsed gate-floor layering precedent; fully reversible prose with pinned behavior. Spot-checked the split: the operational halves of every rationale entry survive in [DESIGN-REVIEWED] cae8c6f |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All verifications are done: the First-Principles-Verdict: CONCERNS The split's own rule — "SKILL.md carries only what the loop executes" — is broken by ~6 rationale sentences now living verbatim in both files. What this change shipsIntent: make one heavily-loaded skill deterministic and cheap by fixing its self-contradictions and moving rationale out of the per-invocation load — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] cae8c6f |
…ions The skill's prompt had grown to 68KB of mixed content: normative instructions, script-behaviour documentation, and post-mortem evidence all on one plane. Six places contradicted themselves outright, so the agent's behaviour depended on which paragraph it weighted. Contradictions fixed: 1. monitor_start arming. "The acknowledgement is not evidence the loop armed" was followed four sentences later by "a plain requested acknowledgement is the success signal". Restated as one rule with two branches: a synchronous refusal means fall back to the in-turn wait loop; anything else, including a bare `requested`, means armed. 2. cron. Phase 3 said to prefer the `pr_watch` script cron for pure-watch stretches; the closing section said never hand the loop to a cron. Now one rule, in one place, with the exemption's precondition spelled out (nothing to answer -- no open concern, no bot post still expected). 3. Iteration budget. Four unaligned numbers (outer 10, inner 10, stall at 3, max_cycles 80). Escalation fires at 3, which made 10 unreachable in any healthy run, so 10 is now stated as a runaway backstop rather than a target. 4. The diff freeze. "Once the PR is open, the diff is FROZEN" negated itself in the same sentence and sat in Phase 0 while describing post-open behaviour. Moved to Phase 3 and reworded as the three push justifications. 5. Mode selection. The prepare-only and full-loop trigger lists overlapped with no precedence, so "push this and make it green" had no defined mode. Now a table plus an explicit precedence rule, and the body list matches the frontmatter's rather than offering a narrower one. 6. Proportionality. ~1.1k characters argued a rebut/push-back distinction that records as the same disposition and produces the same action. Collapsed to two questions with two outcomes; the taxonomy discussion moved to rationale. Rationale moved to references/rationale.md, following the pattern references/gate-floor.md already established (and that test_gate_rationale_reference_exists_and_is_pointed_at endorses): incident evidence, script internals, and design history are needed to justify a deviation, not on every skill load. SKILL.md 68690 -> 37699 bytes. No rule was dropped. Two that had been trimmed along with their surrounding prose are restored explicitly: pr_status.py's exit-20 marker semantics (advisory FINDING counts never gate) and the at-most-two-commits allowance. All existing pins pass, including the disposition-vocabulary ratchet, the charter budget parity check, and the Phase-ordering assertions in test_ai_review_workflows.py.
3cb63f2 to
cae8c6f
Compare
|
Rebased onto main HEAD Frontend Lint & Type Check (jscpd) — inherited, cleared by rebase. jscpd found 2 clone pairs at 0.01% over a
Both fixes are ancestors of current main, so the rebase is the whole fix. Verified against the rebased tree rather than assumed: PR Readiness was downstream of that lint failure and should clear with it. No content change — the diff is byte-identical to |
chenmingwei23
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: refactor (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: refactor (skill doc reorg, no behaviour change) - split rationale out of prepare-pr SKILL.md.
Problem / Motivation
prepare-pris one of the most-loaded skills in the repo, and itsSKILL.mdhad grown to 68,690 bytes of mixed content: normative instructions, script-behaviour documentation, and post-mortem evidence all on one plane with no separation.Six places contradicted themselves outright, so an agent's behaviour depended on which paragraph it happened to weight:
monitor_startarming. "The acknowledgement is not evidence the loop armed — and it cannot be" was followed four sentences later by "a plain requested acknowledgement is the success signal, so ending the turn on it is correct."pr_watchscript cron for a pure-watch stretch; the closing section said "Do not hand the iterate-on-review-feedback loop to a cron job." The two were separated only by the undefined terms "pure-watch" and "iterate-on-review-feedback".max_cycles=80. Escalation fires at 3, which makes 10 unreachable in any healthy run — yet 10 appeared six times and was written into themonitor_startmessage template as a target.rebutteddisposition and that push-back "is not a fifth disposition" — a taxonomy discussion about text that produces identical actions.Why it matters
Every byte here is injected on every invocation of the skill, and this skill runs on most PRs in the repo. Three costs compound:
waitfallback on every successful arm, reinstating the exact 2-hour turn timeout themonitor_startbranch exists to remove. refactor: remove dead legacy compatibility shims #3 means the runaway backstop reads as a target, so a loop can burn ten rounds without the operator learning it should have escalated at three.single_commitskip ×4).What changed (motivation → approach → change)
Root cause: the file had no layering. A rule, the reason for the rule, and the evidence that motivated it were all written at the same level, so de-duplication was impossible without deciding what each sentence was for.
Approach: split by function, following the precedent
references/gate-floor.mdalready set in this same skill — and whichtest_gate_rationale_reference_exists_and_is_pointed_atexplicitly endorses ("the reasons each gate is shaped the way it is moved to a reference file so they do not dilute the operational instructions on every skill load").SKILL.mdkeeps only what the loop executes; the reasons move to a sibling reference read on demand, when an agent needs to justify a deviation rather than on every load.Change:
SKILL.md68,690 → 38,054 bytes (−45% of per-invocation load).references/rationale.md(14 KB) carrying the incident evidence, script internals, and design history — including why each of the six contradictions was resolved the way it was.waitloop and say no loop is running; any other reply, including a barerequested→ treat as armed and end the turn.pr_watchonly when you have nothing to answer (no open concern, no bot post still expected), because it reads no comment bodies.rationale.md.pr_status.py's exit-20 marker semantics (advisoryFINDINGcounts never gate) and the at-most-two-commits allowance.Review rounds on this PR found two more dropped guards, both fixed here:
--reviewers/PREPARE_PR_REVIEWERS) had been dropped entirely.resolve_reviewers()'s own docstring is explicit that without itpr_status.pyruns discovery mode, where "a lane that never posted is not required" — so a profile declaringgpt+opuscould reach exit 0 with no Opus review at all, and Phase 4 arms auto-merge on that. Restored in three places: the script table, the poll invocation, and themonitor_startmessage template (the template mattered — the loop would otherwise drop the pin on every subsequent round).LEASE_SHAcase left a rule that runsgit merge-base --is-ancestoragainst a ref that does not exist yet and reads the failure as a maintainer commit — permanently blocking the first push on any new branch, including this PR's own.Tests
No new tests. This diff is prose in an agent instruction file; the behaviour it governs is already pinned by existing tests, and those pins are what constrained the rewrite:
test/test_ai_review_workflows.py::TestPreparePrPreSubmitReview— phase ordering (Reconcile code and description.<Local review — one subagent per profile reviewer<Push only the reviewed commit.), theREVIEWED_SHAfail-closed strings, both reviewer contract paths,Record dispositions.beforeloop back to Phase 1, and all four disposition names.test/test_deferred_disposition_ratchet.py— the vocabulary ratchet, including the per-line rule that no line may offeraccepted-and-deferredwithoutneeds-a-decisionalongside it.test/test_prepare_pr_profiles.py— charter budget parity against.github/review-prompts/(≤5 BLOCKING, ≤6 advisory FINDING, GPT'sreport-ALLwith no numeric cap) and thereferences/gate-floor.mdpointer in both directions.test/test_prepare_pr_local_review.py— the extractor stays the default and the hand-written charters stay the fallback.751 tests across those plus
test_prepare_pr_status.py,test_skills.py,test_builtin_skill_packaging.py,test_builtin_skill_sync_safety.pyandtest_brand_name_gate.pypass on this commit.Manual verification
N/A for runtime behaviour — no executable code changed. What was verified instead:
AF_UNIX path too long,test_xdist_host_budgetcap, env-dependenttest_source_providers); one base flake actually passed on the branch. Comparison done by node id, over whole files rather than parametrized ids.run_scoped_tests --test. The brand gate caught two proseKiroCrew→Kiro Crewspellings, fixed.gpt-5.6-solmirroringcodex-review.yml,claude-opus-4.8mirroringclaude-review.yml+ base-refAUTOSDE.yaml). GPT found the two dropped guards above; Opus then verified token survival rule-by-rule across both files and returned no BLOCKING, with one low-confidence advisory (thesingle_commitconditional centralized away from its steps) which is applied as a pointer on each of the three governed steps.Related Issues
no linked issue: self-initiated refactor of an agent instruction file, no tracked defect.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Why no screenshot: the diff touches only two markdown agent-instruction files under
src/kiro_crew/builtin_skills/; nothing renders in the dashboard.