Repository navigation
fix(aidd-dev): review leaves out-of-diff phases and the verified percent undefined #858
Copy link
Copy link
Closed
Closed
Copy link
Description
Problem
05-review does not say what happens to plan phases outside the reviewed diff, nor how the verified percent is computed. Reviewing one phase of a multi-phase plan, three models on the same skill produced three incompatible review.md files.
The skill contradicts itself:
SKILL.md:30saysreview.mdis overwritten each run: "a snapshot of the current diff, not a history".02-review-functional.md:23says thePhasessection holds "one block per plan phase".- Nothing says whether the previous
review.mdis read before writing. review-template.md:33renders{{pct}}% ({{n_checked}}/{{n_total}}), but nothing says which phases entern_total.not-applicableis only defined as "neither a finding nor a blocker" (review-rubric.md:19), never for the percent, and never when to use it.
Observed on a 7-phase plan, phase 4 reviewed in three worktrees, aidd-dev 2.4.1 (05-review is unchanged in 2.5.0):
| Tool + model | Diff | Phase blocks | Verified line |
|---|---|---|---|
| Mistral Vibe CLI + Mistral Medium | 6623000...HEAD |
7 | 100% (6/6), with 5 of 6 phase-4 boxes checked |
| Mistral Vibe CLI + GLM | 6623000...69e491a |
1 (phase 4 only) | 100% (6/6) |
| OpenCode + Qwen | 0bb8973...HEAD |
7 | 53% (16/34), while the file renders 21 of 36 boxes checked, and 16/34 is 47% |
- Mistral Medium and Qwen have word-for-word identical blocks for phases 1, 2, 3, 5, 6 and 7: both copied the previous
review.md. Mistral's phase 2 still says "ajoutés par ce diff" while its ownFiles checkedlists only phase-4 files. Copied evidence reads as fresh verification. - GLM dropped the six other phases.
- Qwen tagged the 15 criteria of out-of-diff phases
not-applicable, Mistral used the same tag for a partially verifiable phase-4 criterion: one tag, two meanings.
Scope
- Functional review reads only
plan.md, its phase files, and the current diff. It never reads a previousreview.md, and says so. - Every run renders one block per plan phase, re-derived from the plan.
- The phase projection decides scope. A phase whose projected files are untouched remains present, marked
out of scopein its heading. Every criterion stays unchecked, with no evidence and no status tag. Nothing is copied from an earlier review. - A phase in scope uses only current-diff evidence:
[x]requires afile:linecitation;[ ] fixmeans the current diff does not meet the criterion;[ ] not-applicablemeans the criterion cannot be assessed in this review by nature. - Remove
fixed. It represents historical state and contradicts a snapshot. not-applicableapplies only to a criterion of a phase in scope. It never marks an entire phase out of scope and adds no finding or blocker.Verifiedischecked / (all criteria in scope - not-applicable), rounded to the nearest whole percent, and must match the rendered boxes.- When no criterion remains after exclusions, render
— (0/0), never100%. This adds no finding or blocker by itself; the overall verdict still follows the shared rubric. - Update the
Testsection of02-review-functional.mdand add regression coverage for snapshot isolation, phase scope, status tags, and the verified calculation.
Acceptance criteria
- Reviewing phase 4 of a 7-phase plan renders 7 phase blocks, 6 of them marked
out of scopein their headings. - An out-of-scope block holds only unchecked criteria, with no status tag, gap, evidence,
[x], orfile:line. - Every
[x]in an in-scope phase cites evidence from the current diff. - Re-running the review on the same diff gives the same
PhasesandVerification, whatever the content of the previousreview.md. - With 6 criteria in scope, 5 checked and 1
not-applicable, the report shows100% (5/5). - With 34 countable criteria and 16 checked, the report shows
47% (16/34). - With no countable criterion, the report shows
— (0/0)and does not create a finding or blocker solely for that result. - The
fixedstatus is absent from02-review-functional.md,review-template.md, andreview-rubric.md. -
SKILL.md,02-review-functional.md,review-template.md, andreview-rubric.mdagree on snapshot inputs, phase scope, statuses, and counting. - Regression coverage proves an old contradictory
review.mdcannot affect the generatedPhasesorVerificationsections.
Prior art in this repository
SKILL.md:30: the snapshot rule, kept.02-review-functional.md:17: "the percent and count of checked criteria", to make precise.review-rubric.md:19: the only definition ofnot-applicable, to extend.
Out of scope
- How "the diff against the repository default branch" resolves (5 occurrences in
05-review, never defined). A clone whoseorigin/HEADpoints at a feature branch diffs against it. Separate issue. - Enforcing
review-validator.yml: a section outside the list survived in the three reviews above. Separate issue.
Reported on Discord: https://discord.com/channels/1173363373115723796/1547134531188822047
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Priority
High
Projects
- StatusShow more project fieldsDone
Activity
blafourcade commentedon Sep 14, 2026
The snapshot rule needs to be operational, not merely stated.
Functional review should read only
plan.md, its phase files, and the current diff. It must never read a previousreview.md. Every plan phase is rendered fresh on every run.Use the phase projection to decide scope. A phase whose projected files are untouched is marked
out of scope; every criterion stays unchecked, with no evidence and no status tag. A phase in scope uses only current-diff evidence:[x]requires afile:linecitation.[ ] fixmeans the current diff does not meet the criterion.[ ] not-applicablemeans an in-scope criterion cannot be assessed in this review by nature, and is excluded from the denominator.Remove
fixed. It implies historical state and contradicts a report that is explicitly not history. If no criterion remains in scope after exclusions, render— (0/0), never100%.Verifiedis therefore checked in-scope criteria divided by all in-scope criteria exceptnot-applicable; it must agree with the rendered boxes. This gives a partial review an honest result without copying prior evidence or treating an untouched phase as verified.fix(aidd-dev): a review keeps its rounds, its phase scope and a compu…
fix(aidd-dev): a review keeps its rounds, its phase scope and a compu…
fix(aidd-dev): a review appends a round and scores the whole plan
fix(aidd-dev): a review appends a round and scores the whole plan
fix(aidd-dev): a review appends a round and scores the whole plan
fix(aidd-dev): a review appends a round and scores the whole plan
fix(aidd-dev): a review appends a round and scores the whole plan
fix(aidd-dev): a review appends a round and scores the whole plan
blafourcade commentedon Oct 10, 2026
Fixed differently in #977; the
Problemabove stands, theScopeis superseded.review.mdgains one## Roundper run and is never rewritten.Score: 1/6 met, 1 unmet, 4 out of the diff, three numbers that sum to the plan.Out of the diff:line, not a block of boxes.not-applicableis[x], only for a property no diff could show. An absent subject is[ ].fixedis gone.fix(aidd-dev): a review appends a round and scores the whole plan (#977)