Skip to content

fix(aidd-dev): review leaves out-of-diff phases and the verified percent undefined #858

Closed

Description

@blafourcade

Problem

05-review does not say what happens to plan phases outside the reviewed diff, nor how the verified percent is computed. Reviewing one phase of a multi-phase plan, three models on the same skill produced three incompatible review.md files.

The skill contradicts itself:

  • SKILL.md:30 says review.md is overwritten each run: "a snapshot of the current diff, not a history".
  • 02-review-functional.md:23 says the Phases section holds "one block per plan phase".
  • Nothing says whether the previous review.md is read before writing.
  • review-template.md:33 renders {{pct}}% ({{n_checked}}/{{n_total}}), but nothing says which phases enter n_total. not-applicable is only defined as "neither a finding nor a blocker" (review-rubric.md:19), never for the percent, and never when to use it.

Observed on a 7-phase plan, phase 4 reviewed in three worktrees, aidd-dev 2.4.1 (05-review is unchanged in 2.5.0):

Tool + model Diff Phase blocks Verified line
Mistral Vibe CLI + Mistral Medium 6623000...HEAD 7 100% (6/6), with 5 of 6 phase-4 boxes checked
Mistral Vibe CLI + GLM 6623000...69e491a 1 (phase 4 only) 100% (6/6)
OpenCode + Qwen 0bb8973...HEAD 7 53% (16/34), while the file renders 21 of 36 boxes checked, and 16/34 is 47%
  • Mistral Medium and Qwen have word-for-word identical blocks for phases 1, 2, 3, 5, 6 and 7: both copied the previous review.md. Mistral's phase 2 still says "ajoutés par ce diff" while its own Files checked lists only phase-4 files. Copied evidence reads as fresh verification.
  • GLM dropped the six other phases.
  • Qwen tagged the 15 criteria of out-of-diff phases not-applicable, Mistral used the same tag for a partially verifiable phase-4 criterion: one tag, two meanings.

Scope

  • Functional review reads only plan.md, its phase files, and the current diff. It never reads a previous review.md, and says so.
  • Every run renders one block per plan phase, re-derived from the plan.
  • The phase projection decides scope. A phase whose projected files are untouched remains present, marked out of scope in its heading. Every criterion stays unchecked, with no evidence and no status tag. Nothing is copied from an earlier review.
  • A phase in scope uses only current-diff evidence: [x] requires a file:line citation; [ ] fix means the current diff does not meet the criterion; [ ] not-applicable means the criterion cannot be assessed in this review by nature.
  • Remove fixed. It represents historical state and contradicts a snapshot.
  • not-applicable applies only to a criterion of a phase in scope. It never marks an entire phase out of scope and adds no finding or blocker.
  • Verified is checked / (all criteria in scope - not-applicable), rounded to the nearest whole percent, and must match the rendered boxes.
  • When no criterion remains after exclusions, render — (0/0), never 100%. This adds no finding or blocker by itself; the overall verdict still follows the shared rubric.
  • Update the Test section of 02-review-functional.md and add regression coverage for snapshot isolation, phase scope, status tags, and the verified calculation.

Acceptance criteria

  • Reviewing phase 4 of a 7-phase plan renders 7 phase blocks, 6 of them marked out of scope in their headings.
  • An out-of-scope block holds only unchecked criteria, with no status tag, gap, evidence, [x], or file:line.
  • Every [x] in an in-scope phase cites evidence from the current diff.
  • Re-running the review on the same diff gives the same Phases and Verification, whatever the content of the previous review.md.
  • With 6 criteria in scope, 5 checked and 1 not-applicable, the report shows 100% (5/5).
  • With 34 countable criteria and 16 checked, the report shows 47% (16/34).
  • With no countable criterion, the report shows — (0/0) and does not create a finding or blocker solely for that result.
  • The fixed status is absent from 02-review-functional.md, review-template.md, and review-rubric.md.
  • SKILL.md, 02-review-functional.md, review-template.md, and review-rubric.md agree on snapshot inputs, phase scope, statuses, and counting.
  • Regression coverage proves an old contradictory review.md cannot affect the generated Phases or Verification sections.

Prior art in this repository

Out of scope

  • How "the diff against the repository default branch" resolves (5 occurrences in 05-review, never defined). A clone whose origin/HEAD points at a feature branch diffs against it. Separate issue.
  • Enforcing review-validator.yml: a section outside the list survived in the three reviews above. Separate issue.

Reported on Discord: https://discord.com/channels/1173363373115723796/1547134531188822047

Activity

  1. added theissue type on Sep 11, 2026
  2. moved this from Ideation to Todo in AIDD Roadmapon Sep 11, 2026
  3. blafourcade commented on Sep 14, 2026

    @blafourcade
    ContributorAuthor

    The snapshot rule needs to be operational, not merely stated.

    Functional review should read only plan.md, its phase files, and the current diff. It must never read a previous review.md. Every plan phase is rendered fresh on every run.

    Use the phase projection to decide scope. A phase whose projected files are untouched is marked out of scope; every criterion stays unchecked, with no evidence and no status tag. A phase in scope uses only current-diff evidence:

    • [x] requires a file:line citation.
    • [ ] fix means the current diff does not meet the criterion.
    • [ ] not-applicable means an in-scope criterion cannot be assessed in this review by nature, and is excluded from the denominator.

    Remove fixed. It implies historical state and contradicts a report that is explicitly not history. If no criterion remains in scope after exclusions, render — (0/0), never 100%.

    Verified is therefore checked in-scope criteria divided by all in-scope criteria except not-applicable; it must agree with the rendered boxes. This gives a partial review an honest result without copying prior evidence or treating an untouched phase as verified.

  4. blafourcade commented on Oct 10, 2026

    @blafourcade
    ContributorAuthor

    Fixed differently in #977; the Problem above stands, the Scope is superseded.

    • review.md gains one ## Round per run and is never rewritten.
    • No percent: Score: 1/6 met, 1 unmet, 4 out of the diff, three numbers that sum to the plan.
    • A phase the diff leaves untouched is one Out of the diff: line, not a block of boxes.
    • not-applicable is [x], only for a property no diff could show. An absent subject is [ ].
    • fixed is gone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    High

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions