Skip to content

fix(aidd-dev): a review never checks a runtime criterion from the code - #995

Merged
blafourcade merged 1 commit into
nextfrom
fix/review-runtime-evidence
Oct 11, 2026
Merged

blafourcade merged 1 commit into
nextfrom
fix/review-runtime-evidence

Conversation

@blafourcade

Copy link
Copy Markdown
Contributor

🎯 What & why

The review checked criteria that only running the app could prove (a banner showing, a timing) from the code alone. On next, "submitting the form displays a success banner" came out [x] in 9 runs of 9, including one where a recorded run had seen it fail.

🛠️ How it works

The review stays static (SKILL.md:36). It now says what the code proves and names what still needs a run:

  • A criterion whose outcome only a run could show is unverified at runtime. It is never checked from the code, and it is named in the round together with what a run would show (03-review-functional.md:21).
  • A run recorded beside the plan on the reviewed head decides it (03-review-functional.md:22):
    • passing: checked, citing the record;
    • failing: unchecked;
    • recorded on another head: still unverified. A branch name, a working-tree edit made after the run, or a blocked run does not prove the reviewed code.
  • Unverified criteria are in scope but never block (review-rubric.md:24).
  • The score has four numbers: met + unmet + unverified + out of the diff = the plan's total.

This supersedes #977's "a not-applicable criterion counts as met": such a criterion is now unverified, and not-applicable is gone from the skill.

Why the review does not run the app: running belongs to acceptance QA. The review runs every round, and a QA run done before review fixes goes stale. The unverified list is therefore the hand-off to a QA run on the final head.

🧪 How to verify

  • node scripts/check-tests-leave-git-alone.js -- node --test scripts/__tests__/a-review-appends-a-round-and-scores-the-whole-plan.test.js: 32 pass. Each new guard goes red under its own mutation.
  • 103 real runs (claude -p --plugin-dir plugins/aidd-dev) on fixtures graded against a known truth.
    • 85 functional runs, all scored as expected. They include six traps:
      • a record naming a branch;
      • a working-tree edit made after the run;
      • a stale record covering a criterion the code proves (that criterion stays checked, citing the code);
      • a blocked run;
      • a criterion mixing logic and display;
      • a backend 409 criterion, which stays checked.
    • 18 all-axes runs: verdicts unchanged, and no unverified line on criteria the code proves.

⚠️ Heads-up

  • A round where every criterion is unverified approves with 0/N met, and its score says so. This is deliberate: unverified never blocks.
  • Nothing in the SDLC orchestrator runs QA yet. Check reads $validation_reports, but no zone produces them, so unverified criteria stay unverified until the PR. That gap is for a follow-up issue.
  • refactor(framework): replace skill test sections with scoped rules #985 removes the ## Test tables. The guards here assert ## Process, the template and the rubric, so they survive it.

🔗 Linked issue

Refs #976

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z

On `next`, a criterion only a run could show was checked from the code alone:
"submitting the form displays a success banner" came out `[x]` citing
`src/signup.js:3` in 9 runs of 9, including one where a recorded run had seen
that banner fail.

A review stays static. A criterion whose subject is in the diff but whose outcome
only a run could show, what a screen displays, a timing, an outside system's
answer, is now unverified at runtime: named in the round with what a run would
show, never checked from the code, and never blocking the verdict. A run recorded
beside the plan on the reviewed head decides it instead: passing checks it,
citing the record; failing leaves it unchecked; a record from another head
leaves it unverified. The score gains a fourth number, so met, unmet, unverified
and out of the diff sum to the plan's whole criteria total.

This replaces #977's "a not-applicable criterion counts as met": such a
criterion is now unverified, and `not-applicable` is gone from the skill.

Measured with `claude -p` against the plugin from this tree, 103 runs:
- 85 functional runs: every fixture scored as expected, including six traps:
  a record naming a branch, a working-tree edit after the run, a stale record
  covering a criterion the code proves, a blocked run, a mixed criterion, and a
  backend criterion that must stay checked.
- 18 all-axes runs: verdicts unchanged (approve, blocked, then
  changes-requested), no unverified line on criteria the code proves.

A round whose every criterion is unverified approves with `0/N met`; its score
says so.

Refs #976

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant