Repository navigation
fix(aidd-dev): a review never checks a runtime criterion from the code - #995
Merged
Merged
Conversation
On `next`, a criterion only a run could show was checked from the code alone: "submitting the form displays a success banner" came out `[x]` citing `src/signup.js:3` in 9 runs of 9, including one where a recorded run had seen that banner fail. A review stays static. A criterion whose subject is in the diff but whose outcome only a run could show, what a screen displays, a timing, an outside system's answer, is now unverified at runtime: named in the round with what a run would show, never checked from the code, and never blocking the verdict. A run recorded beside the plan on the reviewed head decides it instead: passing checks it, citing the record; failing leaves it unchecked; a record from another head leaves it unverified. The score gains a fourth number, so met, unmet, unverified and out of the diff sum to the plan's whole criteria total. This replaces #977's "a not-applicable criterion counts as met": such a criterion is now unverified, and `not-applicable` is gone from the skill. Measured with `claude -p` against the plugin from this tree, 103 runs: - 85 functional runs: every fixture scored as expected, including six traps: a record naming a branch, a working-tree edit after the run, a stale record covering a criterion the code proves, a blocked run, a mixed criterion, and a backend criterion that must stay checked. - 18 all-axes runs: verdicts unchanged (approve, blocked, then changes-requested), no unverified line on criteria the code proves. A round whose every criterion is unverified approves with `0/N met`; its score says so. Refs #976 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
blafourcade
marked this pull request as ready for review
October 11, 2026 18:33
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What & why
The review checked criteria that only running the app could prove (a banner showing, a timing) from the code alone. On
next, "submitting the form displays a success banner" came out[x]in 9 runs of 9, including one where a recorded run had seen it fail.🛠️ How it works
The review stays static (SKILL.md:36). It now says what the code proves and names what still needs a run:
This supersedes #977's "a
not-applicablecriterion counts as met": such a criterion is now unverified, andnot-applicableis gone from the skill.Why the review does not run the app: running belongs to acceptance QA. The review runs every round, and a QA run done before review fixes goes stale. The unverified list is therefore the hand-off to a QA run on the final head.
🧪 How to verify
node scripts/check-tests-leave-git-alone.js -- node --test scripts/__tests__/a-review-appends-a-round-and-scores-the-whole-plan.test.js: 32 pass. Each new guard goes red under its own mutation.claude -p --plugin-dir plugins/aidd-dev) on fixtures graded against a known truth.409criterion, which stays checked.0/N met, and its score says so. This is deliberate: unverified never blocks.$validation_reports, but no zone produces them, so unverified criteria stay unverified until the PR. That gap is for a follow-up issue.## Testtables. The guards here assert## Process, the template and the rubric, so they survive it.🔗 Linked issue
Refs #976
✅ I certify
🤖 Generated with Claude Code
https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z