methodology: apply the six owner decisions (D1-D6) - #22
Merged
Conversation
"After the scope lock, accepted plan documents are immutable" contradicted every stage that writes plan.md, status.json and manifest.md, and the SUPERSEDED banner it asked for was itself an edit. Stage 3 change control now names the set: changes/CR-*.md and snapshots/** are never edited once written (what .github/protected-artifacts.sh refuses in this repository); intent.md after acceptance and spec.md after approval are superseded through a Change Record plus one banner line; plan.md collects departures during Build; status.json, manifest.md and the audit baseline are bookkeeping. Stage 2's scope-lock wording, Stage 6's "nothing is rewritten" paragraph, the workspace and workflow pages, and METHODOLOGY say the same. Decision D5 of docs/plans/2026-09-04-methodology-conformance/findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
Stage 5 forbade running any release step, then marked Deploy complete and printed "Released" the moment a named human authorized. The record claimed something nobody had observed. Step D now leaves the stage in status "authorized" with authorized_by/authorized_at; a new Step E records released_by/released_at when a human confirms the release happened, marks the stage complete, and starts Maintain at released_at. The plan skill's resume path asks whether an authorized release happened instead of assuming; plan-status reports plan-to-authorization always and plan-to-release only when released_at exists, and shows an authorized but unconfirmed release as exactly that. New fields are optional, so existing status files need no migration. Decision D4 of docs/plans/2026-09-04-methodology-conformance/findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
…ck-audit (D1)
quick-plan.md said "This uses the same gate as /toque:plan Stage 2" and
spawned the auditor on its own terms: no canary, no evidence validation,
no gate expression. quick-audit had no gate at all. The command's own
rationale was right and its body did not follow it: a lighter copy of a
gate is the route around it, and the route around is the one that gets
used.
Stage 2 Part C is now a delimited <design_gate> block with three bindings
set by the caller: {doc} (the document under audit), {gate_dir} (the folder
that receives audit.md, evidence/ and .canary/) and {generator} (who
revises on NOT PASS, or none). Stage 2 binds them to the plan folder and
the spec writer. quick-plan binds docs/specs/{name}.md, docs/specs/{name}/
and the plan-scaffolder; quick-audit binds the audited file, a gate folder
beside it (or the plan folder for a plan's own document) and no generator,
so it reports NOT PASS with the unmet criteria and stops. Both commands
read the stage file and execute the block; neither restates it.
The auditor loses its conversation-only mode: every caller writes
evidence/{criterion_id}.json and audit.md into the gate folder, because
every caller now runs the validator over them. A standalone gate folder is
committed with the document it audits; .canary/ under docs/specs/*/ joins
the ignore list.
PH5-070 in tests/layer1-repo.sh guards the shape: the block exists and
carries the canary, the validator, the gate expression and the feedback
form; each shortcut names the stage file and the block; neither carries a
second copy of the gate expression or its own canary orchestration.
Negative control run: a copied "PASS = CANARY_OK" line in quick-plan.md
fails the guard. Full suite: 8 of 8 layers pass.
Decision D1 of docs/plans/2026-09-04-methodology-conformance/findings.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
The TDD entry in the testing-methodology guide let the implementation agent write its own tests first, against LINT-18 in the registry and the separate-authorship rule in Stages 2 and 3. It now assigns test generation to a separate agent or a human and keeps the red-green ordering: the test writer generates the suite from the spec, the implementer makes it pass. METHODOLOGY records the guide, the registry and the stages as agreeing. Decision D2 of docs/plans/2026-09-04-methodology-conformance/findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
The BRD template's deep scan said to "update confidence in the baseline" two lines above the rule that baselines are inputs only and are never written back. Writing into another tool's output destroys the ability to tell what that tool reported from what Toque verified. The deep scan now records what it checked, and the confidence it holds, in the BRD's own Feature Coverage table; the baseline file is never edited. Decision D3 of docs/plans/2026-09-04-methodology-conformance/findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
Step 0.2 offered "Apply the same fix? [Y/n/investigate]" on a HIGH match before Phase 1, against the skill's own Iron Law: no permanent fix without root-cause investigation first. A HIGH match now names the earlier cause as the first hypothesis for Phase 1, and the earlier fix is not re-applied until Phase 1 shows the same cause is present in this codebase now. A confirmed match is logged as a recurrence, since it means the earlier fix was insufficient. Containment for SEV1/SEV2 is unchanged. Decision D6 of docs/plans/2026-09-04-methodology-conformance/findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
…ntry decisions.md is additive beside findings.md: what was chosen for D1-D6, why, and where each change landed, plus what the changes do not claim. The manifest points at it. CHANGELOG gains an Unreleased section with one entry per decision; the 11.0.1 heading stays first for the version cross-check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
A six-dimension review of the seven decision commits (consistency,
executability, guards, documentation, records, regression; one verifier per
finding) confirmed 34 findings. Four mattered and are fixed here; the rest
were stale prose and are swept.
The shared <design_gate> block:
- had no branch for a document the canary cannot mark. tq-canary.js inject
exits 2 when none of the five template shapes exist, which would have
stopped quick-audit on any plan written outside the spec template. The
block now audits such a document and reports NOT PASS with the reason.
- let evidence records cite the mutated .canary/ copy, a file nobody
commits. Records are re-anchored to the committed document (path, lines,
sha256) before validation; a quote that only exists in the mutation is
dropped and its criterion re-checked. Pre-existing in Stage 2; D1 put it
on every caller's path.
- ran EVIDENCE REINFORCEMENT for every caller, editing the audited document
in place: against D5 for an approved spec, and against quick-audit's
promise not to touch someone else's document. It now runs only with a
generator bound and an unapproved document; otherwise the notes go in
audit.md.
- kept a standalone run's baseline and gate result in audit.md, which the
fresh auditor rewrites every iteration. They live in gate.json now.
- pointed every caller at Stage 2's human review gate on PASS; the canary
re-run had no seed and would have picked the same class; .canary/ was
never deleted and only ignored under two paths. All three fixed.
The auditor chose Full mode from the existence of a plan folder, so
quick-plan --plan would have audited the plan's spec instead of the new
one; mode follows the caller's bindings. quick-audit binds the plan folder
only for the plan's own spec.md and writes a rerun after design completion
to reaudits/{date}/ rather than overwriting the Stage 2 record.
D4: Step E sets current_phase to maintain; the generic bookkeeping rule
excepts Stage 5; plan-status labels a pre-existing complete-without-
released_at as recorded at authorization; metric sentences name
plan-to-authorization. D6: the recurrence count increments only when Phase 1
confirmed the same cause.
Guards: PH5-070 is renamed PH5-042, the id docs/specs/phase5-verifier-gate.md
reserved for "Same gate in quick-audit.md and quick-plan.md", and now
requires the execute-by-reference directive (CR-stripped, line-joined, with a
self-test), refuses a canary invocation but not a prose mention, refuses a
shortcut that spawns the auditor itself, and checks the auditor's
no-conversation-only rule and the .gitignore pattern. REL-1 guards D4, KB-1
guards D6. Layer 4's B4 can fail again when a shortcut stops naming its
agent. Negative controls run in a scratch copy: a shortcut that only names
the block, one that invokes the canary, Stage 5 reverting to complete at
authorization, and the "Apply the same fix?" prompt each fail their guard;
a prose mention of tq-canary.js passes.
Records: the CHANGELOG Unreleased section is removed, because this
repository writes the entry at release time and tests/mutation refuses a
tree that carries that heading; the draft is
docs/plans/2026-09-04-methodology-conformance/changelog-draft.md.
decisions.md records the audit follow-up. Stale sentences swept in
METHODOLOGY, interop, GUIDE (table, diagram, prose), help, install, both
READMEs, the documentation inventory, and technique notes 02 and 06.
Full suite: 8 of 8 layers pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
Six executors followed the shipped command text in isolated scratch
repositories (standalone quick-plan, quick-plan --plan, quick-audit on a
template spec, on a prose ADR, on a plan's approved spec, on pasted text),
each checked by a deterministic on-disk script and a verifier. Every
on-disk invariant held. The instructions did not all survive contact:
The plugin's own templates/spec.md carried none of the five shapes the
canary attaches to (rollback and go/no-go were bullets, no owned dependency
row, no assumption register), so Stage 2 would have refused its own specs
with "no canary class could be applied"; the assumption-inject class also
planted its row in the first numbered table, usually the Risk Assessment,
which is how the defect stayed hidden. The template and the scaffolder's
template now carry every shape, Stage 2's Delivery instructions name them,
the class anchors on the assumption register, and tests/canary-test.js
fails if the template drops a shape (49 cases pass).
The shared <design_gate> block gains: the exit-2 branch says which later
steps to skip; a no-isolation rule when no fresh auditor can be spawned
(CANARY_OK false, NOT PASS, never an in-context audit presented as a gate);
`detected` runs on the auditor's raw UNMET list, then records and audit.md
line references are re-anchored to the committed document; records are
re-pinned after evidence reinforcement, which had made every record
EVIDENCE-STALE; LINT-14 is the caller's verdict from the baseline
comparison and is N_A whenever no baseline exists; lint results admit N_A
with a reason; the gap total names its terms; gate.json has a fixed
top-level shape; the registry and lint-candidates paths are
${CLAUDE_PLUGIN_ROOT}-relative; the holistic pass is skipped and recorded
when no fresh judge exists; assumption verification and reaudits/{date}/
have a standalone branch; CANARY_OK is defined for the not-planted case.
Auditor and registry: LINT-08 covers falsified HIGH-impact assumptions;
LINT-18 is UNMET when authorship is unspecified; the lint table admits N_A;
Criterion Verdicts points at evidence/ instead of inlining; the specialist
reviewers' docs/audit inputs are optional; the tier mapping follows the
self-audit skill (a plan quote is Tier B, Tier C is never HIGH); the
reviewer count reads five.
The smaller items from the conformance audit close with this: the plan
skill's parallel rule now says three or more, matching its scaling table;
the baseline-regression note no longer claims the gate tolerates
pre-existing gaps; tests/run-all.sh accepts layer 8 as a single-layer
argument and rejects anything else.
Records: changelog-draft.md and decisions.md carry the stress-test
follow-up. Full suite: 8 of 8 layers pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
The suite proves the gate has one definition and that both shortcuts carry the execute-by-reference directive (PH5-042). It cannot prove a live run obeyed it. These are the runs that did: six scenarios covering every binding quick-plan and quick-audit can produce, executed end to end by agents following the shipped text in scratch repositories, each checked by a deterministic script and by a verifier that had not seen the executor's reasoning. Run 1 found fourteen instruction defects and had one flaw of its own: the plugin changed underneath it mid-flight, so parts of its result were void. Run 2 closed that. The plugin was copied read-only, chmod a-w on all 59 instruction files and scripts, hashed before locking and again after; the lock was proved rather than assumed by attempting a shell append and a Node write against the gate stage file and confirming both were refused. Opening and closing digests are identical, so run 2's findings stand on text that could not move. Run 2 reached the intended PASS path end to end for the first time: a fresh auditor spawned via claude -p, a planted defect found and mechanically confirmed, records validated, gate open, every term re-derived from the gate folder by a verifier that did not perform the run. stress-rig/ holds the reusable half: the fixture builder (parameterised, tar rather than rsync, which is not installed in Git Bash here), the deterministic invariant checker, and the template-shaped spec fixture that carries all five canary shapes. The agent half was composed per run and is described rather than shipped. What is worth repeating is the four-layer separation: executor, deterministic script, independent verifier, cross-scenario critic. The record is deliberate about what the runs do not establish, including the one uncertainty run 2 could not close: genuine auditor isolation is proved structurally in two scenarios and only circumstantial in the other four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
…th them Two owner decisions, then the defects six live runs of quick-plan and quick-audit exposed in the shipped text. Method and results in stress-test.md (edba871). D7 — a planned deliverable is not an infrastructure gap. The gate checked whether every named test and monitoring file existed at audit time, while the spec template tells the author to name the file each phase will create. An honest pre-build spec therefore failed its own coverage check: one scenario took 9 INFRA-GAPs on the objective another passed only by citing a pre-existing test. Infrastructure verification now classifies each path PLANNED or CLAIMED and judges only the claims; a path used by a phase that runs before the phase creating it stays CLAIMED and still fails. LINT-15/16 follow the same scope. D8 — a fresh instance is the Agent tool or claude -p from Bash. The canary measures whether an independent auditor finds a planted defect, and nothing said what independent meant; five of six runs concluded it was impossible while the sixth achieved it with a subprocess. Isolation is now decided at STEP 0 before anything is injected, and with neither route available nothing is planted, the output is labelled findings rather than a gate, and the result is NOT PASS. The canary's retry was vacuous. The gate said to re-run a missed canary with the literal seed `retry`, "because the default seed is derived from the file and would pick the same class again". pickClass hashes the SEED ALONE, so `retry` is a constant: about one default seed in five re-planted the class just missed, and the rotation on a sparse document could return to it regardless. Two misses of one trial then failed the gate as "audit untrustworthy" and blocked revision. tq-canary.js now takes --exclude <class>, which drops the class already tried from the rotation and records the exclusion, with exit 3 distinguishing "only one trial was ever possible" from exit 2 "no class applies at all". LINT-20 was stated two contradictory ways inside lint-registry.md, whose own line 6 declares it the only place rule text may be written: "all 3 sections" in the rules table against "none of the 3" in Gate Behavior. Two auditors split on a two-subsection document and the run's only PASS sat on the difference. No shipped template emits three subsections, so the count is gone and the entry fields are what LINT-20 checks; the METHODOLOGY restatement follows verbatim. A spawned auditor that starts and never returns now has a branch of its own. The reason vocabulary was explicitly closed over four values, none of which fitted, and five of six runs hit the case. The gate relaunches once without consuming the sanctioned canary re-run, then records auditor-did-not-return. Canary re-anchoring no longer demotes a citation its own edit split. A quote broken by the injected line is genuine but unlocatable, so it was dropped as fabricated, turning correct MET verdicts into UNMET; the carve-out evidence reinforcement already had now covers the canary step. Also from run 1: the LINT-14 caller record has a write order that survives the validator; the previous baseline is identified explicitly and an unchanged document records auditor variance rather than regressions; a vacuous rule is PASS rather than N_A; rerun semantics are defined for same-day reaudits and standalone gate folders; --plan no longer names the spec after the plan; falsified assumptions count toward the gap total; quick-plan prints a NOT PASS sentence on NOT PASS; the holistic pass writes nothing into the audited repository; pasted text never overwrites a different existing file; two bare registry paths in plan-auditor.md are plugin-relative; audit.md holds verdicts pointing at evidence/; the four command files list Agent alongside Task. An external review (Codex CLI) of the --exclude fix found three edge cases in it, all closed here: exit 3 fired on a document no class could mark, claiming one trial was possible when none was; a repeated --exclude left the second flag to be parsed as the seed and could select the very class the caller excluded; and the "different class" assertion passed against the reverted CLI, which parsed --exclude as a seed. The same review showed the guard's bare --exclude grep survived replacing the retry command with a reseed. Guarded by PH5-042/PH5-043 and 63 canary assertions (up from 49). Both were checked for vacuity: reverting the script makes the test report the same class twice, and rewording the block makes the guard fail. Suite 8/8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The methodology conformance audit (PR #21) left six shipped-instruction conflicts as owner decisions D1 to D6. This branch applies the decisions, one commit each, chosen for the option that stays honest as the plugin ages rather than the cheapest edit. Rationale and touchpoints per decision:
docs/plans/2026-09-04-methodology-conformance/decisions.md.The six commits
c37f73a— the immutable set is enumerated.changes/CR-*.mdandsnapshots/**are never edited once written (what CI refuses here); accepted documents are superseded through a Change Record and one banner line;plan.md,status.json,manifest.mdare living state updated in a named way.a5fa147— authorization and release are two events. Stage 5 leaves Deployauthorizedwithauthorized_by/authorized_at; a later human confirmation writesreleased_by/released_at, marks the stage complete, and starts Maintain. Resume asks;plan-statusreports both metrics and never rounds authorized up to released. New fields are optional; no migration.74b7c37— one design gate. Stage 2 Part C is a delimited<design_gate>block with bindings{doc},{gate_dir},{generator};quick-planandquick-auditexecute that block by reference instead of spawning the auditor on their own terms. Standalone documents get a gate folder beside them (docs/specs/{name}/); the auditor loses its conversation-only mode. Guarded by PH5-070 intests/layer1-repo.sh.4699913— a separate agent writes the tests, TDD included. The guide's TDD entry now matches LINT-18 and the stage files; red-green ordering kept, author changed.8bae489— audit baselines stay read-only. The BRD deep scan records verified confidence in the BRD's own coverage table, never in the baseline file.99eb889— a knowledge-base match is a lead, not a fix. A HIGH match names the earlier cause as Phase 1's first hypothesis; the fix is not re-applied until the cause is confirmed; a confirmed match is logged as a recurrence.9ba8e5e— records.decisions.mdbesidefindings.md, manifest pointer, CHANGELOGUnreleasedsection.Verification
bash tests/run-all.shon the final tree: 8 of 8 layers pass, including the new PH5-070 guard.PASS = CANARY_OK AND EVIDENCE_OKtoquick-plan.mdmakes PH5-070 fail; the line was removed afterwards..github/protected-artifacts.sh ciemulated as a pull request overmain...HEAD: exit 0.quick-planorquick-auditthrough the shared block, or a live Stage 5 release confirmation. These are instruction changes; the suite checks their shape, not agent compliance.Release note
Behavior of two commands and one stage changed (D1, D4). At the next release this warrants a minor version, not a patch. No version bump in this PR.
Audit follow-up (
15ceff4)A six-dimension review of the seven commits (44 agents, one verifier per finding) confirmed 34 findings; four mattered, all fixed in this commit:
tq-canary.js injectexits 2 without the five template shapes). It now audits such a document and reportsNOT PASSwith the reason..canary/copy; they are re-anchored to the committed document before validation. Pre-existing in Stage 2, put on every path by D1.quick-plan --planwould audit the wrong spec; mode follows the caller's bindings, andquick-auditwrites a post-approval rerun toreaudits/{date}/instead of overwriting the Stage 2 record.Guards: PH5-070 renamed to PH5-042 (the id the verifier-gate spec reserved for this) and tightened to require the execute-by-reference directive; REL-1 (D4) and KB-1 (D6) added; layer 4's B4 can fail again. Five negative controls run in a scratch copy: a shortcut that only names the block, one that invokes the canary, Stage 5 reverting to complete-at-authorization, and the "Apply the same fix?" prompt each fail; a prose mention of the canary tool passes. The CHANGELOG
Unreleasedsection is removed (the mutation harness refuses that heading; the entry is written at release time) and the draft moved tochangelog-draft.md. Stale prose swept across METHODOLOGY, interop, GUIDE, help, install, both READMEs, the inventory, and technique notes 02 and 06. Full suite: 8 of 8.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fx5CCdiv5HdUrakHhCfMAq