Skip to content

M5 Phase 3: test_api onto the fixture pack, and a docs pass that is not optional - #62

Merged
kjpatel merged 1 commit into
mainfrom
m5-phase3-fixtures-demo-suites
Aug 6, 2026
Merged

M5 Phase 3: test_api onto the fixture pack, and a docs pass that is not optional#62
kjpatel merged 1 commit into
mainfrom
m5-phase3-fixtures-demo-suites

Conversation

@kjpatel

@kjpatel kjpatel commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Part 4b(ii)b. test_api's failures under deletion go 10 → 4, and three of the four remaining are genuine example tests — their subject is a teaching pack's own wording, so they move with rulepacks/ rather than convert.

_pack() now defaults to the fixture pack and takes a name only for those three.

The find

Writing the money phrasing surfaced a mis-authored guard in the pack I had added one PR earlier:

when: { value: "0.00" }     # matches nothing on a money decision
when: { amount: positive }  # the correct form

A money decision's value is an amount and a currency, not a scalar. The demo did not complain — it fell through to the unguarded case and rendered a plausible answer from the wrong branch.

The fixture keeps the corrected version with a comment naming the mistake, because a pack author will make it and the failure mode reads like success.

Confirmation the edit was inert where it must be: fixtures/receipts is byte-unchanged. Phrasing is presentation and stays out of every hashed body.

One test still needs fixture growth

test_every_placeholder_the_validator_accepts_is_one_the_demo_renders exercises {daysBetween:…} and {fact:…|day}, so the pack needs a date attribute. Cheap, but it is another rebuild — it belongs with part 5's other growth rather than alone.

And the standing policy you asked for

CLAUDE.md's definition of done now requires a documentation pass as the last step of every PR, with a five-question checklist and the reason the ordering matters:

Documentation written before the work is finished describes what you intended; the interesting part is almost always what the work taught, and that is only knowable at the end.

It cites the two cases where that pass caught a claim the code had just falsified — the architecture guide still saying "usually supersedes" after C6 made that unrepresentable, and fixtures/README.md never mentioning a case the corpus had grown two PRs earlier. No need to ask for it again.

Verification

975 passed, 7 skipped
verified 351 cases                        # byte-for-byte
git diff -- golden/ fixtures/receipts     # empty

Deletion re-measured in a detached worktree, which is now the standing method for this phase.

🤖 Generated with Claude Code

…ot optional

Part 4b(ii)b. test_api's failures under deletion go from 10 to 4, and three of
the four remaining are genuine example tests — their subject is a teaching
pack's own wording, so they move with rulepacks/ rather than convert.

_pack() now defaults to the fixture pack and takes a name only for those three.

Writing the money phrasing found a mis-authored guard in the pack I had added
one PR earlier: `when: { value: "0.00" }` matches nothing on a money decision,
whose value is an amount and a currency rather than a scalar. The correct form
is `amount: positive`. The demo did not complain — it fell through to the
unguarded case and rendered a plausible answer from the wrong branch. The
fixture keeps the corrected version with a comment naming the mistake, because
a pack author will make it and the failure mode reads like success.

Confirmation the phrasing edit was inert where it must be: fixtures/receipts is
byte-unchanged. Phrasing is presentation and stays out of every hashed body.

One remaining toolkit test needs fixture growth —
test_every_placeholder_the_validator_accepts_is_one_the_demo_renders exercises
{daysBetween:…} and {fact:…|day}, so the pack needs a date attribute. Cheap,
but another rebuild; it belongs with part 5's other growth rather than alone.

Also, at Kushan's request and as standing policy: CLAUDE.md's definition of
done now *requires* a documentation pass as the last step of every PR, with
the five-question checklist and the reason the ordering matters — docs written
before the work is finished describe what you intended, and the interesting
part is what the work taught, which is only knowable at the end. Two of this
repo's better paragraphs exist because that pass came last and caught a claim
the code had just falsified.

Verification: 975 passed, 7 skipped; verify 351 byte-for-byte; golden/ and
fixtures/receipts untouched; deletion re-measured in a detached worktree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kjpatel
kjpatel merged commit fb436a7 into main Aug 6, 2026
6 checks passed
@kjpatel
kjpatel deleted the m5-phase3-fixtures-demo-suites branch August 6, 2026 14:48
kjpatel added a commit that referenced this pull request Aug 10, 2026
test_rules_api goes from 29 failures under deletion to 0; the demo
directory falls from 57 to 28. 969 passed, 7 skipped; 351 golden receipts
replay byte-for-byte; impact reports 0 of 351 decisions flip.

Two of the 29 were not failures. test_declared_cases_run_green looped
over an empty glob so its body never ran, and the refusal test was
parametrized over one, so pytest collected zero cases for it. Both passed
under deletion. Both are named after claims nothing was checking. That is
the founding rule of this phase in its sharpest form yet, and the two
shapes are now named in CLAUDE.md: assert a glob is non-empty before
using it, and loop inside the test rather than parametrizing over the
filesystem.

Three artifacts the fixtures lacked, each added because a test needed it:
expected.yaml (declared outcomes — the studio runs them beside impact, so
a corpus without them left half the surface untestable), fixtures/dmn/
(one table that compiles, one refused), and a nested
abstentionPolicy.attributes, since two scalars cannot catch a re-emitter
that flattens. Pack version 2026.2.0 -> 2026.3.0; receipts, digest
vectors and the corpus aggregate re-pinned.

Also fx-0006, a restricted widget scoring 60. Every existing case scored
12 or 80 against a threshold of 50, so a threshold edit flipped all three
restricted cases or none — impact analysis over these fixtures could only
ever answer "everything moved", which cannot demonstrate what impact
analysis is for. It clears a new bar for this corpus: not a behaviour
that was unreachable, but a distinction the corpus could not draw.

reload_demo() hoisted into demotest_helpers after being copied into a
second suite, one copy short of the version that gets edited wrong. The
content-root assembly now performs three path rewrites, each of which
fails silently if skipped by offering an empty list rather than raising.

Found on the way, and worth more than the conversion: editing the derived
threshold moves all three decisions while two of the three rules stay
byte-identical on the page. The blast radius of an edit is not a
syntactic property of the edit, which is the argument for the equivalence
panel and is stronger than the one the docs were making. Sharpened into
docs/neuro-symbolic-architecture.md.

Three of the plan's claims about this task were wrong and are corrected:
the suite is 29 failures not 31, test_api has 6 residual not 4 (verified
against #62, so not a c-0 regression), and part 5's "needs a date
attribute" is false — placeholder validation is syntactic. The progress
log had also lost the PR links for #61-63 and never recorded #64.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant