M5 Phase 3: the rule studio onto the fixture corpus

CHANGELOG.md

@@ -82,6 +82,12 @@ Also: `impact.analyze` gained `pack_overrides`, so a candidate pack that exists
 
 *A phrasing guard that matches nothing fails quietly, which is worth a fixture.* Writing the fixture pack's money phrasing, the first guard was `when: { value: "0.00" }` — and a money decision's value is an amount and a currency, not a scalar, so it matched nothing. The demo did not complain: it fell through to the unguarded case and rendered a plausible answer. The correct form is `amount: positive`. The fixture keeps the corrected version with a comment naming the mistake, because a pack author will make it and the failure mode is a verdict that reads fine and came from the wrong branch.
 
+*The two loudest-looking tests were the quietest, and neither showed up as a failure.* Converting the rule studio's suite, the deletion measurement said 29 failures. It was 31. `test_declared_cases_run_green_against_every_committed_pack` looped over a glob of packs, and with the packs deleted the loop body never ran; `test_every_refusal_example_is_reported_not_raised` was `@pytest.mark.parametrize`d over a directory, and parametrize is evaluated at *collection*, so an empty directory produced zero test cases. pytest reports the count that remains and says nothing about what left. Both tests passed. Both were named after claims nothing was checking. The fix in each is one line — assert the glob is non-empty before using it, and loop inside the test rather than parametrizing over the filesystem — and the general form is now in CLAUDE.md, because "asserts over an empty glob" was already written down and these two are the shapes it takes in practice.
+
+*The blast radius of an edit is not a syntactic property of the edit.* The studio shows a diff, a solver-backed equivalence panel and corpus impact side by side, and it is easy to read that as belt-and-braces for careless authors. The fixture pack now makes the real case reproducible: change one derived value from 50 to 70, and all three of the pack's decisions change what they conclude while two of the three rules stay byte-identical on the page — they read the moved value as a `derived` binding, and nothing in their own text, guards, or grid row moves at all. The equivalence panel returns a witness naming a widget scoring exactly 50, the boundary where the two packs part. The diff is complete and honest and it is still not where the effect lives. That is the argument for the panel, and it is stronger than the one the docs had been making.
+
+*A fixture corpus can be too agreeable to be useful.* Every case in `fixtures/` scored either 12 or 80 against a threshold of 50, so any edit to that threshold flipped all three restricted cases at once or none — which meant impact analysis over these fixtures could only ever answer "everything moved", and could not demonstrate the thing it exists for: a pack whose *meaning* moved while every declared outcome stayed green. `fx-0006` scores 60. That is the whole reason it exists, and the bar it clears is a new one for this corpus: not a behaviour that was unreachable, but a *distinction* the corpus could not draw.
+
 *A boolean decision never reaches the machinery that words the answer.* Converting the demo's API suite stopped at once: half its tests assert that verdict wording is **pack data** — `phrasing:` blocks, placeholders, tones — and the fixture pack's only decision was boolean, so it took the kernel's Yes/No fallback and never reached that path. The entire phrasing mechanism was untestable except through the teaching packs, which is precisely the dependency this phase exists to sever. The pack grew a money decision with a phrasing block, and the scenario grew a `sensitivity: pii` fact for the report renderer's redaction path. Growing it is a rebuild rather than an edit — `pack.version` is inside every receipt — and the rebuild found two literals that should always have been derived: a decision-digest vector hardcoding a "later" pack version that the bump made the *current* one, collapsing an equivalence class into its base, and a PROV-O test pinning the pack URN. The equivalence test caught the first, which is the whole reason it checks a relation rather than a table of constants.
 
 *A corpus is not a scenario, and the demo reads scenarios.* The fixture cases ground their facts in attestations, which is honest for synthetic data and leaves the whole span machinery — the evidence browser's highlighting, quotes in the audit report, span verification itself — with nothing to exercise. So the fixtures grew one: a generated PDF, its rendition, and two facts grounded in character spans of that rendition, one of them below the pack's confidence floor so a review arc has something to resolve. The spans are *found* in the rendition text by the builder rather than typed, and a check asserts every quote equals the slice it claims — a hand-counted offset is a fact that lies about its own evidence.

CLAUDE.md

@@ -23,7 +23,7 @@ New to the codebase? README for the argument, [docs/demo_tour.md](docs/demo_tour
 | `rulepacks/` | Six packs (insurance + mortgage closing), each `pack.yaml` + `expected.yaml` (+ `fixtures/`) |
 | `starters/` | Synthetic documents, renditions, span-verified facts, one demo scenario per vertical |
 | `golden/` | 351 committed cases + receipts — the replay/impact baseline |
-| `fixtures/` | The corpus **duly's own suites** run on: one invented domain, four cases, committed receipts. Toolkit, so it survives `git rm -r examples/` — read [fixtures/README.md](fixtures/README.md) before adding to it |
+| `fixtures/` | The corpus **duly's own suites** run on: one invented domain, five cases, committed receipts, declared outcomes, a scenario, DMN inputs. Toolkit, so it survives `git rm -r examples/` — read [fixtures/README.md](fixtures/README.md) before adding to it |
 | `whatif/` | Backward queries: free one input, solve the pack for it, verify every answer by re-running the kernel (`python -m duly_whatif`, optional z3) |
 | `dmn/` | DMN 1.3+ decision-table compiler: S-FEEL cell compiler, hit-policy mapping, deterministic pack emitter, CLI (`python -m duly_dmn`) — [dmn/README.md](dmn/README.md) |
 | `examples/` | Reference wiring: duly consumed from *outside*, by software that is not duly. `minimal-integration` is the whole contract at its smallest (and the only thing proved to run with the source tree absent); `closing-scheduler` is the CP-SAT one — [examples/README.md](examples/README.md) |
@@ -73,13 +73,13 @@ Run the full suite, replay, and spec validation before any commit. A change that
 - **Only `attribute` bindings can prove disjointness — `derived` ones cannot.** Narrower than the boolean-guard gotcha below and less visible: `_equality_guards` inspects only `when` items whose variable resolves to an *attribute* binding, so `category == "ZeroTolerance"` on a `derived` binding proves nothing however string-equal it looks. Two same-priority rules separated only by a derived-value guard need an explicit `overrides`.
 - **Test filenames collide across suites.** Test dirs have no `__init__.py`, so pytest imports by basename: `dmn/tests/test_cli.py` broke collection against `kernel/tests/test_cli.py`. Before adding a suite, run `find . -name "test_*.py" | sed 's#.*/##' | sort | uniq -d`.
 - **Boolean guards don't prove disjointness.** The pack validator's same-priority check accepts only *quoted-string* equality guards (`state == "US-NY"`) as a disjointness proof. Two rules split by `x == true` / `x == false` need an explicit `overrides`, even though they look disjoint.
-- **A test that would still pass with its subject deleted is not a test, and that is why `fixtures/` exists.** The toolkit's suites assert *toolkit* behaviour, so they run on [`fixtures/`](fixtures/README.md) — never on `golden/`, `rulepacks/` or `starters/`, which are example content M5 relocates under `examples/` for an adopter to delete. Delete the example content and a suite pointed at it does not fail: it stops being collected, or skips, or asserts over an empty glob, all of which read exactly like success. Two consequences. Writing a toolkit test, reach for `fixtures/`; writing a test whose *subject* is the example content (that the six packs load, that their declared outcomes hold), leave it pointed there — it moves with them. And the rule reaches further than tests: `spec/decision-digest-vectors.json` was a **contract artifact** generated from `golden/` receipts, which is the same defect with a longer fuse.
+- **A test that would still pass with its subject deleted is not a test, and that is why `fixtures/` exists.** The toolkit's suites assert *toolkit* behaviour, so they run on [`fixtures/`](fixtures/README.md) — never on `golden/`, `rulepacks/` or `starters/`, which are example content M5 relocates under `examples/` for an adopter to delete. Delete the example content and a suite pointed at it does not fail: it stops being collected, or skips, or asserts over an empty glob, all of which read exactly like success. **The two loudest-looking forms are the quietest**, and `test_rules_api` had both: a `for path in glob(...)` loop whose body simply never ran, and a `@pytest.mark.parametrize` over a glob — parametrize is evaluated at *collection*, so an empty directory produced zero test cases and pytest reported only the count that remained. The fix in each is one line: assert the glob is non-empty before using it, and loop inside the test rather than parametrizing over the filesystem. Two consequences. Writing a toolkit test, reach for `fixtures/`; writing a test whose *subject* is the example content (that the six packs load, that their declared outcomes hold), leave it pointed there — it moves with them. And the rule reaches further than tests: `spec/decision-digest-vectors.json` was a **contract artifact** generated from `golden/` receipts, which is the same defect with a longer fuse.
 - **`expected.yaml` is not the corpus.** Pack outcome declarations run in CI, but impact analysis runs *over `golden/`*. A pack without a generator template in `assurance/duly_assurance/generate.py` gets "0 decisions flip" for every edit. Both are required.
 - **Non-boolean decisions need *pack* phrasing.** A decision that isn't boolean renders as a bare `attribute = value` unless its `decisions[]` entry carries a `phrasing:` block ([spec/rule-ir.md](spec/rule-ir.md), "Decision phrasing"). The fix is in the pack, never in `demo/app.py`; `validate_pack` rejects a malformed block, an unknown placeholder, or a tone outside `pos/neg/warn/""` where the pack loads. Booleans still get a Yes/No fallback. Phrasing is presentation and must stay out of every hashed body — putting it in a receipt would change every hash.
 - **Rule ids are permanent, and now conventional.** Ids sit in `rulesFired` on every receipt that cited them, so an id encoding a day count, a year, or a statute section is wrong forever once the law moves — `NY-NR-45` is in 76 golden receipts. New ids are `<PREFIX>-<TOPIC>[-QUALIFIER][-NN]` with `PREFIX = pack.idPrefix`; `validate_pack` refuses digits outside the two-digit tail, a tail echoing the rule's own numbers, and an id outside the pack's family. The 46 pre-convention ids are exempt by an explicit list in [kernel/duly_kernel/rule_ids.py](kernel/duly_kernel/rule_ids.py) — 17 of them would fail today. Checks run only for packs declaring `idPrefix`; a kernel test requires every committed pack to declare one.
 - **The studio's decision tables are a projection, and re-emission drops the arguing part.** `demo/rules_api.py` renders packs as DMN-shaped grids using the *kernel's* expression parser (`duly_kernel.expr.parse` + a Var walk) — never a second parser, and never a DMN decompiler, because `duly_dmn` is one-way on purpose. Two consequences when you touch it. A guard referencing more than one binding belongs in the row's `cross` list, not in a column; flattening it into a cell would misstate the rule. And a structured edit re-emits the pack through `duly_dmn.emit.emit_pack`, which is byte-deterministic but carries no comments — so `DEMO-SYNTHETIC`, `TODO(verify)` and `MODELING BOUNDARY` headers are lost, and *nothing* catches it: validation passes, replay is byte-identical, impact reports zero. That is why the studio shows a normalised diff and the raw file diff side by side and keeps a lossless YAML-text path. `emit_pack` emits every top-level key in input order for the same reason — it used to emit a fixed `pack/decisions/rules` allow-list, which silently dropped `abstentionPolicy` and `calendars` from any hand-written pack it round-tripped.
 - **JavaScript's JSON silently breaks content addressing, and the symptom accuses the wrong party.** JS has one number type, so `JSON.parse`/`JSON.stringify` turns a fact's `"score": 1.0` into `1` — a different canonical body, a different `contentHash`. A browser that parses a receipt and posts it back as an object is asking the server to verify a document nobody emitted, and the server dutifully reports *every* genuine fact as tampered with. So `/api/receipts/inspect` takes raw JSON **text**, one blob per pasted box or chosen file, and Python does the only parse (`_parse_documents`); the server sorts receipts from facts by which field each carries, so no caller ever has to re-serialize to fit a shape. Any future surface that moves hashed bytes through the browser inherits this: move the text, never the object. Python round-trips `1.0` faithfully, which is why the API tests only caught it once they went through the real page.
-- **The evidence browser recomputes liveness on purpose, and `demo/tests/test_api.py` forces fixture mode on everyone.** `demo/evidence_api.py` replays the store's event log instead of calling `FactStore.as_of`, because as_of answers "which facts survive" and a browser also has to show the ones that did not — superseded, retracted, not-yet-known. The two projections must agree, and `test_the_browsers_projection_is_the_stores` walks every case at every point on its timeline asserting they do; keep them as separate implementations or that test becomes a tautology. Two more things bite. Its fact universe is seeded from the far-future `as_of` and walked through supersession chains, so a fact retracted *without* replacement is unreachable — nothing in the demo retracts, and the alternative is reaching past the store's public API. And `demo/tests/test_api.py` sets `DULY_DEMO_FORCE_FIXTURE=1` at **import** time, process-wide, so any store-backed demo suite must `monkeypatch.delenv` it and reset the runtime or it will pass alone and fail in the directory run. **The same binding cuts the other way, and is easier to miss:** a suite that points `DULY_DEMO_CONTENT` at a temp root must reload the demo modules *back* on teardown, because `monkeypatch` unsets the variable and un-reloads nothing — converting one suite to a fixture content root turned 0 failures into 50 across the rest of the directory until that teardown existed. [`demo/tests/demotest_helpers.py`](demo/tests/demotest_helpers.py) is the shape, including how a corpus becomes a content root.
+- **The evidence browser recomputes liveness on purpose, and `demo/tests/test_api.py` forces fixture mode on everyone.** `demo/evidence_api.py` replays the store's event log instead of calling `FactStore.as_of`, because as_of answers "which facts survive" and a browser also has to show the ones that did not — superseded, retracted, not-yet-known. The two projections must agree, and `test_the_browsers_projection_is_the_stores` walks every case at every point on its timeline asserting they do; keep them as separate implementations or that test becomes a tautology. Two more things bite. Its fact universe is seeded from the far-future `as_of` and walked through supersession chains, so a fact retracted *without* replacement is unreachable — nothing in the demo retracts, and the alternative is reaching past the store's public API. And `demo/tests/test_api.py` sets `DULY_DEMO_FORCE_FIXTURE=1` at **import** time, process-wide, so any store-backed demo suite must `monkeypatch.delenv` it and reset the runtime or it will pass alone and fail in the directory run. **The same binding cuts the other way, and is easier to miss:** a suite that points `DULY_DEMO_CONTENT` at a temp root must reload the demo modules *back* on teardown, because `monkeypatch` unsets the variable and un-reloads nothing — converting one suite to a fixture content root turned 0 failures into 50 across the rest of the directory until that teardown existed. Call [`demotest_helpers.reload_demo()`](demo/tests/demotest_helpers.py) on both sides rather than writing the reload out: it was copied into a second suite once, which is one copy short of the version that gets edited wrong. The same module is where a corpus becomes a content root — including the three path rewrites that assembly needs (`case.yaml`'s pack, `expected.yaml`'s `factsFrom`, the scenario's `rulePack`), each of which fails *silently* if skipped, by offering an empty list rather than an error.
 
 - **A receipt viewer is three checks, and only one of them needs the repository.** `demo/receipts_api.py` reports hash, fact-integrity and replay separately because a forged receipt that was *re-sealed* passes the first two — the interesting case, and the one a single "valid/invalid" pill would hide. Three consequences when you touch it. Never collapse the checks into one verdict field: `partial` (inputs unavailable) and `fail` (a check refuted) are different answers, and standalone receipts are the normal case, not the degraded one. Never fall back to the pack now on disk when `rulePack.version` has moved — the module returns `pack-moved` and omits rule text instead, because descriptions from a version the receipt never saw would read as the text its rules carried. And never import `duly_kernel` at module scope here: it is the one surface that can do nothing without the kernel, which is exactly what makes a top-level import tempting, and `demo/app.py` includes this router unconditionally — so that import takes *all four pages* down when the kernel is absent, instead of this one reporting "not checked". It reaches through `_kernel(name)` for the same reason every kernel reach in `app.py` is lazy. Two things also arrive as caller data on `/inspect` and must be treated as such: `rulePack.name` becomes a path (validated against a pack-name pattern before the join), and the receipt bytes themselves (see the JSON-fidelity gotcha above).
 - **`impact.analyze` takes `pack_overrides`; the CLI never passes it.** Keyed by *resolved* pack path, it seeds the pack cache so a candidate that exists only in memory is measured against the corpus slice it governs. Reading the working tree is the point of the command, so the flag is for callers holding a pack that is not (and may never be) a file — an impact number an author sees before writing the file is the only one that can change their mind.

assurance/tests/test_verify.py

@@ -55,7 +55,7 @@ def test_verify_passes_on_the_committed_golden_corpus(capsys):
 def test_verify_passes_on_a_copied_corpus(corpus, capsys):
     """Replay does not depend on where the corpus lives."""
     assert verify.main(["--golden", str(corpus)]) == 0
-    assert "verified 4 cases" in capsys.readouterr().out
+    assert "verified 5 cases" in capsys.readouterr().out
 
 
 def test_verify_fails_on_tampered_receipt_body(corpus, capsys):