diff --git a/CHANGELOG.md b/CHANGELOG.md index e79f175..4445881 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,8 @@ Also: `impact.analyze` gained `pack_overrides`, so a candidate pack that exists *It reached further than tests, which is what made it worth doing early.* [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) — a **contract artifact**, the thing another language implements duly's digest against — was generated from `golden/` receipts. Self-contained, so it would have survived the deletion intact; but the test proving its two "real" receipts are genuinely committed receipts would have quietly stopped meaning anything, and a spec artifact resting on content an adopter is invited to delete is the same category error one layer up. Rebuilt from `fixtures/`. +*The demo's review arc names a starter, and the naming is invisible because the failure is silent.* `demo/app.py` picks its review scenario by literal id — `notice-ny` — and wraps the ingest in `except Exception: pass`. Point the demo at any other content and the arc simply does not appear: no error, no log line, one fewer scenario in a list nobody counts. Degrading quietly is the right behaviour for a missing *demonstration*, which is exactly why hardcoding *which* demonstration it is went unnoticed — the two failures are indistinguishable from outside. The demo is toolkit, so this is example content compiled into it, and it is the last such dependency the surfaces carry. + *A phrasing guard that matches nothing fails quietly, which is worth a fixture.* Writing the fixture pack's money phrasing, the first guard was `when: { value: "0.00" }` — and a money decision's value is an amount and a currency, not a scalar, so it matched nothing. The demo did not complain: it fell through to the unguarded case and rendered a plausible answer. The correct form is `amount: positive`. The fixture keeps the corrected version with a comment naming the mistake, because a pack author will make it and the failure mode is a verdict that reads fine and came from the wrong branch. *A boolean decision never reaches the machinery that words the answer.* Converting the demo's API suite stopped at once: half its tests assert that verdict wording is **pack data** — `phrasing:` blocks, placeholders, tones — and the fixture pack's only decision was boolean, so it took the kernel's Yes/No fallback and never reached that path. The entire phrasing mechanism was untestable except through the teaching packs, which is precisely the dependency this phase exists to sever. The pack grew a money decision with a phrasing block, and the scenario grew a `sensitivity: pii` fact for the report renderer's redaction path. Growing it is a rebuild rather than an edit — `pack.version` is inside every receipt — and the rebuild found two literals that should always have been derived: a decision-digest vector hardcoding a "later" pack version that the bump made the *current* one, collapsing an equivalence class into its base, and a PROV-O test pinning the pack URN. The equivalence test caught the first, which is the whole reason it checks a relation rather than a table of constants. diff --git a/CLAUDE.md b/CLAUDE.md index a9207bd..7569b48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,6 +85,7 @@ Run the full suite, replay, and spec validation before any commit. A change that - **`impact.analyze` takes `pack_overrides`; the CLI never passes it.** Keyed by *resolved* pack path, it seeds the pack cache so a candidate that exists only in memory is measured against the corpus slice it governs. Reading the working tree is the point of the command, so the flag is for callers holding a pack that is not (and may never be) a file — an impact number an author sees before writing the file is the only one that can change their mind. - **Scripted confidences need the stub pin.** If a scenario depends on an exact confidence value, set `"demoExtractor": "stub"` in its `scenario.json` — Docling emits its own measured confidence and silently overrides the scripted one. No test warns about this at authoring time. - **`ReviewQueue.resolve` refuses a `low_confidence` correction that supersedes nothing, and cannot fix it for you.** Resolving such an item is a ruling on the one fact it abstained over, so the correction must carry `supersedes` ([spec/compatibility.md](spec/compatibility.md) C6) — otherwise the below-floor fact stays live and every future receipt carries a `low_confidence` entry for an attribute the decision *used*, which contradicts the fact spec's own definition of `abstentions`. The obvious fix — have the queue stamp the field in from the entry — is unavailable and instructive: a correction arrives content-addressed, so writing into it changes its hash and its identity, and the queue would hand the store a document its author never sealed. It names the required fact id and refuses. Three carve-outs, all deliberate: `conflict` items are untouched (their entries name several facts), `FactStore.ingest` still takes an independent human fact that supersedes nothing (a value from a phone call is not a ruling on an extraction), and `duly_review.golden`'s converter stays permissive because it is handed items by callers it does not control. +- **The demo's review arc is derived from example content by name, and fails silently when it is absent.** `demo/app.py` declares `REVIEW_SOURCE_SCENARIO = "notice-ny"` (plus a matching id, case id and title), and `_ingest_review_case` is wrapped in `except Exception: pass` — so pointing `DULY_DEMO_CONTENT` anywhere else makes the arc quietly not appear rather than complain. Degrading quietly is right for a missing *demonstration*; naming which one inside toolkit code is the defect, and the silence is what hides it. Phase 3 owns the fix (derive the source from whichever scenario carries a below-floor extraction, preferring `notice-ny` when present). Until then, any demo suite you point at fixture content will lose its review-arc coverage without a single failure. - **One entity per `entityType` per case; one live fact per attribute.** Per-document decisions mean one case per document, or the document type as an attribute of a single entity. Two live facts on one attribute is a conflict (lone human outranks; anything else abstains). - **Replay is scoped to a semantics version, and "the digest" is not the receipt hash.** Two things the freeze ([spec/compatibility.md](spec/compatibility.md)) made load-bearing. First: every replay path calls `duly_kernel.semantics.check_replayable` *before* re-adjudicating, and a receipt whose `engine.version` is not in `IMPLEMENTED` is refused rather than replayed. The failure that guard exists for is silent — a kernel implementing V2 replaying a V1 receipt and **passing**, on the subset of cases where the two semantics happen to agree — so never "fix" a refusal by widening `IMPLEMENTED`: an entry there is a claim that this kernel reproduces that version byte-for-byte, substantiated only by corpus cases at that version. Second: `decision_digest()` is a pure function over a receipt's *determinant* fields and is never stored in any document (the receipt schema is closed; a digest inside the body it digests is exactly the extension point C2 refuses). It excludes `caseId`, `rulePack.gitCommit`/`url`, `engine.kernel` and `engine.backend` — everything identifying the run rather than the adjudication — which is what makes it, and not byte equality, the definition of two evaluation backends agreeing. Changing the determinant set is a breaking change to a published contract, not a refactor; [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) and a corpus-aggregate digest both fail on it. - **The whole `engine` block is inside the receipt hash, and `engine.version` is NOT a package version.** Two consequences, and the second one will bite during release work. `engine.backend`: a second evaluation backend cannot produce byte-identical receipts by construction; cross-backend equivalence is an open M5 spec decision, so don't design around an assumption either way. `engine.version`: it is the version of the kernel's **decision semantics**, pinned as `receipt.SEMANTICS_VERSION` and deliberately decoupled from `duly_kernel.__version__` and from the `duly` distribution version. Three scopes, nested one way only: the **distribution** moves on any release of anything in the wheel, `duly_kernel.__version__` moves when the kernel's *code* changes, and `SEMANTICS_VERSION` moves when the kernel's *meaning* changes. A semantics change implies a code change implies a release; never the reverse, because a release can be a demo fix with the kernel byte-identical. All three read `0.0.1` today, which is a coincidence and is exactly what makes this easy to break silently. It used to be `__version__`, which made the corpus a hostage to the release cadence — publishing under any new number would have invalidated all 351 receipts without a rule, a fact, or a decision changing. **Never bump `SEMANTICS_VERSION` as part of a release.** It moves only when what the kernel means changes, and moving it rewrites every committed receipt. The value stays `0.0.1` for the same reason `NY-NR-45` keeps its name: it is sealed into artifacts that are not editable, so it is a handle, not a claim — and what a package version is for has a correctable home in the wheel metadata, while this does not. [kernel/tests/test_engine_identity.py](kernel/tests/test_engine_identity.py) fails on the re-coupling edit, which is exactly the tidying-up someone does while packaging. [docs/release-process.md](docs/release-process.md) is the decision procedure for which of the four scopes moves for a given change. diff --git a/demo/tests/test_content_roots.py b/demo/tests/test_content_roots.py index a885281..b62b57b 100644 --- a/demo/tests/test_content_roots.py +++ b/demo/tests/test_content_roots.py @@ -129,7 +129,14 @@ def test_the_built_in_fixture_scenario_is_content_too(empty_content): def test_the_repo_default_still_finds_everything(): - """The other half of the claim: configurable did not mean broken here.""" + """The other half of the claim: configurable did not mean broken here. + + Example content, and the *only* test in this file that is: its subject is + what this repository happens to ship, so it moves under `examples/` with + the six packs rather than being repointed at a fixture. Everything above + asserts the toolkit behaviour — that the roots are configurable and that + the surfaces stand up against nothing — and stays. + """ assert CONTENT.rulepacks.is_dir() assert CONTENT.golden.is_dir() assert len(list(CONTENT.rulepacks.glob("*/pack.yaml"))) == 6 diff --git a/docs/m5-plan.md b/docs/m5-plan.md index c2298eb..96ecc3a 100644 --- a/docs/m5-plan.md +++ b/docs/m5-plan.md @@ -552,11 +552,34 @@ proves it. exercises `{daysBetween:…}` and `{fact:…|day}`, so the pack needs a **date** attribute. Cheap, but it is another rebuild — do it with part 5's other growth rather than alone.* +- [ ] **Fixtures, part 4b(ii)c-0: the demo's review arc names a starter, and + that is toolkit code holding example content (1 PR).** **Found while + scoping the evidence suite, and it blocks both remaining suites.** + [`demo/app.py`](../demo/app.py) declares `REVIEW_SOURCE_SCENARIO = + "notice-ny"`, plus `REVIEW_SCENARIO_ID`, `REVIEW_CASE_ID` and + `REVIEW_SCENARIO_TITLE` naming that same starter. The demo is toolkit + (D2), so this is a teaching scenario's name compiled into it: point the + demo at any other content and the arc simply does not appear, because + `_ingest_review_case` is wrapped in `except Exception: pass`. + + Degrading quietly is right for a missing *demonstration*; hardcoding + *which* demonstration is not, and the silence is what makes it invisible. + Derive the arc instead — from whichever scenario carries a below-floor + extraction, preferring `notice-ny` when present so this repository's demo + is byte-unchanged. Roughly nine call sites in `app.py` and one constant + `test_review_arc.py` imports; the derived values move onto the runtime. + + *Attempted inside part 4b(ii)c and backed out deliberately: a + half-refactored `app.py` is worse than a routed finding, and the four + constants are load-bearing for two suites at once.* - [ ] **Fixtures, part 4b(ii)c: the last three demo suites (1 PR).** 52 failures: `test_rules_api` 31, `test_evidence_api` 13, `test_review_arc` 8, plus the one in `test_content_roots` that asserts six packs and moves. - All three want the content root `demotest_helpers` already builds; none is - blocked on a missing artifact. + All three want the content root `demotest_helpers` already builds. + `test_evidence_api` and `test_review_arc` also want c-0 above — both + assert on the review arc, which cannot exist in a fixture deployment + until it stops naming a starter. `test_rules_api` (31, the largest) + does not, and can go first. - [ ] **Fixtures, part 5: kernel report, whatif, dmn, extraction (1 PR).** `kernel/tests/test_report.py` — **its blocker is gone.** The scenario supplies document-grounded facts with quotes and spans, a @@ -1021,6 +1044,12 @@ and every exception behind it.** finding is that a default which is right in this repository hides both the wrong-path case and every exception behind it. `wheel_smoke.py` also stopped claiming whatif "has no repo-relative file reads", which was false. +- 2026-08-06 — **Phase 3, part 4b(ii)c scoped** — found that the demo's review + arc names a starter (`REVIEW_SOURCE_SCENARIO = "notice-ny"`) inside toolkit + code and fails silently when it is absent, which blocks two of the three + remaining suites. Routed as its own task rather than half-refactored; + recorded as a CLAUDE.md gotcha so it cannot be rediscovered by surprise. The + one example test in `test_content_roots` is labelled as such. - 2026-08-06 — **Phase 3, part 4b(ii)b: `test_api`** — 10 failures under deletion down to 4, three of which are example tests that move. Found a mis-authored phrasing guard in the pack added one PR earlier: `value:` on a