From 7230a8ad59c98fe5b55a9ce5ba52e4a037e3ae93 Mon Sep 17 00:00:00 2001 From: Kushan Patel Date: Thu, 6 Aug 2026 08:03:06 -0700 Subject: [PATCH 1/2] M5 Phase 3: the demo's review arc becomes content instead of a constant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit demo/app.py named notice-ny in four constants — a teaching scenario's id compiled into toolkit code. A scenario now opts into the arc by carrying a `reviewArc` block in its manifest: which attribute to script below the pack's floor, at what confidence, what to call the result. The demo takes the first scenario declaring one, in sorted order, and names none itself. That is the same shape `demoExtractor` already had. Which demonstration the demo shows is content; that it can show one is code. This repository's arc is unchanged in every field — same id, title, case id, default as-of, domain. That was the check that mattered, because the point was to remove a name from toolkit code, not to change what the demo shows. Removing the name uncovered what it was hiding. A fixture-only deployment still gets no arc: the arc lives only in the session store, and store-backed ingest needs extraction targets that the fixture scenario does not have. One silent failure was standing in front of another, which is the recurring shape of this phase — a hardcoded path does not merely couple two things, it conceals everything downstream of itself. Fixture targets are routed to 4b(ii)c, and the plan says why they are needed rather than only that they are. test_review_arc derives the id from the suffix rather than importing a constant, so a fixture deployment would get its own. Verification: 975 passed, 7 skipped; verify 351 byte-for-byte; golden/, fixtures/cases and fixtures/receipts untouched — the manifest change is presentation and touches no hashed body; the arc re-checked field by field against a live load_scenarios(); deletion re-measured in a detached worktree. Documentation pass per CLAUDE.md: the gotcha rewritten around what is now true (content-declared, store-only, silent by design), the plan carrying the uncovered dependency, and the changelog carrying what the removal taught. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 4 +- CLAUDE.md | 2 +- demo/app.py | 114 +++++++++++++++++++++++-------- demo/tests/test_review_arc.py | 6 +- docs/m5-plan.md | 47 +++++++------ fixtures/build.py | 10 +++ fixtures/scenario/scenario.json | 10 +++ starters/notice-ny/scenario.json | 10 +++ 8 files changed, 147 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4445881..2923c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,7 +76,9 @@ Also: `impact.analyze` gained `pack_overrides`, so a candidate pack that exists *It reached further than tests, which is what made it worth doing early.* [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) — a **contract artifact**, the thing another language implements duly's digest against — was generated from `golden/` receipts. Self-contained, so it would have survived the deletion intact; but the test proving its two "real" receipts are genuinely committed receipts would have quietly stopped meaning anything, and a spec artifact resting on content an adopter is invited to delete is the same category error one layer up. Rebuilt from `fixtures/`. -*The demo's review arc names a starter, and the naming is invisible because the failure is silent.* `demo/app.py` picks its review scenario by literal id — `notice-ny` — and wraps the ingest in `except Exception: pass`. Point the demo at any other content and the arc simply does not appear: no error, no log line, one fewer scenario in a list nobody counts. Degrading quietly is the right behaviour for a missing *demonstration*, which is exactly why hardcoding *which* demonstration it is went unnoticed — the two failures are indistinguishable from outside. The demo is toolkit, so this is example content compiled into it, and it is the last such dependency the surfaces carry. +*The demo's review arc is content now, and removing the hardcoded name uncovered what it was hiding.* `demo/app.py` used to pick its review scenario by literal id — `notice-ny` — in four constants, with the ingest wrapped in `except Exception: pass`. Point the demo at other content and the arc simply did not appear: no error, no log line. Degrading quietly is right for a missing *demonstration*, which is exactly why hardcoding *which* one went unnoticed; the two failures are indistinguishable from outside. A scenario now opts in with a `reviewArc` block in its manifest — the attribute to script below the pack's floor, the confidence, the title — the same way it opts into the stub extractor. This repository's arc is unchanged in every field, which was the check that mattered: the point was to remove a name from toolkit code, not to change what the demo shows. + +*And the name was hiding a second dependency.* With it gone, a fixture-only deployment still gets no arc — because the arc lives only in the session store, and store-backed ingest needs extraction *targets* that the fixture scenario does not have. One silent failure was standing in front of another. That is the recurring shape of this phase: a hardcoded path does not merely couple two things, it conceals everything downstream of itself. *A phrasing guard that matches nothing fails quietly, which is worth a fixture.* Writing the fixture pack's money phrasing, the first guard was `when: { value: "0.00" }` — and a money decision's value is an amount and a currency, not a scalar, so it matched nothing. The demo did not complain: it fell through to the unguarded case and rendered a plausible answer. The correct form is `amount: positive`. The fixture keeps the corrected version with a comment naming the mistake, because a pack author will make it and the failure mode is a verdict that reads fine and came from the wrong branch. diff --git a/CLAUDE.md b/CLAUDE.md index 7569b48..afce8ad 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,7 +85,7 @@ Run the full suite, replay, and spec validation before any commit. A change that - **`impact.analyze` takes `pack_overrides`; the CLI never passes it.** Keyed by *resolved* pack path, it seeds the pack cache so a candidate that exists only in memory is measured against the corpus slice it governs. Reading the working tree is the point of the command, so the flag is for callers holding a pack that is not (and may never be) a file — an impact number an author sees before writing the file is the only one that can change their mind. - **Scripted confidences need the stub pin.** If a scenario depends on an exact confidence value, set `"demoExtractor": "stub"` in its `scenario.json` — Docling emits its own measured confidence and silently overrides the scripted one. No test warns about this at authoring time. - **`ReviewQueue.resolve` refuses a `low_confidence` correction that supersedes nothing, and cannot fix it for you.** Resolving such an item is a ruling on the one fact it abstained over, so the correction must carry `supersedes` ([spec/compatibility.md](spec/compatibility.md) C6) — otherwise the below-floor fact stays live and every future receipt carries a `low_confidence` entry for an attribute the decision *used*, which contradicts the fact spec's own definition of `abstentions`. The obvious fix — have the queue stamp the field in from the entry — is unavailable and instructive: a correction arrives content-addressed, so writing into it changes its hash and its identity, and the queue would hand the store a document its author never sealed. It names the required fact id and refuses. Three carve-outs, all deliberate: `conflict` items are untouched (their entries name several facts), `FactStore.ingest` still takes an independent human fact that supersedes nothing (a value from a phone call is not a ruling on an extraction), and `duly_review.golden`'s converter stays permissive because it is handed items by callers it does not control. -- **The demo's review arc is derived from example content by name, and fails silently when it is absent.** `demo/app.py` declares `REVIEW_SOURCE_SCENARIO = "notice-ny"` (plus a matching id, case id and title), and `_ingest_review_case` is wrapped in `except Exception: pass` — so pointing `DULY_DEMO_CONTENT` anywhere else makes the arc quietly not appear rather than complain. Degrading quietly is right for a missing *demonstration*; naming which one inside toolkit code is the defect, and the silence is what hides it. Phase 3 owns the fix (derive the source from whichever scenario carries a below-floor extraction, preferring `notice-ny` when present). Until then, any demo suite you point at fixture content will lose its review-arc coverage without a single failure. +- **The demo's review arc is content, and it is store-only.** A scenario opts in with a `reviewArc` block in its manifest (attribute to script below the floor, confidence, title, case id) — `demo/app.py` names no scenario, having previously hardcoded `notice-ny` in four constants. Two things still bite. The arc exists **only** in the session store, so a deployment whose scenarios cannot be store-ingested has none — and store ingest needs extraction *targets* under `starters/tools/targets`, which is a second dependency the hardcoded name used to hide. And `_ingest_review_case` is wrapped in `except Exception: pass`, so every one of those failures is silent by design: right for a missing demonstration, and the reason a missing *arc* tells you nothing about why. - **One entity per `entityType` per case; one live fact per attribute.** Per-document decisions mean one case per document, or the document type as an attribute of a single entity. Two live facts on one attribute is a conflict (lone human outranks; anything else abstains). - **Replay is scoped to a semantics version, and "the digest" is not the receipt hash.** Two things the freeze ([spec/compatibility.md](spec/compatibility.md)) made load-bearing. First: every replay path calls `duly_kernel.semantics.check_replayable` *before* re-adjudicating, and a receipt whose `engine.version` is not in `IMPLEMENTED` is refused rather than replayed. The failure that guard exists for is silent — a kernel implementing V2 replaying a V1 receipt and **passing**, on the subset of cases where the two semantics happen to agree — so never "fix" a refusal by widening `IMPLEMENTED`: an entry there is a claim that this kernel reproduces that version byte-for-byte, substantiated only by corpus cases at that version. Second: `decision_digest()` is a pure function over a receipt's *determinant* fields and is never stored in any document (the receipt schema is closed; a digest inside the body it digests is exactly the extension point C2 refuses). It excludes `caseId`, `rulePack.gitCommit`/`url`, `engine.kernel` and `engine.backend` — everything identifying the run rather than the adjudication — which is what makes it, and not byte equality, the definition of two evaluation backends agreeing. Changing the determinant set is a breaking change to a published contract, not a refactor; [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) and a corpus-aggregate digest both fail on it. - **The whole `engine` block is inside the receipt hash, and `engine.version` is NOT a package version.** Two consequences, and the second one will bite during release work. `engine.backend`: a second evaluation backend cannot produce byte-identical receipts by construction; cross-backend equivalence is an open M5 spec decision, so don't design around an assumption either way. `engine.version`: it is the version of the kernel's **decision semantics**, pinned as `receipt.SEMANTICS_VERSION` and deliberately decoupled from `duly_kernel.__version__` and from the `duly` distribution version. Three scopes, nested one way only: the **distribution** moves on any release of anything in the wheel, `duly_kernel.__version__` moves when the kernel's *code* changes, and `SEMANTICS_VERSION` moves when the kernel's *meaning* changes. A semantics change implies a code change implies a release; never the reverse, because a release can be a demo fix with the kernel byte-identical. All three read `0.0.1` today, which is a coincidence and is exactly what makes this easy to break silently. It used to be `__version__`, which made the corpus a hostage to the release cadence — publishing under any new number would have invalidated all 351 receipts without a rule, a fact, or a decision changing. **Never bump `SEMANTICS_VERSION` as part of a release.** It moves only when what the kernel means changes, and moving it rewrites every committed receipt. The value stays `0.0.1` for the same reason `NY-NR-45` keeps its name: it is sealed into artifacts that are not editable, so it is a handle, not a claim — and what a package version is for has a correctable home in the wheel metadata, while this does not. [kernel/tests/test_engine_identity.py](kernel/tests/test_engine_identity.py) fails on the re-coupling edit, which is exactly the tidying-up someone does while packaging. [docs/release-process.md](docs/release-process.md) is the decision procedure for which of the four scopes moves for a given change. diff --git a/demo/app.py b/demo/app.py index 669615c..d36afd1 100644 --- a/demo/app.py +++ b/demo/app.py @@ -237,10 +237,19 @@ def _fmt_score(value: Any) -> str: # the server restarts, and the fixture fallback path keeps working without a # store at all. -REVIEW_SCENARIO_ID = "notice-ny-review" -REVIEW_SOURCE_SCENARIO = "notice-ny" -REVIEW_CASE_ID = "case:policy:HO-77401-NY:review-demo" -REVIEW_SCENARIO_TITLE = "NY nonrenewal — review arc (low-confidence extraction)" +# The review arc is **content, not code**. A scenario opts in by carrying a +# `reviewArc` block in its manifest — which attribute to script below the +# floor, at what confidence, and what to call the result — exactly as it opts +# into the stub extractor with `demoExtractor`. +# +# This used to be four constants naming `notice-ny`: a teaching scenario's id +# compiled into the toolkit. Pointing `DULY_DEMO_CONTENT` anywhere else made +# the arc silently not appear, because the ingest is wrapped in +# `except Exception: pass` — and "this deployment has no review scenario" and +# "it has one but we only look for notice-ny" are indistinguishable from +# outside. That is the failure mode that hid the hardcoding. +REVIEW_ID_SUFFIX = "-review" +REVIEW_RUN_ID_TEMPLATE = "run:{scenario}:review-demo" # Scenario domains group the picker by regulated vertical. The slug comes from # the starter manifest's optional `domain` field — presentation metadata only, @@ -258,12 +267,29 @@ def _domain_fields(slug: str | None) -> dict[str, str]: slug = (slug or DEFAULT_DOMAIN).strip().lower() or DEFAULT_DOMAIN label = DOMAIN_LABELS.get(slug, slug.replace("-", " ").title()) return {"domain": slug, "domainLabel": label} -REVIEW_MAILED_ATTRIBUTE = "nc:noticeMailedDate" -# Mirrors golden/cases/review-0001: 0.62 sits below the notice pack 2026.3.0 -# per-attribute floor of 0.9, so the mailed date abstains with low_confidence. -REVIEW_MAILED_CONFIDENCE = {"score": 0.62, "method": "platt"} -REVIEW_RUN_ID = "run:review-demo:001" -REVIEW_DEFAULT_AS_OF = "2026-07-25" + + +def _review_spec(manifest: dict[str, Any]) -> dict[str, Any] | None: + """A scenario's `reviewArc` block, validated enough to build from. + + `attribute` and `confidence` are required — without them there is nothing + to script below the floor, and an arc that abstains over nothing is not a + demonstration. Everything else has a defensible default. + """ + block = manifest.get("reviewArc") + if not isinstance(block, dict): + return None + attribute = block.get("attribute") + confidence = block.get("confidence") + if not isinstance(attribute, str) or not isinstance(confidence, dict): + return None + return { + "attribute": attribute, + "confidence": dict(confidence), + "title": block.get("title") or f"{manifest.get('title', manifest['id'])} — review arc", + "caseId": block.get("caseId") or f"{manifest['caseId']}:review-demo", + "defaultAsOf": block.get("defaultAsOf"), + } SESSION_NOTE = ( "Corrections live in this demo process only — the in-memory fact store " @@ -486,6 +512,29 @@ def _ingest_starter_case( return meta +def _find_review_source() -> tuple[Path, dict[str, Any], dict[str, Any]] | None: + """The first scenario whose manifest opts into the review arc. + + Sorted, so a deployment with two of them gets a stable answer rather than + a filesystem-order one — the same reason every other discovery here is + sorted. + """ + if not STARTERS_DIR.is_dir(): + return None + for scenario_dir in sorted(STARTERS_DIR.iterdir()): + manifest_path = scenario_dir / "scenario.json" + if not scenario_dir.is_dir() or not manifest_path.exists(): + continue + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + spec = _review_spec(manifest) + if spec is not None: + return scenario_dir, manifest, spec + return None + + def _ingest_review_case( runtime: _DemoRuntime, targets_by_doc: dict[str, dict[str, Any]], @@ -501,13 +550,16 @@ def _ingest_review_case( confidence would clear the floor and defeat the demonstration. The UI labels the extractor, so the scripting is visible, not hidden. """ - source_dir = STARTERS_DIR / REVIEW_SOURCE_SCENARIO - manifest_path = source_dir / "scenario.json" - if not manifest_path.exists(): + found = _find_review_source() + if found is None: return - manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + source_dir, manifest, spec_block = found + scenario_id = f'{manifest["id"]}{REVIEW_ID_SUFFIX}' + review_case_id = spec_block["caseId"] meta: dict[str, Any] = { - "caseId": REVIEW_CASE_ID, + "caseId": review_case_id, + "reviewSpec": spec_block, + "sourceId": manifest["id"], "packPath": manifest["rulePack"], "renditions": {}, "extraction": None, @@ -524,11 +576,11 @@ def _ingest_review_case( if spec is None or pdf_path is None or rend_path is None: return spec = copy.deepcopy(spec) - spec["caseId"] = REVIEW_CASE_ID - spec["runId"] = REVIEW_RUN_ID + spec["caseId"] = review_case_id + spec["runId"] = REVIEW_RUN_ID_TEMPLATE.format(scenario=manifest["id"]) for target in spec.get("facts", []): - if target.get("attribute") == REVIEW_MAILED_ATTRIBUTE: - target["confidence"] = dict(REVIEW_MAILED_CONFIDENCE) + if target.get("attribute") == spec_block["attribute"]: + target["confidence"] = dict(spec_block["confidence"]) document = source_document_cls.from_bytes(doc["id"], pdf_path.read_bytes()) rendition_text = rend_path.read_text(encoding="utf-8") result = stub_cls(rendition_text).extract(document, spec) @@ -536,7 +588,7 @@ def _ingest_review_case( meta["renditions"][doc["id"]] = result.rendition.text meta["extraction"] = _extraction_meta(result, "stub") if meta["renditions"]: - runtime.cases[REVIEW_SCENARIO_ID] = meta + runtime.cases[scenario_id] = meta # --------------------------------------------------------------------------- @@ -738,10 +790,14 @@ def _build_review_scenario(runtime: _DemoRuntime) -> dict[str, Any] | None: """The review-arc scenario, synthesized from the runtime's ingested case (never from starters/ — the below-floor fact set exists only in the session store).""" - meta = runtime.cases.get(REVIEW_SCENARIO_ID) - if meta is None: + scenario_id = next( + (k for k, v in runtime.cases.items() if v.get("reviewArc")), None + ) + if scenario_id is None: return None - manifest_path = STARTERS_DIR / REVIEW_SOURCE_SCENARIO / "scenario.json" + meta = runtime.cases[scenario_id] + spec_block = meta["reviewSpec"] + manifest_path = STARTERS_DIR / meta["sourceId"] / "scenario.json" try: manifest = json.loads(manifest_path.read_text(encoding="utf-8")) pack = yaml.safe_load((CONTENT.root / meta["packPath"]).read_text(encoding="utf-8")) @@ -756,15 +812,15 @@ def _build_review_scenario(runtime: _DemoRuntime) -> dict[str, Any] | None: for doc_id, text in meta["renditions"].items() } return { - "id": REVIEW_SCENARIO_ID, - "title": REVIEW_SCENARIO_TITLE, - **_domain_fields("insurance"), - "caseId": REVIEW_CASE_ID, + "id": scenario_id, + "title": spec_block["title"], + **_domain_fields(manifest.get("domain")), + "caseId": meta["caseId"], "documents": documents, - "facts": runtime.store.as_of(REVIEW_CASE_ID, knowledge=_now_knowledge()), + "facts": runtime.store.as_of(meta["caseId"], knowledge=_now_knowledge()), "pack": pack, "questions": questions, - "defaultAsOf": REVIEW_DEFAULT_AS_OF, + "defaultAsOf": spec_block.get("defaultAsOf") or manifest.get("defaultAsOf"), "source": "review-demo", "extraction": meta.get("extraction"), "review": {"available": True, "note": None}, diff --git a/demo/tests/test_review_arc.py b/demo/tests/test_review_arc.py index 8c59cd3..391d8d5 100644 --- a/demo/tests/test_review_arc.py +++ b/demo/tests/test_review_arc.py @@ -30,7 +30,11 @@ from fastapi.testclient import TestClient # noqa: E402 import demo.app as demo_app # noqa: E402 -from demo.app import REVIEW_SCENARIO_ID, app # noqa: E402 +from demo.app import REVIEW_ID_SUFFIX, app # noqa: E402 + +# Derived, not imported: the arc's id follows whichever scenario opts into it, +# so a fixture deployment gets its own rather than this repository's. +REVIEW_SCENARIO_ID = f"notice-ny{REVIEW_ID_SUFFIX}" client = TestClient(app) diff --git a/docs/m5-plan.md b/docs/m5-plan.md index 96ecc3a..b70cab6 100644 --- a/docs/m5-plan.md +++ b/docs/m5-plan.md @@ -552,34 +552,33 @@ proves it. exercises `{daysBetween:…}` and `{fact:…|day}`, so the pack needs a **date** attribute. Cheap, but it is another rebuild — do it with part 5's other growth rather than alone.* -- [ ] **Fixtures, part 4b(ii)c-0: the demo's review arc names a starter, and - that is toolkit code holding example content (1 PR).** **Found while - scoping the evidence suite, and it blocks both remaining suites.** - [`demo/app.py`](../demo/app.py) declares `REVIEW_SOURCE_SCENARIO = - "notice-ny"`, plus `REVIEW_SCENARIO_ID`, `REVIEW_CASE_ID` and - `REVIEW_SCENARIO_TITLE` naming that same starter. The demo is toolkit - (D2), so this is a teaching scenario's name compiled into it: point the - demo at any other content and the arc simply does not appear, because - `_ingest_review_case` is wrapped in `except Exception: pass`. - - Degrading quietly is right for a missing *demonstration*; hardcoding - *which* demonstration is not, and the silence is what makes it invisible. - Derive the arc instead — from whichever scenario carries a below-floor - extraction, preferring `notice-ny` when present so this repository's demo - is byte-unchanged. Roughly nine call sites in `app.py` and one constant - `test_review_arc.py` imports; the derived values move onto the runtime. - - *Attempted inside part 4b(ii)c and backed out deliberately: a - half-refactored `app.py` is worse than a routed finding, and the four - constants are load-bearing for two suites at once.* +- [x] **Fixtures, part 4b(ii)c-0: the review arc becomes content (1 PR).** + The four constants naming `notice-ny` are gone. A scenario opts into the + arc by carrying a `reviewArc` block in its manifest — which attribute to + script below the floor, at what confidence, and what to call the result — + exactly as it opts into the stub extractor with `demoExtractor`. The demo + finds the first scenario declaring one, in sorted order. + *This repository's arc is unchanged in every field: same id, title, case + id, default as-of, domain. That is the check that mattered, because the + point was to remove a name from toolkit code, not to change what the demo + shows.* + *One thing the derivation exposed and did not fix: a fixture-only + deployment still gets **no** arc, because store-backed ingest needs + extraction **targets** (`starters/tools/targets`) and the fixture + scenario has none — it falls back to disk-backed, and the arc is + store-only by construction. The hardcoded name was hiding a second + dependency behind it. Fixture targets are part of 4b(ii)c below.* - [ ] **Fixtures, part 4b(ii)c: the last three demo suites (1 PR).** 52 failures: `test_rules_api` 31, `test_evidence_api` 13, `test_review_arc` 8, plus the one in `test_content_roots` that asserts six packs and moves. All three want the content root `demotest_helpers` already builds. - `test_evidence_api` and `test_review_arc` also want c-0 above — both - assert on the review arc, which cannot exist in a fixture deployment - until it stops naming a starter. `test_rules_api` (31, the largest) - does not, and can go first. + `test_evidence_api` and `test_review_arc` assert on the review arc, which + c-0 made content-declared but which a fixture deployment still cannot + produce: store-backed ingest needs extraction **targets**, and the + fixture scenario has none. Add them here — the shape is + `starters/tools/targets/.json`, and `fixtures/build.py` should + emit them beside the scenario it already builds. `test_rules_api` (31, + the largest) needs none of this and can go first. - [ ] **Fixtures, part 5: kernel report, whatif, dmn, extraction (1 PR).** `kernel/tests/test_report.py` — **its blocker is gone.** The scenario supplies document-grounded facts with quotes and spans, a diff --git a/fixtures/build.py b/fixtures/build.py index 7dff7f9..a060ff2 100644 --- a/fixtures/build.py +++ b/fixtures/build.py @@ -266,6 +266,16 @@ def _build_scenario() -> None: "caseId": f"case:fixture:{SCENARIO_CASE}", "ontology": dict(ONTOLOGY), "demoExtractor": "stub", + # Opt into the demo's review arc. The arc is content, not code: + # a scenario names the attribute to script below the pack's floor + # and the demo derives the rest (see demo/app.py `_review_spec`). + "reviewArc": { + "attribute": "fx:score", + "confidence": {"score": 0.55, "method": "platt"}, + "title": "Widget inspection — review arc (fixture)", + "caseId": f"case:fixture:{SCENARIO_CASE}:review-demo", + "defaultAsOf": "2026-06-01", + }, "documents": [ { "id": doc["id"], diff --git a/fixtures/scenario/scenario.json b/fixtures/scenario/scenario.json index 73a7c93..d269059 100644 --- a/fixtures/scenario/scenario.json +++ b/fixtures/scenario/scenario.json @@ -8,6 +8,16 @@ "version": "0.1.0" }, "demoExtractor": "stub", + "reviewArc": { + "attribute": "fx:score", + "confidence": { + "score": 0.55, + "method": "platt" + }, + "title": "Widget inspection — review arc (fixture)", + "caseId": "case:fixture:fx-0005:review-demo", + "defaultAsOf": "2026-06-01" + }, "documents": [ { "id": "doc:widget-report:FX-INSPECTION-0005", diff --git a/starters/notice-ny/scenario.json b/starters/notice-ny/scenario.json index 40bc156..2e441a9 100644 --- a/starters/notice-ny/scenario.json +++ b/starters/notice-ny/scenario.json @@ -7,6 +7,16 @@ "ontology": "duly-starter-notice", "version": "0.1.0" }, + "reviewArc": { + "attribute": "nc:noticeMailedDate", + "confidence": { + "score": 0.62, + "method": "platt" + }, + "title": "NY nonrenewal — review arc (low-confidence extraction)", + "caseId": "case:policy:HO-77401-NY:review-demo", + "defaultAsOf": "2026-07-25" + }, "documents": [ { "id": "doc:dec-page:HO-77401-NY:2025-09-01", From 794db3479c59a1763cfd4e55e039347f04dadd08 Mon Sep 17 00:00:00 2001 From: Kushan Patel Date: Thu, 6 Aug 2026 08:11:31 -0700 Subject: [PATCH 2/2] Route CLAUDE.md's gotcha growth to the move PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md is 5,352 words and has grown 5x in a week; the gotchas are 3,139 of them, 59% of the file, in 27 bullets. That is loaded into every session, and phases 4-6 are still ahead of it. Recorded as a task on Phase 3's move rather than as its own PR, with the measurement, because the move rewrites the examples/ gotchas and creates the directory a nested CLAUDE.md belongs in — splitting first means splitting twice. Two halves. Route by directory using nested CLAUDE.md files, which stay auto-loaded: a docs/gotchas.md linked from root is the doc form of "a test that would still pass with its subject deleted", so the rejected option is written down with its reason. And a graduation rule — a gotcha exists because nothing catches the defect, so when a test starts catching it the bullet collapses to a line naming the test. Three already qualify. One landmine found in the writing: the deletion gate removes examples/CLAUDE.md one PR later. Correct for the two gotchas routed there, silent data loss for anything parked there for tidiness. Docs-only: one prose file, no code, command or path changed, so the suite and replay were not re-run. Co-Authored-By: Claude Opus 5 --- docs/m5-plan.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/docs/m5-plan.md b/docs/m5-plan.md index b70cab6..a8bdd0c 100644 --- a/docs/m5-plan.md +++ b/docs/m5-plan.md @@ -599,6 +599,51 @@ proves it. verify commands, every README path reference. `review-0001` is preserved-forever and pins `duly-starter-notice` — it moves intact; `schemaRef` is a name, not a path, so no hash moves. + + **Also in this PR: split CLAUDE.md's gotchas, and give them a lifecycle.** + Measured 2026-08-06 — the file is **5,352 words** (~7k tokens, loaded + every session) and has grown **5× in one week** (1,029 → 5,352). The + gotchas are **3,139 of those words, 59% of the file**, in 27 bullets, the + longest 295 words. Phases 4–6 are still ahead of it. + + *Two things to get right, and the second is the one that stops the + growth.* + + *First, **route by directory, using nested `CLAUDE.md` files** — which + Claude Code loads when work touches that directory. The obvious fix, a + `docs/gotchas.md` linked from the root, is wrong for this repository's own + stated reason: a gotcha in a file nothing auto-loads is a gotcha nobody + reads, which is the doc form of "a test that would still pass with its + subject deleted". The clustering pays for the split on its own: + `demo/` is **5 bullets and 896 words, 28.5% of the gotchas**, relevant to + roughly one session in five; `rulepacks/` is 8 bullets and 503 words and + folds into `rulepacks/README.md`, which CLAUDE.md already mandates before + touching a pack; `assurance/` + `whatif/` is 3 and 277; `examples/` is 2 + and 158. That leaves **9 bullets and 1,305 words in root**, and a file + around 3,500 words instead of 5,352.* + + *What stays root-level is what is genuinely cross-cutting: the + deleted-subject rule, package-vs-repo paths, the `engine` block, + semantics-scoped replay, `schemaRef` inside the fact hash, one entity per + case. Root keeps a one-line pointer per nested file (~30 words) so an + agent planning a change **before** opening a file in that directory still + knows the gotchas exist — the split's one real failure mode.* + + *Second, **a graduation rule**. A gotcha exists because nothing catches + the defect; when something starts catching it, the bullet collapses to one + line naming the check. Several already qualify — the rule-id convention is + enforced by `validate_pack`, the semantics/package re-coupling by + `kernel/tests/test_engine_identity.py`, the demo reload leak by + `demo/tests/demotest_helpers.py` — and each is currently a 110–295-word + narrative explaining a trap that now fails loudly. Add a sixth question to + CLAUDE.md's documentation pass: **did this work turn a gotcha into a + test?** If so, shrink it in the same PR. That is the repo's own principle + — an executable check beats prose — turned on the prose.* + + *Why here rather than its own PR: the move rewrites the `examples/` + gotchas and creates the directory a nested file belongs in. Splitting + first means splitting twice, and the second pass is the one that gets + skipped.* - [ ] **The deletion gate (1 PR).** A CI job: `git rm -r examples/`, run the toolkit suites, run `verify` (expect `verified 0 cases` per Phase 1), **boot the demo and assert the honest empty state** — the surfaces @@ -618,6 +663,12 @@ proves it. - The M4 ontology consolidation (CLAUDE.md's `schemaRef` gotcha) is the template for how to verify a mass path move: targets + fixtures + templates together, `notice-*`/`review-*` proven byte-untouched. +- **`examples/CLAUDE.md` is deleted by the gate**, one PR later. That is + correct for the two gotchas routed there — CP-SAT's nondeterminism and the + pytest-paths carve-out are both about content the adopter removes, and they + should leave with it. It is a trap for anything else: a gotcha placed there + for tidiness disappears silently, and the gate reports success. Route to + `examples/` only what dies with `examples/`. ## 8. Phase 4 — distribution (3–4 PRs)