M5 Phase 3: the demo's review arc becomes content instead of a constant

CHANGELOG.md

@@ -76,7 +76,9 @@ Also: `impact.analyze` gained `pack_overrides`, so a candidate pack that exists
 
 *It reached further than tests, which is what made it worth doing early.* [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) — a **contract artifact**, the thing another language implements duly's digest against — was generated from `golden/` receipts. Self-contained, so it would have survived the deletion intact; but the test proving its two "real" receipts are genuinely committed receipts would have quietly stopped meaning anything, and a spec artifact resting on content an adopter is invited to delete is the same category error one layer up. Rebuilt from `fixtures/`.
 
-*The demo's review arc names a starter, and the naming is invisible because the failure is silent.* `demo/app.py` picks its review scenario by literal id — `notice-ny` — and wraps the ingest in `except Exception: pass`. Point the demo at any other content and the arc simply does not appear: no error, no log line, one fewer scenario in a list nobody counts. Degrading quietly is the right behaviour for a missing *demonstration*, which is exactly why hardcoding *which* demonstration it is went unnoticed — the two failures are indistinguishable from outside. The demo is toolkit, so this is example content compiled into it, and it is the last such dependency the surfaces carry.
+*The demo's review arc is content now, and removing the hardcoded name uncovered what it was hiding.* `demo/app.py` used to pick its review scenario by literal id — `notice-ny` — in four constants, with the ingest wrapped in `except Exception: pass`. Point the demo at other content and the arc simply did not appear: no error, no log line. Degrading quietly is right for a missing *demonstration*, which is exactly why hardcoding *which* one went unnoticed; the two failures are indistinguishable from outside. A scenario now opts in with a `reviewArc` block in its manifest — the attribute to script below the pack's floor, the confidence, the title — the same way it opts into the stub extractor. This repository's arc is unchanged in every field, which was the check that mattered: the point was to remove a name from toolkit code, not to change what the demo shows.
+
+*And the name was hiding a second dependency.* With it gone, a fixture-only deployment still gets no arc — because the arc lives only in the session store, and store-backed ingest needs extraction *targets* that the fixture scenario does not have. One silent failure was standing in front of another. That is the recurring shape of this phase: a hardcoded path does not merely couple two things, it conceals everything downstream of itself.
 
 *A phrasing guard that matches nothing fails quietly, which is worth a fixture.* Writing the fixture pack's money phrasing, the first guard was `when: { value: "0.00" }` — and a money decision's value is an amount and a currency, not a scalar, so it matched nothing. The demo did not complain: it fell through to the unguarded case and rendered a plausible answer. The correct form is `amount: positive`. The fixture keeps the corrected version with a comment naming the mistake, because a pack author will make it and the failure mode is a verdict that reads fine and came from the wrong branch.
 

CLAUDE.md

@@ -85,7 +85,7 @@ Run the full suite, replay, and spec validation before any commit. A change that
 - **`impact.analyze` takes `pack_overrides`; the CLI never passes it.** Keyed by *resolved* pack path, it seeds the pack cache so a candidate that exists only in memory is measured against the corpus slice it governs. Reading the working tree is the point of the command, so the flag is for callers holding a pack that is not (and may never be) a file — an impact number an author sees before writing the file is the only one that can change their mind.
 - **Scripted confidences need the stub pin.** If a scenario depends on an exact confidence value, set `"demoExtractor": "stub"` in its `scenario.json` — Docling emits its own measured confidence and silently overrides the scripted one. No test warns about this at authoring time.
 - **`ReviewQueue.resolve` refuses a `low_confidence` correction that supersedes nothing, and cannot fix it for you.** Resolving such an item is a ruling on the one fact it abstained over, so the correction must carry `supersedes` ([spec/compatibility.md](spec/compatibility.md) C6) — otherwise the below-floor fact stays live and every future receipt carries a `low_confidence` entry for an attribute the decision *used*, which contradicts the fact spec's own definition of `abstentions`. The obvious fix — have the queue stamp the field in from the entry — is unavailable and instructive: a correction arrives content-addressed, so writing into it changes its hash and its identity, and the queue would hand the store a document its author never sealed. It names the required fact id and refuses. Three carve-outs, all deliberate: `conflict` items are untouched (their entries name several facts), `FactStore.ingest` still takes an independent human fact that supersedes nothing (a value from a phone call is not a ruling on an extraction), and `duly_review.golden`'s converter stays permissive because it is handed items by callers it does not control.
-- **The demo's review arc is derived from example content by name, and fails silently when it is absent.** `demo/app.py` declares `REVIEW_SOURCE_SCENARIO = "notice-ny"` (plus a matching id, case id and title), and `_ingest_review_case` is wrapped in `except Exception: pass` — so pointing `DULY_DEMO_CONTENT` anywhere else makes the arc quietly not appear rather than complain. Degrading quietly is right for a missing *demonstration*; naming which one inside toolkit code is the defect, and the silence is what hides it. Phase 3 owns the fix (derive the source from whichever scenario carries a below-floor extraction, preferring `notice-ny` when present). Until then, any demo suite you point at fixture content will lose its review-arc coverage without a single failure.
+- **The demo's review arc is content, and it is store-only.** A scenario opts in with a `reviewArc` block in its manifest (attribute to script below the floor, confidence, title, case id) — `demo/app.py` names no scenario, having previously hardcoded `notice-ny` in four constants. Two things still bite. The arc exists **only** in the session store, so a deployment whose scenarios cannot be store-ingested has none — and store ingest needs extraction *targets* under `starters/tools/targets`, which is a second dependency the hardcoded name used to hide. And `_ingest_review_case` is wrapped in `except Exception: pass`, so every one of those failures is silent by design: right for a missing demonstration, and the reason a missing *arc* tells you nothing about why.
 - **One entity per `entityType` per case; one live fact per attribute.** Per-document decisions mean one case per document, or the document type as an attribute of a single entity. Two live facts on one attribute is a conflict (lone human outranks; anything else abstains).
 - **Replay is scoped to a semantics version, and "the digest" is not the receipt hash.** Two things the freeze ([spec/compatibility.md](spec/compatibility.md)) made load-bearing. First: every replay path calls `duly_kernel.semantics.check_replayable` *before* re-adjudicating, and a receipt whose `engine.version` is not in `IMPLEMENTED` is refused rather than replayed. The failure that guard exists for is silent — a kernel implementing V2 replaying a V1 receipt and **passing**, on the subset of cases where the two semantics happen to agree — so never "fix" a refusal by widening `IMPLEMENTED`: an entry there is a claim that this kernel reproduces that version byte-for-byte, substantiated only by corpus cases at that version. Second: `decision_digest()` is a pure function over a receipt's *determinant* fields and is never stored in any document (the receipt schema is closed; a digest inside the body it digests is exactly the extension point C2 refuses). It excludes `caseId`, `rulePack.gitCommit`/`url`, `engine.kernel` and `engine.backend` — everything identifying the run rather than the adjudication — which is what makes it, and not byte equality, the definition of two evaluation backends agreeing. Changing the determinant set is a breaking change to a published contract, not a refactor; [`spec/decision-digest-vectors.json`](spec/decision-digest-vectors.json) and a corpus-aggregate digest both fail on it.
 - **The whole `engine` block is inside the receipt hash, and `engine.version` is NOT a package version.** Two consequences, and the second one will bite during release work. `engine.backend`: a second evaluation backend cannot produce byte-identical receipts by construction; cross-backend equivalence is an open M5 spec decision, so don't design around an assumption either way. `engine.version`: it is the version of the kernel's **decision semantics**, pinned as `receipt.SEMANTICS_VERSION` and deliberately decoupled from `duly_kernel.__version__` and from the `duly` distribution version. Three scopes, nested one way only: the **distribution** moves on any release of anything in the wheel, `duly_kernel.__version__` moves when the kernel's *code* changes, and `SEMANTICS_VERSION` moves when the kernel's *meaning* changes. A semantics change implies a code change implies a release; never the reverse, because a release can be a demo fix with the kernel byte-identical. All three read `0.0.1` today, which is a coincidence and is exactly what makes this easy to break silently. It used to be `__version__`, which made the corpus a hostage to the release cadence — publishing under any new number would have invalidated all 351 receipts without a rule, a fact, or a decision changing. **Never bump `SEMANTICS_VERSION` as part of a release.** It moves only when what the kernel means changes, and moving it rewrites every committed receipt. The value stays `0.0.1` for the same reason `NY-NR-45` keeps its name: it is sealed into artifacts that are not editable, so it is a handle, not a claim — and what a package version is for has a correctable home in the wheel metadata, while this does not. [kernel/tests/test_engine_identity.py](kernel/tests/test_engine_identity.py) fails on the re-coupling edit, which is exactly the tidying-up someone does while packaging. [docs/release-process.md](docs/release-process.md) is the decision procedure for which of the four scopes moves for a given change.

demo/app.py

@@ -237,10 +237,19 @@ def _fmt_score(value: Any) -> str:
 # the server restarts, and the fixture fallback path keeps working without a
 # store at all.
 
-REVIEW_SCENARIO_ID = "notice-ny-review"
-REVIEW_SOURCE_SCENARIO = "notice-ny"
-REVIEW_CASE_ID = "case:policy:HO-77401-NY:review-demo"
-REVIEW_SCENARIO_TITLE = "NY nonrenewal — review arc (low-confidence extraction)"
+# The review arc is **content, not code**. A scenario opts in by carrying a
+# `reviewArc` block in its manifest — which attribute to script below the
+# floor, at what confidence, and what to call the result — exactly as it opts
+# into the stub extractor with `demoExtractor`.
+#
+# This used to be four constants naming `notice-ny`: a teaching scenario's id
+# compiled into the toolkit. Pointing `DULY_DEMO_CONTENT` anywhere else made
+# the arc silently not appear, because the ingest is wrapped in
+# `except Exception: pass` — and "this deployment has no review scenario" and
+# "it has one but we only look for notice-ny" are indistinguishable from
+# outside. That is the failure mode that hid the hardcoding.
+REVIEW_ID_SUFFIX = "-review"
+REVIEW_RUN_ID_TEMPLATE = "run:{scenario}:review-demo"
 
 # Scenario domains group the picker by regulated vertical. The slug comes from
 # the starter manifest's optional `domain` field — presentation metadata only,
@@ -258,12 +267,29 @@ def _domain_fields(slug: str | None) -> dict[str, str]:
     slug = (slug or DEFAULT_DOMAIN).strip().lower() or DEFAULT_DOMAIN
     label = DOMAIN_LABELS.get(slug, slug.replace("-", " ").title())
     return {"domain": slug, "domainLabel": label}
-REVIEW_MAILED_ATTRIBUTE = "nc:noticeMailedDate"
-# Mirrors golden/cases/review-0001: 0.62 sits below the notice pack 2026.3.0
-# per-attribute floor of 0.9, so the mailed date abstains with low_confidence.
-REVIEW_MAILED_CONFIDENCE = {"score": 0.62, "method": "platt"}
-REVIEW_RUN_ID = "run:review-demo:001"
-REVIEW_DEFAULT_AS_OF = "2026-07-25"
+
+
+def _review_spec(manifest: dict[str, Any]) -> dict[str, Any] | None:
+    """A scenario's `reviewArc` block, validated enough to build from.
+
+    `attribute` and `confidence` are required — without them there is nothing
+    to script below the floor, and an arc that abstains over nothing is not a
+    demonstration. Everything else has a defensible default.
+    """
+    block = manifest.get("reviewArc")
+    if not isinstance(block, dict):
+        return None
+    attribute = block.get("attribute")
+    confidence = block.get("confidence")
+    if not isinstance(attribute, str) or not isinstance(confidence, dict):
+        return None
+    return {
+        "attribute": attribute,
+        "confidence": dict(confidence),
+        "title": block.get("title") or f"{manifest.get('title', manifest['id'])} — review arc",
+        "caseId": block.get("caseId") or f"{manifest['caseId']}:review-demo",
+        "defaultAsOf": block.get("defaultAsOf"),
+    }
 
 SESSION_NOTE = (
     "Corrections live in this demo process only — the in-memory fact store "
@@ -486,6 +512,29 @@ def _ingest_starter_case(
     return meta
 
 
+def _find_review_source() -> tuple[Path, dict[str, Any], dict[str, Any]] | None:
+    """The first scenario whose manifest opts into the review arc.
+
+    Sorted, so a deployment with two of them gets a stable answer rather than
+    a filesystem-order one — the same reason every other discovery here is
+    sorted.
+    """
+    if not STARTERS_DIR.is_dir():
+        return None
+    for scenario_dir in sorted(STARTERS_DIR.iterdir()):
+        manifest_path = scenario_dir / "scenario.json"
+        if not scenario_dir.is_dir() or not manifest_path.exists():
+            continue
+        try:
+            manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+        except (OSError, json.JSONDecodeError):
+            continue
+        spec = _review_spec(manifest)
+        if spec is not None:
+            return scenario_dir, manifest, spec
+    return None
+
+
 def _ingest_review_case(
     runtime: _DemoRuntime,
     targets_by_doc: dict[str, dict[str, Any]],
@@ -501,13 +550,16 @@ def _ingest_review_case(
     confidence would clear the floor and defeat the demonstration. The UI
     labels the extractor, so the scripting is visible, not hidden.
     """
-    source_dir = STARTERS_DIR / REVIEW_SOURCE_SCENARIO
-    manifest_path = source_dir / "scenario.json"
-    if not manifest_path.exists():
+    found = _find_review_source()
+    if found is None:
         return
-    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    source_dir, manifest, spec_block = found
+    scenario_id = f'{manifest["id"]}{REVIEW_ID_SUFFIX}'
+    review_case_id = spec_block["caseId"]
     meta: dict[str, Any] = {
-        "caseId": REVIEW_CASE_ID,
+        "caseId": review_case_id,
+        "reviewSpec": spec_block,
+        "sourceId": manifest["id"],
         "packPath": manifest["rulePack"],
         "renditions": {},
         "extraction": None,
@@ -524,19 +576,19 @@ def _ingest_review_case(
         if spec is None or pdf_path is None or rend_path is None:
             return
         spec = copy.deepcopy(spec)
-        spec["caseId"] = REVIEW_CASE_ID
-        spec["runId"] = REVIEW_RUN_ID
+        spec["caseId"] = review_case_id
+        spec["runId"] = REVIEW_RUN_ID_TEMPLATE.format(scenario=manifest["id"])
         for target in spec.get("facts", []):
-            if target.get("attribute") == REVIEW_MAILED_ATTRIBUTE:
-                target["confidence"] = dict(REVIEW_MAILED_CONFIDENCE)
+            if target.get("attribute") == spec_block["attribute"]:
+                target["confidence"] = dict(spec_block["confidence"])
         document = source_document_cls.from_bytes(doc["id"], pdf_path.read_bytes())
         rendition_text = rend_path.read_text(encoding="utf-8")
         result = stub_cls(rendition_text).extract(document, spec)
         ingest_envelope(runtime.store, result.envelope, result.facts, result.rendition.text)
         meta["renditions"][doc["id"]] = result.rendition.text
         meta["extraction"] = _extraction_meta(result, "stub")
     if meta["renditions"]:
-        runtime.cases[REVIEW_SCENARIO_ID] = meta
+        runtime.cases[scenario_id] = meta
 
 
 # ---------------------------------------------------------------------------
@@ -738,10 +790,14 @@ def _build_review_scenario(runtime: _DemoRuntime) -> dict[str, Any] | None:
     """The review-arc scenario, synthesized from the runtime's ingested case
     (never from starters/ — the below-floor fact set exists only in the
     session store)."""
-    meta = runtime.cases.get(REVIEW_SCENARIO_ID)
-    if meta is None:
+    scenario_id = next(
+        (k for k, v in runtime.cases.items() if v.get("reviewArc")), None
+    )
+    if scenario_id is None:
         return None
-    manifest_path = STARTERS_DIR / REVIEW_SOURCE_SCENARIO / "scenario.json"
+    meta = runtime.cases[scenario_id]
+    spec_block = meta["reviewSpec"]
+    manifest_path = STARTERS_DIR / meta["sourceId"] / "scenario.json"
     try:
         manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
         pack = yaml.safe_load((CONTENT.root / meta["packPath"]).read_text(encoding="utf-8"))
@@ -756,15 +812,15 @@ def _build_review_scenario(runtime: _DemoRuntime) -> dict[str, Any] | None:
         for doc_id, text in meta["renditions"].items()
     }
     return {
-        "id": REVIEW_SCENARIO_ID,
-        "title": REVIEW_SCENARIO_TITLE,
-        **_domain_fields("insurance"),
-        "caseId": REVIEW_CASE_ID,
+        "id": scenario_id,
+        "title": spec_block["title"],
+        **_domain_fields(manifest.get("domain")),
+        "caseId": meta["caseId"],
         "documents": documents,
-        "facts": runtime.store.as_of(REVIEW_CASE_ID, knowledge=_now_knowledge()),
+        "facts": runtime.store.as_of(meta["caseId"], knowledge=_now_knowledge()),
         "pack": pack,
         "questions": questions,
-        "defaultAsOf": REVIEW_DEFAULT_AS_OF,
+        "defaultAsOf": spec_block.get("defaultAsOf") or manifest.get("defaultAsOf"),
         "source": "review-demo",
         "extraction": meta.get("extraction"),
         "review": {"available": True, "note": None},

demo/tests/test_review_arc.py

@@ -30,7 +30,11 @@
 from fastapi.testclient import TestClient  # noqa: E402
 
 import demo.app as demo_app  # noqa: E402
-from demo.app import REVIEW_SCENARIO_ID, app  # noqa: E402
+from demo.app import REVIEW_ID_SUFFIX, app  # noqa: E402
+
+# Derived, not imported: the arc's id follows whichever scenario opts into it,
+# so a fixture deployment gets its own rather than this repository's.
+REVIEW_SCENARIO_ID = f"notice-ny{REVIEW_ID_SUFFIX}"
 
 client = TestClient(app)