Declare Chronicle consumer_fact.v5 and refuse feeds on undeclared row contracts

changelog.d/1509-chronicle-consumer-fact-v5.added.md

@@ -0,0 +1,9 @@
+Declare Chronicle's `chronicle.consumer_fact.v5` row contract, the one Chronicle `main` writes since PolicyEngine/chronicle#377 (`44eebe037`), and refuse a Chronicle feed whose pin or rows name a row contract Microcosm does not declare (microcosm#1509).
+
+- **The declaration.** `microcosm.build.chronicle_epoch` declares v5 in the chronicle epoch beside `ledger.consumer_fact.v1` and `chronicle.consumer_fact.v2` to v4, matching Chronicle's own `CONSUMER_FACT_ROW_CONTRACTS`. Before this change nothing outside the tests read that list, so a v5 feed would already have loaded.
+- **Pins.** The UK and US pin loaders, `chronicle_feed_identity` (used by reference factors, money normalization and the transport composer) and `validate_chronicle_feed_pin` refuse a pin whose `consumer_fact_schema_versions` names an undeclared row contract. The message names it and says to declare it in `chronicle_epoch` first; an id outside Chronicle's namespaces, such as `arch.consumer_fact.v1`, is named as not a Chronicle row contract, which `chronicle_epoch` cannot declare.
+- **Rows.** `validate_chronicle_feed_pin` compares the rows' `schema_version` values with the pin before it checks them, so the transport kernels and the NZ and Belgian generators, which use it, cover the rows. Code that holds a feed to the committed UK or US pin by digest now also checks the rows it reads (`validate_pinned_row_contracts`): rows on an undeclared row contract are refused, and rows of the committed feed must carry exactly the pin's list, which a re-pin types by hand. That covers the UK full and national builds, the UK vendoring tool, the US fiscal-refresh release tool and the US target-parity generator (`require_uk_chronicle_feed_row_contracts`, `require_us_chronicle_feed_row_contracts`). A feed admitted by `--allow-unpinned-feed` is held to the declared row contracts only. The UK release certifier holds its feed to the operator's digests rather than the pin's, so it applies the exact rule only when it is given the committed feed, and the declared-only rule otherwise. The UK reference generators, the compile-parity tool and the DESNZ constituency builder, which read a subset of the pinned feed or only some of its rows, require each row's contract to be one the pin lists.
+- **Loads without a pin.** A feed loaded by path for diagnostics and experiments, or a bare feed with no manifest that no reviewed pin covers, still carries its per-row ids unchecked; the July 2026 US fiscal-refresh feed, whose rows carry `arch.consumer_fact.v1`, loads that way. So do the readers that hold their feed to no reviewed pin: the US release preflight and ACS local tools, the scoring and diagnostic tools, and the UK builders that read the feed by path into committed resources (the high-value home stock, the BRMA rent quantiles and the Scotland BRMA overlap) and the childcare take-up fit tools, which take an operator-supplied digest. Their outputs are used beside the pinned feed, whose rows the builds check.
+- **The new field.** v5 adds an optional `package_dimension_value_labels`: a source package's own wording of a dimension value where it differs from the label the row exports. The loader keeps it on the parsed rows, as it keeps every published field, and Microcosm never takes a label from it: each value keeps the one label Chronicle gives it in `dimension_value_labels` and `layout.groupby_value_label`.
+- **Its shape.** When a target with a calibration hierarchy (a provider and category seed) is compiled, each selected fact's field must be an object of dimension id to an object of value id to non-empty text, the shape the v5 schema requires. Any other shape raises `LedgerHierarchyMetadataError`, the error a missing Chronicle label raises, and callers treat it the same way: the UK compilers report the reference as unsupported, which stops the build year and is recorded for validation periods. This check walks every entry of the field, where the check of `dimension_value_labels` reads only the selected entries.
+- **No feed pin moves.** The UK, US, NZ and Belgian feeds stay on their current Chronicle commits until their own re-pins.

docs/calibration-target-hierarchy.md

@@ -89,6 +89,14 @@ metadata, not hints for constructing display text.
   exactly one non-empty `dimension_labels` entry and one non-empty
   `dimension_value_labels` entry for each selected dimension and value.
   Missing or conflicting labels fail compilation.
+- A `chronicle.consumer_fact.v5` row (PolicyEngine/chronicle#377) may also
+  carry `package_dimension_value_labels`: a source package's own wording of a
+  value, present only where it differs from the label the row exports.
+  Microcosm never takes a label from it, so each value keeps the one label
+  Chronicle gives it in `dimension_value_labels` and
+  `layout.groupby_value_label`. When the field is present on a selected fact
+  it must map each dimension id to an object of value id to non-empty text,
+  as the v5 schema requires; any other shape fails compilation.
 - A layout dimension and a fact dimension merge only when their identifiers
   are exactly equal. Similar labels do not establish identity.
 - A multi-fact aggregate inherits only dimensions present with the same value

docs/uk-chronicle-feed-repin.md

@@ -24,7 +24,26 @@ restate it: `tests/engine_free/uk/test_uk_chronicle_feed.py` (the pin and the
 consumer-fact schema digest) and `tests/engine_free/uk/test_uk_full_targets.py`
 (`test_current_national_and_local_pins_share_one_reviewed_identity`). A schema
 digest moves for every country whose feed is built after the Chronicle change,
-so the NZ pin (`nz/chronicle_feed.json`) moves at its own next re-pin. Then
+so the NZ pin (`nz/chronicle_feed.json`) moves at its own next re-pin. Set the
+pin's `consumer_fact_schema_versions` from the manifest, not by hand from the
+last pin. A feed whose rows carry a newer per-row `schema_version` needs that
+row contract declared in `microcosm.build.chronicle_epoch` first
+(microcosm#1509):
+
+- the UK pin loader, and `chronicle_feed_identity` and
+  `validate_chronicle_feed_pin` where the factor and transport code use them,
+  refuse a pin naming an undeclared row contract;
+- the UK code that reads the feed's rows refuses rows on an undeclared row
+  contract, or rows whose contracts differ from the pin's list: the full and
+  national builds and the vendoring tool on the committed artifact (exactly
+  the pin's list), the release certifier likewise when it is given the
+  committed feed (it holds the feed to the operator's digests), and the
+  reference generators, the compile-parity tool and the DESNZ constituency
+  builder on a subset or on the rows they parse (each a contract the pin
+  lists).
+
+Microcosm declares `ledger.consumer_fact.v1` and `chronicle.consumer_fact.v2`
+to `v5`; the v5 schema's digest is `acd33a09…`. Then
 regenerate. Vendor
 the per-concern fact resources (step 5) before the national generator: the
 generator refuses to compile against a vendored resource whose feed identity

docs/us-chronicle-feed-repin.md

@@ -13,6 +13,32 @@ any other feed, and `test_us_chronicle_feed.py` fails if the parity resources,
 the generator and the pin disagree, or if the scope file changes without a
 new pin.
 
+## A re-pin onto a newer row contract
+
+The US feed is on `chronicle.consumer_fact.v3`; Chronicle `main` writes
+`chronicle.consumer_fact.v5` since PolicyEngine/chronicle#377. The US pin
+loader refuses a pin whose `consumer_fact_schema_versions` names a row contract
+that `microcosm.build.chronicle_epoch` does not declare (microcosm#1509). The
+pin describes a bare feed, so the full feed-pin check cannot run on it;
+instead the fiscal-refresh release tool and `tools/build_us_target_parity_manifest.py`,
+after their digest checks, refuse rows on an undeclared row contract and, on
+the committed feed, rows whose contracts are not exactly the pin's list
+(`require_us_chronicle_feed_row_contracts`). So set the list from the rebuilt
+feed's rows at a re-pin: `tools/build_us_chronicle_feed.py` does not write it,
+and the parity generator, which the rebuild runs, refuses a stale list. v5 is
+declared, and its schema digest is `acd33a09…`.
+
+v5 rows may carry `package_dimension_value_labels`, a package's own wording of
+a value where it differs from Chronicle's label. The congressional-district
+vintage translation (`us_runtime/congressional_district_vintage.py`:
+`_translated_fact_shell` and `_state_total_proxy_source_district_fact`) copies
+each source row whole and rewrites only the geography, the layout and the
+group-by entry of `dimension_value_labels`. So after a re-pin onto v5, a
+derived district row keeps any package wording keyed by the source row's value
+ids (the 118th-Congress district, or the state row's value), not the derived
+row's. No code reads the field, so no label, value or refusal changes; v4's
+`geography.publisher_name` is copied the same way.
+
 ## The 2026-11-03 re-pin: Federal Student Aid loan portfolio tables (microcosm#1074)
 
 microcosm#1357 binds Federal Student Aid's loan portfolio tables as
@@ -84,7 +110,9 @@ and `consumer_fact_schema_sha256` stays `bdb51e2a…`: Chronicle's
 `policyengine_chronicle/schemas/consumer_fact.v3.schema.json`, the file
 whose bytes that field hashes (`policyengine_chronicle/schema.py:59-70`), is
 identical at `a9026cc` and `41b2fb2b`. The US loader carries a row's stamp
-without gating on it (`ledger_artifact.py`, `_load_fact_rows`).
+without checking it (`ledger_artifact.py`, `_load_fact_rows`); since
+microcosm#1509 the release tool and the target-parity tool check the stamps
+against the pin (see "A re-pin onto a newer row contract").
 
 The B4923C anchor test follows the committed re-pin reports to the pin
 (`test_us_esi_premiums.py`). A report may now add rows when it names a

packages/microcosm-build/src/microcosm/build/chronicle_epoch.py

@@ -67,6 +67,7 @@
     "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION",
     "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V3",
     "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V4",
+    "CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V5",
     "CHRONICLE_EPOCH",
     "CHRONICLE_FACT_KEY_DOMAINS",
     "CHRONICLE_NAMESPACES",
@@ -87,6 +88,7 @@
     "MICROCOSM_CONSUMER_ARTIFACT_SCHEMA_VERSION",
     "PUBLISHED_CONSUMER_ARTIFACT_SCHEMA_VERSION",
     "UNDECLARED",
+    "UndeclaredConsumerFactSchemaVersionError",
     "consumer_artifact_schema_epoch",
     "consumer_fact_schema_epoch",
     "declared_identity",
@@ -100,6 +102,7 @@
     "is_accepted_consumer_fact_schema_version",
     "is_chronicle_fact_key",
     "parse_fact_key",
+    "require_accepted_consumer_fact_schema_versions",
     "row_fact_key_epochs",
 ]
 
@@ -206,17 +209,25 @@ def _declare(identity: str, *, epoch: str, kind: str) -> DeclaredIdentity:
 #: The per-row consumer-fact ``schema_version`` Chronicle stamped before
 #: chronicle#261, its chronicle-era successor (the v2 row adds the dimension
 #: and value labels), the v3 row chronicle#266 (PR #267) added (the v2 row
-#: plus the publisher's ``geography.name``), and the v4 row chronicle#281
-#: (PR #282) emits today: the v3 row with ``geography.name`` resolved to one
-#: spelling per identifier where Chronicle's register carries it, the
-#: publisher's own text moving to an optional ``geography.publisher_name``.
-#: Every declared row contract keeps loading; the name and the labels are
-#: display metadata and never enter a fact key, so the keys of a v4 row are
-#: the v2 bytes.
+#: plus the publisher's ``geography.name``), the v4 row chronicle#281 (PR
+#: #282) added (the v3 row with ``geography.name`` resolved to one spelling
+#: per identifier where Chronicle's register carries it, the publisher's own
+#: text moving to an optional ``geography.publisher_name``), and the v5 row
+#: chronicle#376 (PR #377, merged as 44eebe037) emits today: the v4 row with
+#: ``dimension_value_labels`` and ``layout.groupby_value_label`` carrying the
+#: one label Chronicle's register gives each dimension value, and an optional
+#: ``package_dimension_value_labels`` keeping a source package's own wording
+#: where it differs. Microcosm never takes a label from that field
+#: (:mod:`microcosm.build.ledger_targets` checks only its shape). Every
+#: declared row contract keeps loading; the names and the labels are display
+#: metadata and never enter a fact key, so the keys of a v5 row are the v2
+#: bytes. Chronicle's own list is ``CONSUMER_FACT_ROW_CONTRACTS`` in
+#: ``chronicle/epoch.py``.
 LEDGER_CONSUMER_FACT_SCHEMA_VERSION = "ledger.consumer_fact.v1"
 CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION = "chronicle.consumer_fact.v2"
 CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V3 = "chronicle.consumer_fact.v3"
 CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V4 = "chronicle.consumer_fact.v4"
+CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V5 = "chronicle.consumer_fact.v5"
 
 #: Fact-key domains, ledger-era spelling beside its chronicle-era successor.
 #: The ledger column is what Chronicle's ``main`` hashes with today; the
@@ -282,6 +293,11 @@ def _declare(identity: str, *, epoch: str, kind: str) -> DeclaredIdentity:
         epoch=CHRONICLE_EPOCH,
         kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND,
     ),
+    _declare(
+        CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V5,
+        epoch=CHRONICLE_EPOCH,
+        kind=CONSUMER_FACT_SCHEMA_IDENTITY_KIND,
+    ),
     *(
         _declare(identity, epoch=epoch, kind=FACT_KEY_IDENTITY_KIND)
         for ledger_domain, chronicle_domain in _FACT_KEY_DOMAIN_DECLARATIONS
@@ -569,6 +585,69 @@ def describe_accepted_consumer_fact_schema_versions() -> str:
     return _describe(CONSUMER_FACT_SCHEMA_IDENTITY_KIND)
 
 
+class UndeclaredConsumerFactSchemaVersionError(ValueError):
+    """A Chronicle feed names a per-row row contract this module does not declare."""
+
+
+def require_accepted_consumer_fact_schema_versions(
+    versions: object, *, subject: str
+) -> None:
+    """Refuse a row-contract list unless every entry is declared here.
+
+    ``versions`` is a feed's ``consumer_fact_schema_versions``, or the
+    ``schema_version`` values its rows carry. Feed pins, the identities
+    restating them, the full feed-pin check, and the UK and US builds' checks
+    of the rows they compile call this, so a feed on a row contract Microcosm
+    has not been told about is refused until :data:`DECLARED_IDENTITIES`
+    declares it. An id outside Chronicle's namespaces is not a Chronicle row
+    contract at all, and :func:`_declare` cannot declare it, so the message
+    says so instead of advising a declaration.
+    """
+    if not isinstance(versions, (list, tuple)):
+        raise ValueError(
+            f"{subject}: consumer_fact_schema_versions must be a list of row "
+            f"contracts, got {type(versions).__name__}."
+        )
+    undeclared = [
+        version
+        for version in versions
+        if not is_accepted_consumer_fact_schema_version(version)
+    ]
+    if not undeclared:
+        return
+    declarable = [
+        version
+        for version in undeclared
+        if isinstance(version, str)
+        and version.partition(".")[0] in CHRONICLE_NAMESPACES
+    ]
+    foreign = [version for version in undeclared if version not in declarable]
+    problems = []
+    if declarable:
+        problems.append(
+            f"undeclared Chronicle row contract(s) {declarable!r}: declare "
+            f"{declarable!r} in microcosm.build.chronicle_epoch "
+            "(DECLARED_IDENTITIES) before pinning a feed on it"
+        )
+    if foreign:
+        single = len(foreign) == 1
+        problems.append(
+            f"{foreign!r} "
+            + (
+                "is not a Chronicle row contract: its namespace is"
+                if single
+                else "are not Chronicle row contracts: their namespaces are"
+            )
+            + f" not among {sorted(CHRONICLE_NAMESPACES)!r}, so "
+            "microcosm.build.chronicle_epoch cannot declare "
+            + ("it" if single else "them")
+        )
+    raise UndeclaredConsumerFactSchemaVersionError(
+        f"{subject}: " + "; ".join(problems) + ". Declared row contracts: "
+        f"{describe_accepted_consumer_fact_schema_versions()}."
+    )
+
+
 def _describe(kind: str) -> str:
     return ", ".join(repr(identity) for identity in _identities_of(kind))