Include geography_constants in the target-compilation cache keys

changelog.d/1492-target-compile-keys-geography-constants.fixed.md

@@ -0,0 +1 @@
+The graph kernels that compile targets now include `microcosm.calibrate.geography_constants` in their implementation hashes, so an edit to a geography label recompiles the targets instead of reusing a stored surface that carries the old label (microcosm#1492). `ledger_targets` labels a target's geography from `UK_GEOGRAPHY_ID_TO_LABEL` when the Chronicle facts give no name, and `uk_runtime/target_weights` reads `UK_WIDE_GEOGRAPHY_IDS` for the UK loss weights, both through `from … import`, which `source_hash` does not follow. The module joins the transport compile modules (hashed by `targets.compile@1`, `takeup.compare@1`, `takeup.gap@1`, `transport.create@1`/`@2` and `transport.quantile_map@1`/`@2`), the UK target compile modules (hashed by the full graph's target, selection, measures and problem kernels and the national graph's target and problem kernels) and the UK problem kernel's loss-weight hash. The UK list also gains `microcosm.calibrate.registry`, `target` and `hierarchy`, which the transport list already hashed. The UK target and problem kernels also now hash code they ran unhashed: `calibrate.target_selection` (full graph), `calibrate.matrix` and `calibrate.artifacts`, `calibrate.solve` (the full problem's loss scales only; `calibrate.adam@1`, whose hash the UK calibration kernels embed, already hashed it), and `uk_runtime.weighted_integrity`, whose exclusion clock `full_targets` reads. In national builds an edit to `weighted_integrity` now re-runs the national targets, problem and solve; before, only the national validation node after the solve hashed it. `_ledger_provenance` and `_validate_band_edge_registry`, which `full_targets` runs, move from `uk_runtime.calibration_run` into a small `uk_runtime.ledger_surface` that the UK list names, and `calibration_run` re-exports them. The rest of `calibration_run` stays outside the target kernels' hashes: in builds that construct the spine in the graph, an edit to it re-runs no target compilation and no solve, and in builds from a spine checkpoint it re-keys the bound spine node and everything after it, as before. The transport compile list and its 18 exclusions move from `transport/target_kernels.py`, which 13 kernels hash, to `transport/compile_modules.py`, which none does, as the UK list already sat apart. Two checks now pin the hashes: a shared list's imports must be listed or excluded, reading every `microcosm.*` import rather than only `microcosm.build` ones, and the walk from every module a compile kernel hashes, its own module included, must meet only hashed modules, the list's exclusions or the imports no review has classified yet, recorded per group of kernels sharing one hash and tracked by microcosm#1497 (66 modules for the UK, 35 for transport), a record that can only shrink. Node keys move once: the target compilation of every transport country and everything after it; the UK target, selection, measures, problem and national target and problem nodes and everything after them, including the UK solve; because `transport/target_kernels.py` itself changes, `calibration.prepare@1` nodes, `gates.battery@1` nodes at a phase before compilation, and `takeup.assign@1` nodes given no compiled surface; and, because `calibration_run.py` changes, the bound spine checkpoint node (`uk.full.bound_spine@1`) and the population nodes after it.

packages/microcosm-build/src/microcosm/build/transport/compile_modules.py

@@ -0,0 +1,106 @@
+"""The modules whose source decides a transport target surface.
+
+``targets.compile@1`` and every other transport kernel built on target
+compilation hash :func:`transport_compile_modules` (through
+``target_kernels._COMPILE_MODULES``). ``source_hash`` hashes the files it is
+given and does not follow imports, so the tuple has to name every module the
+compile reads; ``ledger_targets`` labels a target's geography from
+``microcosm.calibrate.geography_constants``, which no compile kernel hashed
+before microcosm#1492.
+
+The list lives here, in a module no kernel hashes, as the UK's lives in
+``uk_runtime.target_compile_modules``: editing a name or a reason re-keys
+only the kernels whose hashed modules the edit adds or removes, not every
+kernel that hashes ``target_kernels``.
+
+``EXCLUDED_DIRECT_IMPORTS`` names every first-party (``microcosm.*``) module
+a listed module imports but the tuple leaves out, with the reason. A test
+pins that the two together cover every direct import of every listed module,
+so a new import must be listed or excluded. The exclusions cover only the
+listed modules' own imports; a second test walks every module each compile
+kernel hashes, its own module included, and records the imports no review
+has classified yet (microcosm#1497).
+"""
+
+from __future__ import annotations
+
+import importlib
+from collections.abc import Mapping
+from types import MappingProxyType, ModuleType
+
+TRANSPORT_COMPILE_MODULE_NAMES: tuple[str, ...] = (
+    "microcosm.build.transport.graph_inputs",
+    "microcosm.build.transport.artifact_types",
+    "microcosm.build.country_spec",
+    "microcosm.build.ledger_targets",
+    "microcosm.build.ledger_artifact",
+    "microcosm.build.chronicle_epoch",
+    "microcosm.build.reference_factors",
+    "microcosm.build.reference_profiles",
+    "microcosm.calibrate.registry",
+    "microcosm.calibrate.target",
+    "microcosm.calibrate.hierarchy",
+    "microcosm.calibrate.geography_constants",
+    "microcosm.build.transport.reference_selection",
+)
+
+_COUNTRY_SPEC_LOADER = (
+    "country_spec is listed for the reference validators target compilation "
+    "calls, _validate_target_references and _carried_value_keys; it imports "
+    "this module for its country-spec loader and stage plans, which those "
+    "validators do not use"
+)
+_GRAPH_RUNTIME = (
+    "the graph runtime (kernel protocol, declarations and the canonical JSON "
+    "encoder node keys are computed with); no kernel in the repository "
+    "hashes a microcosm.graph module"
+)
+
+EXCLUDED_DIRECT_IMPORTS: Mapping[str, str] = MappingProxyType(
+    {
+        "microcosm.build.plan": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.source_manifest": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.trace": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.canonical": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.engine_abi": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.loader": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.model": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.schemas": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.take_up_semantics": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.spec_engine.yaml12": _COUNTRY_SPEC_LOADER,
+        "microcosm.build.reference_intervals": (
+            "reference_profiles reaches it only when given an interval "
+            "document; compile_target_surface calls validate_reference_resolution "
+            "without one. calibration.prepare@1 and targets.problem@1, which "
+            "apply interval tables, hash it"
+        ),
+        "microcosm.calibrate.artifacts": (
+            "artifact_types imports PROBLEM_TYPE to declare targets.problem@1's "
+            "output; target compilation checks only its own surface output"
+        ),
+        "microcosm.diagnostics.schema": (
+            "artifact_types imports the diagnostics schema version to declare "
+            "the diagnostics output type; target compilation emits no diagnostics"
+        ),
+        "microcosm.frame.bundle": (
+            "the population container: graph_inputs.context_frame builds one, "
+            "calibrate.target measures a target on one and country_spec's stage "
+            "plans transform one; target compilation reads facts, not a population"
+        ),
+        "microcosm.frame.schema": (
+            "graph_inputs.context_frame's entity schema; target compilation "
+            "reads no population"
+        ),
+        "microcosm.graph.canonical": _GRAPH_RUNTIME,
+        "microcosm.graph.decl": _GRAPH_RUNTIME,
+        "microcosm.graph.kernel": _GRAPH_RUNTIME,
+    }
+)
+
+
+def transport_compile_modules() -> tuple[ModuleType, ...]:
+    """The modules of :data:`TRANSPORT_COMPILE_MODULE_NAMES`, imported."""
+
+    return tuple(
+        importlib.import_module(name) for name in TRANSPORT_COMPILE_MODULE_NAMES
+    )

packages/microcosm-build/src/microcosm/build/transport/target_kernels.py

@@ -50,10 +50,7 @@
 
 import microcosm.build.calibration_columns as calibration_columns_module
 import microcosm.build.calibration_predicates as calibration_predicates_module
-import microcosm.build.chronicle_epoch as chronicle_epoch_module
-import microcosm.build.country_spec as country_spec_module
 import microcosm.build.ledger_artifact as ledger_artifact_module
-import microcosm.build.ledger_targets as ledger_targets_module
 import microcosm.build.reference_factors as reference_factors_module
 import microcosm.build.reference_intervals as reference_intervals_module
 import microcosm.build.reference_profiles as reference_profiles_module
@@ -105,6 +102,7 @@
     COMPARISON_TYPE,
     TARGET_SURFACE_TYPE,
 )
+from .compile_modules import transport_compile_modules
 from .graph_inputs import (
     canonical_document_param,
     context_frame,
@@ -565,20 +563,9 @@ def _optional_float(value: float) -> float | None:
     return value if math.isfinite(value) else None
 
 
-_COMPILE_MODULES = (
-    graph_inputs,
-    artifact_types,
-    country_spec_module,
-    ledger_targets_module,
-    ledger_artifact_module,
-    chronicle_epoch_module,
-    reference_factors_module,
-    reference_profiles_module,
-    registry_module,
-    target_module,
-    hierarchy_module,
-    reference_selection_module,
-)
+#: The modules target compilation reads, named in ``compile_modules``, which
+#: no kernel hashes, so an edit to the list re-keys only what it changes.
+_COMPILE_MODULES = transport_compile_modules()
 _MATRIX_MODULES = (
     matrix_module,
     monetary_binding_module,

packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py

@@ -46,6 +46,14 @@
 from microcosm.build.uk_runtime.etb_services import (
     UK_NHS_SPENDING_COMPONENT_COLUMNS,
 )
+
+# Moved to ledger_surface, which the UK target kernels hash, so an edit
+# elsewhere in this module no longer re-keys them (microcosm#1492). Kept
+# importable from here for this module's other callers.
+from microcosm.build.uk_runtime.ledger_surface import (  # noqa: F401
+    _ledger_provenance,
+    _validate_band_edge_registry,
+)
 from microcosm.build.uk_runtime.national_frame import (
     uk_household_weight_kind,
     uk_time_period,
@@ -307,45 +315,6 @@ def new_uk_calibration_attempt_id(*, timestamp: datetime) -> str:
 _new_calibration_attempt_id = new_uk_calibration_attempt_id
 
 
-def _validate_band_edge_registry(
-    *,
-    register_registry: TargetRegistry,
-    band_edge_registry: TargetRegistry,
-    exclusion_receipt: Mapping[str, Mapping[str, str]],
-) -> None:
-    """Require the band-edge register to reconstitute the compiled roster.
-
-    The reconciliation always runs — an empty receipt is a claim that nothing
-    was pruned, so the two rosters must then be name-identical; it is never
-    permission to skip the check. Receipt keys are spec names by the
-    applier's construction: ``apply_uk_calibration_measure_exclusions``
-    builds the receipt from the matched ``spec.name`` set and raises on any
-    exclusion matching zero registry specs, so ``pruned + receipt keys ==
-    compiled`` is an exact identity, not a heuristic.
-    """
-
-    register_names = _registry_spec_names(register_registry)
-    edge_names = _registry_spec_names(band_edge_registry)
-    excluded_names = {str(name) for name in exclusion_receipt}
-    if not register_names <= edge_names:
-        missing = sorted(register_names - edge_names)
-        raise ValueError(
-            "band-edge register must include every materialized registry spec; "
-            f"missing={missing}."
-        )
-    extra = edge_names - register_names
-    if extra != excluded_names:
-        raise ValueError(
-            "band-edge register extra spec names must match the measure "
-            f"exclusion receipt; extra={sorted(extra)}, "
-            f"receipt={sorted(excluded_names)}."
-        )
-
-
-def _registry_spec_names(registry: TargetRegistry) -> set[str]:
-    return {str(spec.name) for spec in registry.specs}
-
-
 def load_bound_spine_sidecar(path: Path, frame: Frame) -> dict[str, object]:
     if not path.is_file():
         raise ValueError(f"input H5 build sidecar absent: {path}")
@@ -834,71 +803,6 @@ def uk_aggregate_admin_totals(
     return totals, receipt
 
 
-#: The manifest fields the UK run seals into ``run_config``. Narrower than
-#: the artifact's whole manifest on purpose: the identity digest should name
-#: *which* published artifact was compiled, not re-hash its manifest.
-_LEDGER_MANIFEST_IDENTITY_FIELDS = (
-    "artifact_id",
-    "profile",
-    "schema_version",
-    "generated_at",
-)
-
-#: Epoch witnesses :meth:`LedgerConsumerArtifact.provenance` supplies, split
-#: by shape. Named here so the delegation below cannot quietly drop one: a
-#: run that compiled a chronicle-era or mixed-epoch feed has to say so in its
-#: own evidence, not only in the loader's return value
-#: (PolicyEngine/chronicle#143).
-_LEDGER_EPOCH_WITNESS_SCALARS = ("schema_epoch",)
-_LEDGER_EPOCH_WITNESS_LISTS = (
-    "fact_key_epochs",
-    "undeclared_fact_key_domains",
-    "fact_schema_versions",
-)
-
-
-def _ledger_provenance(artifact: Any) -> dict[str, object]:
-    """The verified identity of the Chronicle consumer feed this run compiled.
-
-    Delegates to :meth:`microcosm.build.ledger_artifact.LedgerConsumerArtifact.provenance`
-    rather than rebuilding the block field by field. Rebuilding it is how the
-    epoch witnesses went missing here in the first place: the loader learned
-    which era resolved the targets and the UK run kept reporting only the
-    hashes. Anything the shared block gains, this block gains.
-
-    The whole block is sealed into ``run_config`` and so into the run's
-    identity digest: the epoch witnesses enter it beside the hashes, which is
-    why this delegation is a change to the digest of an otherwise identical
-    configuration, once. Only the manifest sub-block is UK-shaped, and it
-    stays narrow so the digest names the artifact rather than re-hashing its
-    whole manifest.
-
-    A bare ``consumer_facts.jsonl`` feed carries no manifest, so its
-    Chronicle-side provenance is recorded as absent rather than invented. So
-    is a stand-in that predates the shared block: the fields it cannot supply
-    are recorded ``None``/empty rather than fabricated.
-    """
-
-    shared = getattr(artifact, "provenance", None)
-    block: Mapping[str, Any] = shared() if callable(shared) else {}
-    provenance: dict[str, object] = {
-        field: block.get(field, getattr(artifact, field, None))
-        for field in ("facts_sha256", "fact_row_count", "manifest_sha256")
-    }
-    for field in _LEDGER_EPOCH_WITNESS_SCALARS:
-        provenance[field] = block.get(field)
-    for field in _LEDGER_EPOCH_WITNESS_LISTS:
-        provenance[field] = list(block.get(field) or ())
-    manifest = getattr(artifact, "manifest", None)
-    if isinstance(manifest, Mapping):
-        provenance["manifest"] = {
-            key: manifest.get(key)
-            for key in _LEDGER_MANIFEST_IDENTITY_FIELDS
-            if manifest.get(key) is not None
-        }
-    return provenance
-
-
 #: Packages whose versions the seam pins into the signed diagnostics build
 #: block. The release assembler treats these as the only legitimate source
 #: for manifest runtime/compatibility pins.

packages/microcosm-build/src/microcosm/build/uk_runtime/full_targets.py

@@ -14,10 +14,6 @@
 from typing import Any
 
 from microcosm.build.ledger_artifact import load_ledger_consumer_artifact
-from microcosm.build.uk_runtime.calibration_run import (
-    _ledger_provenance,
-    _validate_band_edge_registry,
-)
 from microcosm.build.uk_runtime.chronicle_feed import (
     load_uk_chronicle_feed,
     require_committed_uk_chronicle_feed_pin,
@@ -27,6 +23,10 @@
     uk_derived_fact_rows,
 )
 from microcosm.build.uk_runtime.frs_release import load_uk_frs_release
+from microcosm.build.uk_runtime.ledger_surface import (
+    _ledger_provenance,
+    _validate_band_edge_registry,
+)
 from microcosm.build.uk_runtime.ledger_targets import (
     _uk_contract_targets,
     assert_uk_local_deferrals_in_force,

packages/microcosm-build/src/microcosm/build/uk_runtime/graph_national.py

@@ -472,6 +472,10 @@ class _NationalKernel(KernelBase):
     )
 
     def implementation_hash(self) -> str:
+        import microcosm.calibrate.artifacts as calibrate_artifacts
+        import microcosm.calibrate.matrix as calibrate_matrix
+
+        from . import weighted_integrity
         from .target_compile_modules import uk_target_compile_modules
 
         implementation = source_hash(
@@ -489,6 +493,17 @@ def implementation_hash(self) -> str:
             national_calibration,
             national_doctrine,
             support_floor_rules,
+            # The constraint matrices and their encoding, as the dense kernels
+            # hash them (microcosm#1492). Not calibrate.solve: neither the
+            # national target nor the problem path calls it. Its one caller
+            # here, national_calibration_manifest, runs in
+            # uk.full.national-gates@1, which reads the solve's result, whose
+            # key moves with calibrate.solve through calibrate.adam@1.
+            calibrate_matrix,
+            calibrate_artifacts,
+            # full_targets takes its exclusion clock from here; hashed
+            # directly, as the dense target kernels do.
+            weighted_integrity,
             *uk_target_compile_modules(),
         )
         return hashlib.sha256(