Repository navigation
Description
#1493 (for #1492) adds a check that walks every module each target-compilation kernel passes to source_hash, the kernel's own module included. It requires every first-party module those modules import to be hashed by that kernel, or recorded with a reason. source_hash hashes only the files it is given and does not follow imports, so an import nobody hashes is code a kernel can run without its node key changing.
The kernels checked, in groups that share one hash:
- UK:
- the full graph's
uk.full.target_compilation@1,target_selection@1andmeasures@1; uk.full.problem@1;- the national graph's
national_targets@1andnational_problem@1.
- the full graph's
- Transport:
targets.compile@1,takeup.compare@1,takeup.gap@1,transport.quantile_map@1and@2, each alone;transport.create@1with@2.
On the first run the check found imports that no review has classified yet. #1493 records them as they are, in an UNCLASSIFIED_IMPORTS mapping. Each importer -> imported edge names the groups that leave it unhashed:
- UK: 66 modules, 112 edges, in
packages/microcosm-build/tests/engine_free/uk/test_uk_target_compile_modules.py. - Transport: 35 modules, 74 edges, in
packages/microcosm-build/tests/engine_free/shared/test_transport_compile_modules.py.
The check fails on an unhashed edge a group meets that isn't recorded for that group, and on an edge recorded for a group that no longer meets it. So the mapping can only shrink, a new import of unhashed code fails at once, and removing a module from one group's hash fails even where another group records the same edge.
Classify these first: code the kernels run on every build today.
uk_runtime.national_calibration, whichuk.full.measures@1runs throughfull_measure:CalibrationFrameAdapter,inject_measure_inputsanddrop_injected_measure_inputs. The national kernels already hash this module; the four full-graph kernels don't.uk_runtime.rowwise_dataset.ladder_clone_index_column, whichuk.full.measures@1runs throughfull_measure. No UK kernel hashesrowwise_dataset.uk_runtime.graph_population.context_frame, which rebuilds the population in the full measures and problem kernels and in the national problem. None of the six UK kernels hashesgraph_population.
Goal: classify every entry, then delete the mappings. For each entry, one of:
- Hash it in the kernels that run it. Over-keying is safe, but each module hashed by the UK target kernels also re-runs the UK solve when it is edited. That works against reusing the solve between two builds (UK local-area release: publish a K=15, 150,000-household dense dataset in at most two builds #1301).
- Move what the kernel runs into a small module that the kernels hash. Include geography_constants in the target-compilation cache keys #1493 does this for
calibration_run._ledger_provenanceand_validate_band_edge_registry, now inuk_runtime/ledger_surface.py. - Record why the kernel doesn't need it hashed, for example that the import serves another kernel or the graph builder in the same file. One recorded entry already carries such a note:
graph_national -> calibrate.solve, which onlyuk.full.national-gates@1runs.
What the other entries are. Most fall into three groups:
- imports of a kernel's own module that serve the graph builder or other kernels in the same file, such as
graph_national -> graph_build,graph_calibration,staging_v2andlogbook_adoption; - the UK measurement code's other imports, such as
measure_simulation -> frs_spine,frame.adapters.policyengine_uk,lcfs_consumptionanduc_target_measurements, many of which probably run when the measures kernel runs; - the
microcosm.frameandmicrocosm.graphmodules, and the imports ofcountry_spec's loader (spec_engine.*).
Out of scope: the modules #1493 hashes: calibrate.target_selection, calibrate.matrix, calibrate.artifacts, calibrate.solve (in the full problem only), uk_runtime.weighted_integrity and the new uk_runtime.ledger_surface.
Activity
Check every module each compile kernel hashes for unhashed imports (red)
Hold every transport kernel to the derived binding rule (#931)