Skip to content

Classify the imports the target-compilation kernels' hashes do not yet cover #1497

Open

Description

@anth-volk

#1493 (for #1492) adds a check that walks every module each target-compilation kernel passes to source_hash, the kernel's own module included. It requires every first-party module those modules import to be hashed by that kernel, or recorded with a reason. source_hash hashes only the files it is given and does not follow imports, so an import nobody hashes is code a kernel can run without its node key changing.

The kernels checked, in groups that share one hash:

  • UK:
    • the full graph's uk.full.target_compilation@1, target_selection@1 and measures@1;
    • uk.full.problem@1;
    • the national graph's national_targets@1 and national_problem@1.
  • Transport:
    • targets.compile@1, takeup.compare@1, takeup.gap@1, transport.quantile_map@1 and @2, each alone;
    • transport.create@1 with @2.

On the first run the check found imports that no review has classified yet. #1493 records them as they are, in an UNCLASSIFIED_IMPORTS mapping. Each importer -> imported edge names the groups that leave it unhashed:

  • UK: 66 modules, 112 edges, in packages/microcosm-build/tests/engine_free/uk/test_uk_target_compile_modules.py.
  • Transport: 35 modules, 74 edges, in packages/microcosm-build/tests/engine_free/shared/test_transport_compile_modules.py.

The check fails on an unhashed edge a group meets that isn't recorded for that group, and on an edge recorded for a group that no longer meets it. So the mapping can only shrink, a new import of unhashed code fails at once, and removing a module from one group's hash fails even where another group records the same edge.

Classify these first: code the kernels run on every build today.

  • uk_runtime.national_calibration, which uk.full.measures@1 runs through full_measure: CalibrationFrameAdapter, inject_measure_inputs and drop_injected_measure_inputs. The national kernels already hash this module; the four full-graph kernels don't.
  • uk_runtime.rowwise_dataset.ladder_clone_index_column, which uk.full.measures@1 runs through full_measure. No UK kernel hashes rowwise_dataset.
  • uk_runtime.graph_population.context_frame, which rebuilds the population in the full measures and problem kernels and in the national problem. None of the six UK kernels hashes graph_population.

Goal: classify every entry, then delete the mappings. For each entry, one of:

What the other entries are. Most fall into three groups:

  • imports of a kernel's own module that serve the graph builder or other kernels in the same file, such as graph_national -> graph_build, graph_calibration, staging_v2 and logbook_adoption;
  • the UK measurement code's other imports, such as measure_simulation -> frs_spine, frame.adapters.policyengine_uk, lcfs_consumption and uc_target_measurements, many of which probably run when the measures kernel runs;
  • the microcosm.frame and microcosm.graph modules, and the imports of country_spec's loader (spec_engine.*).

Out of scope: the modules #1493 hashes: calibrate.target_selection, calibrate.matrix, calibrate.artifacts, calibrate.solve (in the full problem only), uk_runtime.weighted_integrity and the new uk_runtime.ledger_surface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions