Skip to content

Tie US year files to the source dataset and policyengine-us version they were cut from #591

Open

Description

@MaxGhenis

Follow-up to #532 / #590.

Problem

ensure_datasets() names each US year file {data_folder}/{stem}_year_{year}.h5, where the stem is dataset_logical_name(resolve_dataset_reference("us", dataset)). dataset_logical_name drops the @revision, and create_datasets() names files by materialize_dataset(...).name, which is the same stem. Since #590, a year file is reused when its stamp (legacy_inputs.YEAR_FILE_STAMP_H5_DATASET) holds this release's register of renamed inputs and format revision. The stamp does not record which source the file was cut from, or under which policyengine-us version:

  • A re-pinned managed dataset. When the bundle points a logical name at a new revision or sha256, year files cut from the old release keep the same names and pass the stamp check, so ensure_datasets() and load_datasets() reuse them. This is about to happen to populace_us_2024_acs_local, whose replacement (Publish complete ACS-local WIC participation and remove the temporary .py assumption microcosm#1154) ends the temporary ACS WIC assumption (Remove temporary ACS WIC assumption after replacement certification #563).
  • An edited local source. A local dataset rewritten in place (the usual smoke-test workflow with allow_unmanaged=True) keeps its year files, which still hold the old data. Two local sources with the same file name in different folders share year files too.
  • Another policyengine-us version. Year files store each input as calculated for that year by the installed policyengine-us. Their values can depend on that version (for example, its uprating parameters). A year file cut under one version is reused under another.

Proposal

Extend the stamp, and raise YEAR_FILE_STAMP_REVISION to 2:

  • source: the identity of the dataset the file was cut from. For a managed dataset, the resolved hf://…@revision URI and the sha256 that materialize_dataset verified against the bundle manifest. For an unmanaged Hugging Face URI, the URI as given. For a local file, its resolved path, size and modification time.
  • policyengine_us: the installed policyengine-us version.

ensure_datasets() and load_datasets() compare the stamp with the identity they can compute without downloading or reading the source. For managed datasets that is the bundle manifest entry, for an unmanaged URI the URI, and for a local path one stat(). They also compare the version from importlib.metadata, which does not import policyengine-us. The check still reads only the stamp and the renames record from each year file. A mismatch makes ensure_datasets() cut the year files again and load_datasets() refuse, with a reason that names both identities.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions