You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tie US year files to the source dataset and policyengine-us version they were cut from #591
ensure_datasets() names each US year file {data_folder}/{stem}_year_{year}.h5, where the stem is dataset_logical_name(resolve_dataset_reference("us", dataset)). dataset_logical_name drops the @revision, and create_datasets() names files by materialize_dataset(...).name, which is the same stem. Since #590, a year file is reused when its stamp (legacy_inputs.YEAR_FILE_STAMP_H5_DATASET) holds this release's register of renamed inputs and format revision. The stamp does not record which source the file was cut from, or under which policyengine-us version:
An edited local source. A local dataset rewritten in place (the usual smoke-test workflow with allow_unmanaged=True) keeps its year files, which still hold the old data. Two local sources with the same file name in different folders share year files too.
Another policyengine-us version. Year files store each input as calculated for that year by the installed policyengine-us. Their values can depend on that version (for example, its uprating parameters). A year file cut under one version is reused under another.
Proposal
Extend the stamp, and raise YEAR_FILE_STAMP_REVISION to 2:
source: the identity of the dataset the file was cut from. For a managed dataset, the resolved hf://…@revision URI and the sha256 that materialize_dataset verified against the bundle manifest. For an unmanaged Hugging Face URI, the URI as given. For a local file, its resolved path, size and modification time.
policyengine_us: the installed policyengine-us version.
ensure_datasets() and load_datasets() compare the stamp with the identity they can compute without downloading or reading the source. For managed datasets that is the bundle manifest entry, for an unmanaged URI the URI, and for a local path one stat(). They also compare the version from importlib.metadata, which does not import policyengine-us. The check still reads only the stamp and the renames record from each year file. A mismatch makes ensure_datasets() cut the year files again and load_datasets() refuse, with a reason that names both identities.
Activity
Test and document year-file source and package identity
Check the source again before stamping, refuse shared stems, read lin…
Decode raw stamp and record nodes in the totality oracle
Snapshot the source file after materializing and require it unchanged…
Compare the source snapshot by size and mtime only; refuse unsupporte…
Hash explicit downloads and re-hash before stamping; validate hf:// U…
Say the swapped-and-restored gap applies to every source
Check the source again only when year files are cut from it