Stamp US year files with the rename register they were cut under

[MaxGhenis]

Fixes #532

The defect

ensure_datasets() reuses a US year file ({data_folder}/{stem}_year_{year}.h5) whenever it exists and passes its staleness check. #531 made that check "does the file carry the renames record?" (policyengine_legacy_input_renames). The record shows that a file was cut with the legacy-input mapping, not under which register. #531's own body and the comment on RENAMES_H5_DATASET both say so. Once LEGACY_INPUT_RENAMES gains an entry (policyengine-us renames another input), year files cut before it still carry a record and are reused. They can lose the newly renamed input exactly as files cut by policyengine.py 6.1.1 and earlier lost the WIC take-up draw: about $11.5B of WIC instead of $6.6B in 2024 (PolicyEngine/microcosm#1026).

A subset rule ("the file's register covers today's") would not close the gap either:

• A file cut by a newer release under a larger register stores that entry's live name. An older release whose engine does not define that name skips the column, because _build_simulation_from_dataset sets only columns in system.variables.
• A removed or retargeted entry fails the same way.

What changes

The stamp. Each year file create_datasets() writes now ends with an H5 dataset, policyengine_year_file_stamp (legacy_inputs.YEAR_FILE_STAMP_H5_DATASET). It holds UTF-8 JSON {"register": <the whole LEGACY_INPUT_RENAMES>, "revision": 1}:

• It is written last. PolicyEngineUSDataset.save() opens pd.HDFStore(mode="w"), which truncates the file, then writes the tables and then the renames record. Only after that is the stamp written. A write that stops at any point leaves no stamp, so the file is cut again.
• It holds the whole register, not only the entries the engine needed, so checking it never imports policyengine-us.
• YEAR_FILE_STAMP_REVISION invalidates every year file whenever extraction changes in a way the register cannot express. Its comment names one such change that is coming: removing the temporary ACS WIC assumption from Temporarily handle missing ACS WIC participation inputs #566 (planned in Remove temporary ACS WIC assumption after replacement certification #563). Year files cut while that assumption applies store assumed claiming as takes_up_wic_if_eligible.
• save() neither writes nor keeps the stamp. Re-saving a year file drops it, so the file is cut again.

The check. legacy_inputs.year_file_stamp_problem(path) returns None only when all of these hold:

• the stamp has exactly the keys register and revision;
• its revision is an int (not bool) equal to YEAR_FILE_STAMP_REVISION;
• its register is a str→str map equal to this release's register;
• the renames record is a str→str map listing only renames in the stamped register.

Otherwise it returns a reason clause, built from the register, with no WIC special case. It reads only those two H5 datasets, never the tables or the source dataset. Any other OSError while opening or reading, or content that cannot be parsed (including JSON nested past the recursion limit), counts as out of date. A PermissionError, IsADirectoryError or BlockingIOError is raised instead, because cutting the file again would only fail on the write.

Callers.

• ensure_datasets() checks every year file and logs each out-of-date one with its reason. It then cuts all of them again, as before.
• load_datasets() raises ValueError("US year file {path} is out of date: {reason}. Regenerate it with ensure_datasets() or create_datasets().").
• read_renames_record() now returns None for a malformed record (a JSON list, non-string values, bad JSON, a group) instead of accepting [] as {} or raising.

Examples of the reasons:

• a file cut by 6.1.1 or earlier: "it has no record of the renamed stored inputs mapped when it was cut, so it was cut by an older policyengine.py or not written completely, and may have lost any of: would_claim_wic -> takes_up_wic_if_eligible";
• a file written under Map the stored WIC take-up draw onto takes_up_wic_if_eligible when loading US data #531, with the record but no stamp: "it has no stamp of the register of renamed inputs it was cut under, so it may have lost any of: …";
• a file cut before an entry was registered: "it was cut before these renames were registered, so it may have lost them: …";
• a file cut by a newer release: "it was cut under renames this version of policyengine.py does not register: …".

Files written under #531 carry the record but no stamp, so they are cut again once. Grandfathering them would mean hard-coding #531's register, which is a WIC special case.

Invariants

All of these are executed, not just stated.

Invariant	Property tests (Hypothesis)	Example and integration tests
Exactness. A file is current iff its stamp has this release's revision and register, and its record lies within the stamped register.	test_a_year_file_is_current_only_under_this_releases_exact_stamp (300 examples; registers drawn from a 4-name alphabet, so subsets, supersets and retargets are common)	test_a_future_register_entry_makes_a_stamped_file_out_of_date, test_retiring_or_retargeting_an_entry_…, test_raising_the_format_revision_makes_older_files_out_of_date
Totality. For any stamp and record content, the check returns a reason or None. It raises only the three errors unrelated to content.	test_the_year_file_problem_is_total_and_exact_for_any_content (300 examples: arbitrary JSON, raw text, raw bytes, numbers, arrays, groups)	test_a_malformed_stamp_is_out_of_date (14 cases), test_json_nested_too_deeply_is_out_of_date_not_an_error, test_a_recursion_error_while_parsing_is_unreadable_content, test_errors_unrelated_to_the_contents_are_raised and …_while_reading
Refinement. The stamp never accepts a file the #531 record check refused.	inside the exactness property (read_renames_record(path) == record whenever current)
Round trip. A stamp reads back as written, writing again replaces it, and the tables and record are untouched.	test_a_year_file_stamp_reads_back_as_written (50 examples)	test_the_stamp_is_stored_as_utf8_json, test_a_stamp_and_record_stored_as_fixed_length_strings_are_read
Named renames. The reason names every entry the two registers disagree on, as old -> new, and no entry they agree on.	test_a_stale_year_file_names_each_rename_it_may_lack (200 examples)	test_the_refusal_names_the_file_and_what_it_may_lack
No partial file is current. Covers every truncation, every save that stops part-way, and every save that does not stamp.	test_a_truncated_year_file_is_never_current (100 prefix lengths)	test_a_save_that_stops_part_way_leaves_no_stamp, test_saving_a_year_file_again_drops_its_stamp, test_create_datasets_stamps_each_year_file_after_saving_it (call order), test_a_year_file_whose_stamp_was_never_written_is_out_of_date
Ensure agrees with load. ensure_datasets cuts again exactly when a file is missing or load_datasets would refuse one, logs exactly the out-of-date files with their reasons, and reuses the results.	test_ensure_cuts_again_exactly_when_load_would_refuse (60 examples over 14 file states × 1–3 years)
Stamp-only decision. Deciding never opens a table or the source dataset.		test_ensure_decides_from_the_stamps_alone (patches h5py.File so only the stamp and record can be read, and makes pd.HDFStore and materialize_dataset raise)

Integration over the real policyengine-us 2.2.1 engine uses the three-person Los Angeles WIC fixture.

• test_a_year_file_cut_before_a_register_entry_is_regenerated replays the WIC history with no fake engine. It cuts under an empty register, and the file stores neither name, so the infant gets WIC. Back under the real register, load_datasets refuses the file, naming would_claim_wic -> takes_up_wic_if_eligible. ensure_datasets cuts it again, and the infant's WIC is 0.
• test_a_future_register_entry_regenerates_year_files adds a fictitious entry whose names the engine does not define. The file goes stale and is cut again, the data is unchanged, and the new stamp carries the entry.
• test_ensure_datasets_regenerates_a_year_file_with_only_the_renames_record covers a file written under Map the stored WIC take-up draw onto takes_up_wic_if_eligible when loading US data #531.
• Map the stored WIC take-up draw onto takes_up_wic_if_eligible when loading US data #531's test_ensure_datasets_regenerates_a_year_file_cut_before_the_mapping still passes unchanged.

Mutation testing. All 32 mutants were killed. 30 ran through a harness on a separate copy of the source, with the Hypothesis database cleared before each mutant. Two were run by hand: a table read in the predicate, and pd.read_hdf in the decision. Each was killed by the test aimed at it:

• Register comparison: subset, reverse subset, keys only.
• Revision: check dropped, < instead of !=, isinstance accepting True, hardcoded 1.
• Stamp and record handling:
  • unstamped files counted as current;
  • an absent record treated as {};
  • the record-within-register check dropped;
  • extra stamp keys accepted;
  • a malformed register accepted;
  • fixed-length strings rejected.
• Error handling:
  • permission and lock errors swallowed on open, and separately on read;
  • unreadable files raising;
  • the JSON reader raising on bad content;
  • RecursionError not caught.
• Readers and writers:
  • read_renames_record accepting lists;
  • writing without deleting first;
  • the register frozen at import.
• create_datasets and save(): no stamp written; stamp written before save(); save() appending instead of truncating.
• ensure_datasets and load_datasets:
  • keeping the record-only check (ensure_datasets, and separately load_datasets);
  • ignoring the stamp;
  • stopping at the first stale file;
  • logging the wrong file;
  • load_datasets skipping existing files;
  • reading a table node or pd.read_hdf during the decision.

The harness and its log are in ~/PolicyEngine/_reviews/pepy-590/ (mutate2.py, mutate2.log). Two mutants can't be killed, as the review noted: the node.shape != () guard is equivalent, and KeyError in the reader's catch is defensive, since dangling links read as absent.

Review

An independent GPT-6.1 Sol review (hard tier) asked for changes. The review is in ~/PolicyEngine/_reviews/pepy-590/review-1.md. Every finding was fixed in 10201bf8:

• uncaught RecursionError on deeply nested JSON;
• read-time permission and lock errors were swallowed;
• the decision test still allowed table reads through h5py;
• missing tests for fixed-length strings, a raised revision and exact log lines;
• four docs claims that overstated the code.

The one exception is the source-revision finding, which is a separate follow-up (see below). A delta review (served by Opus 5.5 when no Codex lane was free) approved 9cfc1ed2 (review-2.md). Its three nits:

• Fixed in 96a7e974. The decision test now also blocks pandas.io.pytables.HDFStore and tables.open_file. A mutant adding pd.read_hdf(filepath, "person") to the decision is now caught, which I confirmed.
• Pinned. Dangling soft and external links under the stamp or record names read as absent.
• Kept on purpose. The node.shape != () guard is an equivalent mutant, kept as documentation.

A third review, of 9cfc1ed2..96a7e974 (Opus 5.5, review-3.md), approved with two optional test nits. Both were applied in 69296e0e, as it prescribed:

• the HDFStore patches now use a raising class, so pandas' isinstance checks still work and a pd.read_hdf mutant fails with the intended message, which I confirmed;
• the dangling external link now points under tmp_path.

A fourth review, of 96a7e974..318e109d (GPT-6.1 Sol, review-4.md), approved the platform fix below. It confirmed both parameters of the forced-RecursionError test fail when the catch is removed, and that the temporary global json.loads patch and the nested read-only HDF5 opens are safe. Its one nit was that the comment gave the macOS/Linux split as a rule when it is observed behaviour. That was fixed in f0db8f31, a comment-only commit.

Not covered here

• Saved simulation outputs. Simulation.load() still gates only on the record. The comment on RENAMES_H5_DATASET and docs/microsim.md say so. This is proposed as a separate session.
• Source dataset identity. dataset_logical_name drops @revision, so year files cut from one release of a dataset are reused after the bundle re-pins the same logical name. This is proposed as a separate session. It matters before the ACS-local replacement (microcosm#1154, Remove temporary ACS WIC assumption after replacement certification #563).
• Concurrency and atomicity. Two concurrent ensure_datasets calls on one folder are not locked, as before. A crash mid-write loses the old file, which is then cut again; it is not replaced atomically.
• Shared data folders. Two policyengine.py versions with different registers that share one data folder cut each other's year files again. The docs say so.

What I verified

Locally with Python 3.14.4 on macOS, HF_HUB_OFFLINE=1, no downloads, peak RSS 3.7 GiB:

• at 10201bf8, tests/test_us_year_file_stamp.py, tests/test_us_legacy_inputs.py and tests/test_us_legacy_inputs_integration.py: 174 passed. Later commits change only tests/test_us_year_file_stamp.py, whose 71 tests pass at 318e109d (f0db8f31 changes only a comment);
• dataset suites (test_dataset_*, test_release_manifests, test_run_record, test_us_long_term_datasets, test_us_native_*): 139 passed;
• ruff 0.16.9 and 0.17.0 (CI installs the latest): format --check and check clean.

CI at 69296e0e found a platform difference. CPython 3.14 on Linux parses the test's 100,000-deep JSON into nested lists instead of raising RecursionError, as 3.11–3.13 and 3.14 on macOS do. The check gave the right reason either way: a list is a malformed stamp. Only my test's expectation of read_year_file_stamp(...) is None assumed the parser fails. 318e109d makes that test hold either way, and adds a test that forces RecursionError, so the guard is covered (and its mutant caught) on every platform.

CI: pending on the latest head.

axiom: n/a: infra (US year-file cache staleness; no policy logic)

🤖 Generated with Claude Code

[MaxGhenis]

Merged at head f0db8f31 with a merge commit (4d19fc68) after every gate passed:

• CI: gh pr checks exits 0 on the latest run at f0db8f31; all 13 checks pass, including Test 3.11–3.14. The branch contained origin/main (5dddc0a2), so CI tested exactly what merged.
• Mergeability: MERGEABLE, merge state CLEAN; no CHANGES_REQUESTED review.
• Independent reviews, read-only Subfleet lanes. Outputs are kept in ~/PolicyEngine/_reviews/pepy-590/:
  1. GPT-6.1 Sol (hard tier) at 9cdb8702: REQUEST_CHANGES. All in-scope findings were fixed in 10201bf8.
  2. Opus 5.5 at 9cfc1ed2: APPROVE. Its nits were fixed in 96a7e974.
  3. Opus 5.5 on 9cfc1ed2..96a7e974: APPROVE. Its nits were fixed in 69296e0e.
  4. GPT-6.1 Sol on 96a7e974..318e109d, the Linux 3.14 deep-JSON test fix: APPROVE. Its comment nit was fixed in f0db8f31, a comment-only commit.
• Mutation testing: all 32 mutants were killed. The harness and log are in the same folder.

Follow-ups proposed as separate sessions:

• tie year files to the source dataset revision, so re-pinning a logical name such as the ACS-local replacement cuts them again;
• give saved US outputs the same register stamp;
• stop load_datasets reading each year file twice.