Simulation(dataset=path) fails for files written by UKMultiYearDataset.save: the multi-year check never passes #2416

What is wrong

Simulation(dataset=<path>) fails for every .h5 file written by UKMultiYearDataset.save.

The multi-year check never passes. UKMultiYearDataset.validate_file_path looks for a top-level key that splits on / into two parts, like person/2025 (L315-L320). But h5py.File.keys() lists only the top-level groups (benunit, household, person, time_period). save writes the years as children of those groups (person/{year}, L253-L280), so no top-level key ever has two parts. The check returns False for every file save writes.

The single-year check then passes by mistake. build_from_file next tries UKSingleYearDataset.validate_file_path. That check only asks whether time_period, person, benunit and household exist, and in a multi-year file they exist as parent groups, so it returns True.

Loading then fails. UKSingleYearDataset(path) reads f["person"] from an HDFStore and raises TypeError: cannot create a storer if the object is not existing nor a value are passed.

UKMultiYearDataset(path) itself works, because its constructor ignores the False and reads through HDFStore, whose keys are full paths such as /person/2025. So only dispatch by validation is broken.

test_simulation_dataset_sources.py::test_build_from_file_loads_local_multi_year_dataset replaces UKMultiYearDataset with a fake whose validate_file_path returns True, so it hides this.

Reproduction (MOCK data: two synthetic households)

Using policyengine-uk at 6b1e199 (build_from_file and dataset_schema.py are unchanged on main 175c6d1):

h5py top-level keys: ['benunit', 'household', 'person', 'time_period']
HDFStore keys (first 4): ['/benunit/2025', '/benunit/2026', '/benunit/2027', '/benunit/2028']
UKMultiYearDataset.validate_file_path -> False
UKSingleYearDataset.validate_file_path -> True
UKMultiYearDataset(path) years -> [2025, 2026, 2027] ...
Simulation(dataset=path) raised TypeError cannot create a storer if the object is not existing nor a value are passed

The script builds the file with extend_single_year_dataset(UKSingleYearDataset(file_path=<mock single-year h5>), system.parameters).save(path), then calls the two validators, the UKMultiYearDataset constructor and Simulation(dataset=path).

Fix

• Multi-year check: walk the groups, for example with isinstance(f[name], h5py.Group) and a year-named child under each of person, benunit and household. Or read pd.HDFStore(path, "r").keys() and require /person/<year>-style keys.
• Single-year check: reject a file whose person group holds year children rather than a pandas table, so the two checks are mutually exclusive.
• Tests:
  • a real round trip, UKMultiYearDataset.save → Simulation(dataset=path), with no stub;
  • a property test that each saved file passes exactly one validator.

Related: #1256 (closed July 2025, "UKMultiYearDataset fails when downloading from HuggingFace … mostly because of a failure in validation").

Reported by the session working on #2404 and #2405, which reproduced it on main twice. The UK hub re-reproduced it and checked the code.