You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
UK nation regions fail because the household data has no country column #558
Every UK nation region fails on the certified UK data. country/england, country/scotland, country/wales and country/northern_ireland scope with RowFilterStrategy(variable_name="country", ...) (src/policyengine/countries/uk/regions.py). That filter reads a country column from the input household table. The certified year file does not store one: policyengine-uk derives country from region in a formula.
Reproduced on 2026-10-08 with main at 9663cf2 (policyengine-uk 2.102.3). pe.uk.ensure_datasets(years=[2026]) wrote enhanced_frs_2024_25_year_2026.h5, with 52,846 households. The household table has region, region_code_oa, la_code_oa and constituency_code_oa, but no country. Both of these raise:
region=uk_region_registry.get("country/scotland") # same for country/englandSimulation(dataset=year_2026, tax_benefit_model_version=pe.uk.uk_latest,
scoping_strategy=region.scoping_strategy).run()
ValueError: Variable 'country' not found in household data. Available columns: ['household_id', 'household_weight', 'region', ...]
#553 (section 3) reports the same failure on the Microcosm national file.
Fix
Before scoping, the model version checks the household variables the strategy filters on. If one is missing from the input household table, policyengine-uk calculates it on the unscoped dataset. The derived column is used for the filter only and dropped from the scoped input. policyengine.py then keeps no copy of the region-to-nation mapping: the nations are whatever policyengine-uk's country formula says. A strategy now declares the variables it filters on.
On the 2026 year file, computing country through policyengine-uk takes about 2.7s on the full dataset. It splits the 52,846 households 37,961 England / 5,405 Scotland / 3,821 Wales / 5,659 Northern Ireland, with each region mapped to exactly one nation.
Fixed on main by #565 (released in 6.2.3), from #564: the nation regions now filter the stored region column.
I checked it on the certified enhanced_frs_2024_25_year_2026.h5 with main at 07bae75 and policyengine-uk 2.102.3. Each nation filter keeps exactly the households policyengine-uk's own country formula assigns to that nation:
region
households
weighted households
country/england
37,961
26,354,698.47
country/scotland
5,405
2,729,247.77
country/wales
3,821
1,590,754.53
country/northern_ireland
5,659
783,847.56
The four sets are disjoint and cover all 52,846 households (31,458,548.32 weighted). Simulation(..., scoping_strategy=country/scotland).run() returns the same 5,405 households and leaves the caller's tables unchanged.
#568 no longer changes scoping. It now adds the tests from #569, which pin #565's region lists to policyengine-uk's country formula. Its earlier approach calculated country through the model before filtering, which the repository guidance added in #565 rules out. Separately, scoped UK runs still carry UK-wide business rates and region-relative ranks: #567.
Activity
MaxGhenis commentedon Oct 9, 2026
Fixed on
mainby #565 (released in 6.2.3), from #564: the nation regions now filter the storedregioncolumn.I checked it on the certified
enhanced_frs_2024_25_year_2026.h5with main at 07bae75 and policyengine-uk 2.102.3. Each nation filter keeps exactly the households policyengine-uk's owncountryformula assigns to that nation:The four sets are disjoint and cover all 52,846 households (31,458,548.32 weighted).
Simulation(..., scoping_strategy=country/scotland).run()returns the same 5,405 households and leaves the caller's tables unchanged.#568 no longer changes scoping. It now adds the tests from #569, which pin #565's region lists to policyengine-uk's
countryformula. Its earlier approach calculatedcountrythrough the model before filtering, which the repository guidance added in #565 rules out. Separately, scoped UK runs still carry UK-wide business rates and region-relative ranks: #567.