policyengine-uk 2.102.4+ needs policyengine-core 3.32.9+, which the certified US default refuses

Problem

From policyengine-uk 2.102.4 (2026-09-30), policyengine-uk requires policyengine-core>=3.32.9; 2.102.3 and earlier accept >=3.30.1. microcosm's workspace resolves one policyengine-core for both country extras, so uv lock --upgrade-package policyengine-uk==2.104.2 moves core 3.32.5 → 3.32.12 for the US environment too.

The certified US default that latest.json points to, populace-us-2024-spm-20260915, declares compatible_core_packages: policyengine-core ==3.32.5. With core 3.32.12 installed, microcosm.data.loader._enforce_package_compatibility on that release manifest raises:

_CertifiedPackageCompatibilityError: Certified release 'populace-us-2024-spm-20260915' is incompatible with installed policyengine-core 3.32.12; release certification requires ==3.32.5.

That is the failure #936 fixed for policyengine-us. Until it is resolved, microcosm cannot lock any policyengine-uk release from 2.102.4 on without breaking US loading.

The certified UK default, populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z, already fails the same check on main. It pins policyengine-uk ==2.89.2 and policyengine-core ==3.27.1, while main locks 2.100.0 and 3.32.5.

Options

A. Resolve each country's core separately with [tool.uv] conflicts between the US-side and UK-side extras. A first trial on main 3598c38 (nine pairwise sets, microcosm-build/data/frame us × uk, with policyengine-uk 2.104.2) is incomplete. These parts work:
- uv lock forks core into 3.32.5 and 3.32.12.
- uv sync --all-packages --locked --extra us keeps core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0.
- --extra uk gets core 3.32.12.
These do not:
- --package microcosm-data --extra us, --package microcosm-frame --extra policyengine and --all-packages --extra policyengine still draw core 3.32.12 with policyengine-us 2.2.1.
- --extra policyengine --extra uk installs both engines together.
A complete version needs three more things:
- conflict sets that cover microcosm-frame's policyengine extra;
- a core pin on the US-side extras;
- a lock-level test that every US install path resolves the certified trio and that every US × UK combination is refused.
CI already installs one extra per job.
B. Hold policyengine-uk at 2.100.0 until a certified US release moves core.
C. Re-certify the US default under the newer core.

What the next policyengine-uk bump touches

This was rehearsed against policyengine-uk 2.104.2 plus PolicyEngine/policyengine-uk#2021, with the engine-uk suite run under each.

Uprating pins. tools/pin_uk_uprating_engine_values.py moves only the version stamp; the values are unchanged.
Concept mapping. Review the mapping, set engine_version, then run tools/refresh_concept_coverage.py --engine policyengine-uk.
- 28 inputs were added since 2.100.0: alcohol and tobacco quantities, car attributes, lifetime_isa_balance, mortgage_debt and consumer_debt.
- None matches an existing concept. fact:household.mortgage_principal is the repayment flow, already bound to mortgage_capital_repayment.
- brma leaves the input list once Default the BRMA from the household's region policyengine-uk#2021 is in.
UK spine graph. tools/graph_uk_spine_fixture.py rewrites only uk_spine.json, and only the numerical_dependencies version strings change.
Lock hash. APPROVED_UV_LOCK_SHA256 in us_runtime/worker_identity.py follows the new lock hash.

These four are the only engine-uk failures at 2.104.2. Adding PolicyEngine/policyengine-uk#2021 does not change that list.

The eFRS parity reference (optional, and deliberately fenced)

uk/efrs_parity_reference.json still records policyengine-uk 2.89.0. Regenerating it with any later engine changes the surface: 2.89.1 adds bus_fare_spending, and 2.89.2 adds employment_sector and sic_industry_division. With 2.89.2 or later it does three things:

It adds bus_fare_spending, employment_sector and sic_industry_division, taking the reference from 145 to 148 populated layers. This already happens at the lock's 2.100.0.
It makes build_uk_release_input_coverage_manifest.py require a --candidate-h5 evidence refresh.
With Default the BRMA from the household's region policyengine-uk#2021, it adds brma to formula_owned_persisted_overrides_included, taking that list from 13 to 14.

The refresh is purely additive. All three new columns carry signal in the certified candidate, which gives 148 required columns and 0 exclusions, and the spine produces all three. The engine-free tests still pin the frozen candidate at 2.89.0 and 145 columns (test_frozen_candidate_retains_its_original_engine_provenance and three siblings), so the re-pin should be a deliberate step, not a side effect of a bump. experiments/791-household-composition-receipts.md says it belongs with #749.

PolicyEngine/policyengine-uk#2021 needs nothing here

brma computed from the pinned enhanced FRS for 2024, 2025, 2026, 2027, 2030 and 2032 is byte-identical with and without PolicyEngine/policyengine-uk#2021. The same holds on microcosm's certified candidate for 2023, 2024, 2025, 2026, 2028 and 2030; see the comment below. microcosm's frs_brma stage sets brma for every household, so an explicit input always beats the new region default. The only brma change is the reference bookkeeping above.

Also checked on microcosm's certified UK candidate populace_uk_2023.h5 (revision populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z): brma for 2023, 2024, 2025, 2026, 2028 and 2030 is byte-identical under policyengine-uk 2.104.2 with and without PolicyEngine/policyengine-uk#2021, for all 535,080 households (same sha256 every year). The new formula only fills households with no stored BRMA, and microcosm stores one for every household.

Corrected the description after an independent review:

The option A trial is incomplete. microcosm-data[us] and microcosm-frame[policyengine] still drew core 3.32.12, and policyengine + uk could be installed together.
The certified UK default already fails the loader's check on main (==2.89.2 / ==3.27.1).
The measured years are now listed exactly.
The 2.89.1 and 2.89.2 layer additions are now split out.

@MaxGhenis

Two ways to fix this. Which do you prefer, @MaxGhenis?

Why it's urgent. #1095 tracks the uk-data fixes microcosm still needs. Since #1100 merged, most of its open items wait only on a policyengine-uk bump, and policyengine-uk has now released all of them:

Replace generic child and adult flags with each programme's legal definitions policyengine-uk#1896 (2.107.0): is_hbai_dependent_child and is_claimant_or_partner;
Let a dataset supply each benefit unit's own Pension Credit capital policyengine-uk#2018 (2.105.0): Pension Credit reported capital;
Date the mixed-age couple Pension Credit exclusion and model the SI 2019/37 saving policyengine-uk#1940 (2.109.4): the mixed-age couple Pension Credit saving;
Give the legacy severe disability premium its statutory conditions (stacked on #1896) policyengine-uk#1946 (2.112.0): the severe disability flag;
Give sharers, boarders and lodgers their own rent, tenure and LHA category policyengine-uk#2006 (2.113.0): shared rent, and the rent boarders and lodgers pay.

The same bump also brings engine-only fixes:

pension-age Housing Benefit new claims (Let pension-age families make new Housing Benefit claims policyengine-uk#1901);
interest, dividends and rent leaving UC unearned income (Stop counting interest, dividends and rent as Universal Credit unearned income policyengine-uk#1950);
published LHA rates (Use published LHA determinations and apply Sch 3B paras 3 and 3A policyengine-uk#2022);
is_renting and Child Benefit opt-outs (Fix Child Benefit opt-outs, rental flags and bus fare modelling policyengine-uk#2140).

Every one of these needs policyengine-core 3.32.9 or later, and 3.32.12 or later from policyengine-uk 2.109.8. The latest releases are policyengine-uk 2.121.0 and core 3.32.17. So microcosm UK can't take any of them until this is fixed.

Option 1: build and certify a new US default under a newer core.

Both countries keep one core, and the lock stays as simple as it is today.
Re-claiming the same H5 bytes with a wider core range, as populace-us-2024-spm-20260915 did for policyengine-us, probably won't hold here. Core 3.32.8, 3.32.16 and 3.32.17 change how uprating and auto-carry-over work, so projected-year outputs can move. That makes this a new build and certification.
The coupling comes back the next time policyengine-uk raises its core floor.

Option 2: give each country its own core pin (completing option A above).

Add [tool.uv] conflicts between each US-side and each UK-side engine extra, and add policyengine-core==3.32.5 on the US-side extras. Every US install path then resolves the certified trio, while the UK fork moves on its own.
The 2 October trial had two gaps:
- its pairs used microcosm-frame[us], which installs no engine, instead of microcosm-frame[policyengine], which installs policyengine-us;
- the US extras had no core pin, so some US install paths still drew 3.32.12.
CI already installs one country extra per job, and docs/agent-guide.md already sets up one environment per country, so nothing needs both engines in one environment.
The PR would add a CI check that:
- every US install path resolves core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0;
- every US × UK combination is refused.
It would also re-pin APPROVED_UV_LOCK_SHA256.
On its own it changes no engine version. The UK bump would follow as a separate PR, and the US would stay on core 3.32.5 until its next certification.

Which would you prefer?

Lock policyengine-uk 2.122.2 and policyengine-core 3.32.19 (#1095)

@juaristi22

@juaristi22 Option 2: give each country its own core pin.

Why not option 1:

The delay is open-ended. Moving the US to a newer core is a full new build, not a re-claim, because core 3.32.8, 3.32.16 and 3.32.17 change uprating and carry-over. A clean US build takes about two days, and the last one took nine. Even then the lock can't move until latest.json points at the new release. Under d844 that promotion waits on US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578's exact-k gate.
It comes back. The next policyengine-uk core floor raise would recreate the same coupling.

Nothing needs both engines in one environment. No module imports both policyengine_us and policyengine_uk, and CI installs one country extra per job.

For the PR:

Pin the US extras. Pin policyengine-core==3.32.5 on every US-side extra, including microcosm-frame[policyengine], which installs policyengine-us. Add [tool.uv] conflicts between each US-side and each UK-side engine extra.
Add a lock-level test. Every US install path should resolve core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0, and every US × UK combination should be refused.
Re-pin the lock digest. Re-pin APPROVED_UV_LOCK_SHA256 in us_runtime/worker_identity.py. Any uv.lock change fails the stacked-spine tests until it's re-pinned.
Keep it to the lock. The PR should change no engine version. The policyengine-uk bump follows as its own PR.

The US moves to the newer core with its next certified build, likely the Route A rerun on the CT-fixed ladder (d713), not before.

Thanks for laying out both options so clearly.

Give each country its own policyengine-core pin (#1086)

Lock policyengine-uk 2.122.2 and policyengine-core 3.32.19 on the UK …

Give each country its own policyengine-core pin (#1086)

Lock policyengine-uk 2.122.2 and policyengine-core 3.32.19 on the UK …

Merge pull request #1143 from PolicyEngine/uk-per-country-core-pin

Lock policyengine-uk 2.122.2 and policyengine-core 3.32.19 on the UK …

Lock policyengine-uk 2.123.0 for the landlord inputs (#1106)

Merge pull request #1169 from PolicyEngine/microcosm-pe-uk-core-split…

#1169 merged (4ed3e64). It finishes what this issue asked for, on top of two earlier PRs:

Give each country its own policyengine-core pin (#1086) #1143 (4cb4352) carried out the ruling: option 2, each country keeps its own policyengine-core pin;
UK: port the remaining uk-data fixes tracked in #1095, engine items included #1121 moved the UK side to policyengine-uk 2.122.2 and core 3.32.19.

What #1169 added:

Agent guide, Commands section. uv refuses any sync that combines an extra that installs policyengine-us (microcosm-build[us], microcosm-data[us] or microcosm-frame[policyengine]) with one that installs policyengine-uk. That includes --all-packages --all-extras. Each country needs its own UV_PROJECT_ENVIRONMENT.
Lock-level test. It now also checks --all-packages --extra policyengine, the path the 2 October trial found drawing core 3.32.12, and --all-packages --extra policyengine --extra us. It has 16 cases.

State of the lock at the merge (uv sync --locked --dry-run, empty environment):

All seven US install paths resolve policyengine-core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0, which is what the certified US default populace-us-2024-spm-20260915 requires.
All five UK install paths resolve policyengine-core 3.32.19 and policyengine-uk 2.122.2.
All six US × UK combinations exit 2 with "incompatible with the declared conflicts".
uv lock --check exits 0, and APPROVED_UV_LOCK_SHA256 equals the lock's sha256.
The certified UK default populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z still fails the loader's compatibility check (policyengine-uk ==2.89.2, policyengine-core ==3.27.1). That hasn't changed since before Give each country its own policyengine-core pin (#1086) #1143, and a new certified UK build is what clears it.

PolicyEngine/policyengine-uk#2021:

It's still an open draft. I read policyengine_uk/variables/household/BRMA.py in every policyengine-uk wheel on PyPI from 2.104.0 to 2.125.0 (68 wheels, each checked against PyPI's sha256). None has def formula or REGION_DEFAULT_BRMA, and #2021's diff adds both. So no release gives brma a formula yet.
When one does, microcosm's data doesn't change. frs_brma stores brma for every household. On 2026-10-02, brma was byte-identical per year with and without #2021, on the pinned eFRS and on the certified candidate (comments above).
The eFRS parity reference stays at engine 2.89.0, with 13 formula-owned overrides and no brma. That follows the ruling's "keep it to the lock". At the next deliberate re-pin under a release that carries #2021, brma joins formula_owned_persisted_overrides_included (13 to 14). test_uk_release_input_coverage.py asserts len(overrides) == 13, so the re-pin can't miss it.

The issue stays closed. Nothing more is needed here for #2021 or d816.

Lock policyengine-uk 2.123.0 for the landlord inputs (#1106)

Lock policyengine-uk 2.127.0 (#1095)

Navigation Menu

Repository navigation

policyengine-uk 2.102.4+ needs policyengine-core 3.32.9+, which the certified US default refuses #1086

Problem

Options

What the next policyengine-uk bump touches

The eFRS parity reference (optional, and deliberately fenced)

PolicyEngine/policyengine-uk#2021 needs nothing here

Two ways to fix this. Which do you prefer, @MaxGhenis?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

policyengine-uk 2.102.4+ needs policyengine-core 3.32.9+, which the certified US default refuses #1086

Description

Problem

Options

What the next policyengine-uk bump touches

The eFRS parity reference (optional, and deliberately fenced)

PolicyEngine/policyengine-uk#2021 needs nothing here

Activity

MaxGhenis commented on Oct 2, 2026

MaxGhenis commented on Oct 2, 2026

juaristi22 commented on Oct 5, 2026

Two ways to fix this. Which do you prefer, @MaxGhenis?

MaxGhenis commented on Oct 7, 2026

MaxGhenis commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions