Revive the UK leg as an internal audit of policyengine-uk

[MaxGhenis]

Summary

This revives PolicyBench's UK leg as an internal audit tool for policyengine-uk. Nothing is published: the public release stays US-only, and no snapshot, payload or dashboard file changes.

Bit-rot fixed (each one observed, with a test)

1. Engine pin. policyengine-uk 2.89.0 → 2.125.1 (latest on PyPI, 2026-10-10). The 5 October branch bumped policyengine-core from 3.32.8 to 3.32.17 and changed no US reference: a fresh 100-household US sample gave identical references under both, 2,292 of 2,292 rows. Main's lock now carries 3.32.29.
2. PIP silently zero. The pinned transfer artifact (uk-data 9514dfb) stored pip_*_reported, which policyengine-uk stopped reading in #1656, so PIP was 0 for every household. It is re-pinned to uk-data 6b1f80e (sha256 c663daea…), which stores PIP categories.
3. Prompt and reference disagreed on amounts. The prompt showed the stored 2025 amounts; the reference used PE-UK's uprated 2026-27 values (employment income ×1.034, rent ×1.021). Given the prompted facts, the engine's income tax matched the old reference in 24 of 100 households. The loader now reads every stored variable at the reference period.
4. References read facts the prompt omitted. UK references came from the full microdata record: UC deductions, a seeded draw of new LCWRA claimants, sex, and a seeded date of birth. They now come from a PE-UK simulation of exactly the prompted facts, one per household, as on the US path. calculate_uk_transfer_microsimulation_values keeps the old path as a cross-check: 691 of 700 cells agree. The 9 that don't are explained: 8 by UC deductions, 1 by the LCWRA draw.
5. Silent skips on renamed variables. is_child_or_QYP was removed in policyengine-uk#1655 and the loader skipped it without warning. The loader now fails loudly on any stored variable the engine doesn't define, on a renamed derived variable, and on benefit-unit inputs the prompt can't show.
6. Derived facts the engine relies on. The prompt now states them:
   • sex and date of birth (State Pension age, the savings credit cutoff);
   • education status at 16-19 (qualifying young person rules; PE-UK otherwise imputes it from age);
   • LCWRA held since before 6 April 2026, in place of "is disabled for benefits", with a child-specific line for children;
   • annual hours worked.
7. Supervisor was US-only. It never passed --country, so every UK worker failed. It now passes each scenario's country and output set. US workload fingerprints are unchanged (tested).

One canonical scenario (review rounds 2-6)

The prompt renderer and the reference builder both start from canonical_uk_scenario / canonical_uk_person, so a scenario loaded from a manifest gives each the same facts. It applies these conventions, each stated in the prompt:

• Facts as shown. Amounts are stored as the whole number the prompt shows and rates to the four significant figures it shows. Explicit zeros are kept as stated facts.
• Relationships. Two listed adults are "a couple, not married or in a civil partnership"; the reference sets claimant and partner flags and no marriage, so PE-UK's presumed-child rule and married-couple presumption don't apply.
• Locality. A private renter without a Broad Rental Market Area gets a fixed BRMA in their region, stated in the prompt (PE-UK otherwise reads Maidstone's LHA everywhere, policyengine-uk#2256).
• Statuses PE-UK would fill in. Someone aged 16 to 19 with no education status is not in education. Non-advanced education is "full-time ... below higher-education level", with the age it began (16 where none is given). Someone with self-employment income and no determination is not in gainful self-employment.
• Pension contributions. PE-UK deducts both kinds from taxable income and neither from adjusted net income (policyengine-uk#2243). The labels state that convention: employee contributions under a net pay arrangement, personal contributions as a gross amount; neither is described as relief at source.
• Rejected inputs. Tax-unit and benefit-unit inputs, and person or household fields the prompt filters out, raise instead of reaching the engine unseen.

Reference adversary for the UK

The reference adversary from #200 now audits the UK leg:

• consensus-flags and adversary-prepare take --country; load_payload reads one country's payload.
• UK cases get UK prompts: fiscal year 2026-27, legislation.gov.uk and GOV.UK sources, pounds, region, and a 2026-10-10 freeze with the schema to match.
• All 62 committed US cases re-render byte for byte (checked against main's module), and the US schemas are unchanged.

Invariants (tested)

• Prompted facts only. A UK reference situation holds exactly the prompted facts (Hypothesis property test over random inputs).
• No batch dependence. A household's reference does not depend on the other households in the batch.
• Reference-period inputs. Prompted amounts are the reference-period values, not the stored ones.
• No silent drops. Every stored artifact variable is one the installed engine defines.
• Protected LCWRA claim. A lone LCWRA household gets more than the new-claimant rate, which backs the prompt's "held since before 6 April 2026". This is a slow test against the real engine.
• US unchanged. US supervisor fingerprints and US adversary prompts are unchanged.

Verification

Run locally on Python 3.13 at the final head:

• tests/test_scenarios.py, test_eval_no_tools.py, test_held_references.py, test_reference_adversary.py: 435 passed (5 slow deselected).
• After the merge of Record two references held against a model consensus, and let the adversary list them (evidence only; no score changes) #213 and Audit the Tax Table against the rate schedule in the federal income tax references (evidence only; no score changes) #214: those plus test_consensus.py, test_cli_helpers.py and test_cli_reference_outputs.py, 505 passed.
• Earlier in the branch: the slow LCWRA guard and real-engine tests passed (4), and before the first rebase the full suite passed (1,573).
• Ruff: clean. CI green.

Six rounds of GPT-6.1 Sol review; round 6 approved 869e9bb2. The reviews and prompts are in ~/reviews/policybench-uk-revival-2026-10/reviews/.

What this found

The internal UK run on 100 transfer-path households cost $11.08 across six models. A blind reference audit of 79 cells (59 consensus, 20 single-model misses) found 21 cells on policyengine-uk defects, 34 on unstated facts (now stated, above), 9 on rounding (fixed here) and 13 where the reference held. Engine fixes from this work:

• Raise the Test job timeout to 120 minutes policyengine-uk#2258 (merged): the Test job limit goes to 120 minutes.
• Pay pre-2026 UC health element claimants the legislated protected amount policyengine-uk#2248 (merged): pre-2026 claimants get the legislated £429.80 protected LCWRA amount (SI 2026/113).
• Count a loss on one source as nil in Pension Credit income and earnings policyengine-uk#2259 (merged): a loss on one source no longer reduces other Pension Credit income or earnings.
• Stop deciding a household's UC health element rate by its position policyengine-uk#2251: a household's UC health element no longer depends on its position in the simulation.
• Give Gift Aid relief by raising the rate limits, not by deducting the donation policyengine-uk#2271: Gift Aid raises the rate limits instead of being deducted from income.
• Issues: policyengine-uk#2252 (UC rebalancing reforms ignored unless applied before the data load) and #2256 (default BRMA).

The full record is ~/reviews/policybench-uk-revival-2026-10/README.md. Whether to publish a UK board is Max's call (d1254).

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 10, 2026 10:51pm UTC

[MaxGhenis]

Merging on the gates.

• Head: 869e9bb2.

• Checks: all green (lint, paper, test, app, Vercel).

• Mergeable: MERGEABLE, CLEAN, not a draft, no change request.

• Independent review: GPT-6.1 Sol, six rounds. Rounds 1 to 5 requested changes:

  • relationships and marriage;
  • BRMA, rounding on manifest load, entry age, hidden household inputs;
  • explicit zeros, missing education, rates;
  • the personal pension label and rate precision;
  • pension contributions and adjusted net income.

  All were fixed. Round 6 approved this head.

• Scope: no snapshot, payload or dashboard file changes, and no score changes; the public release stays US-only.

• Merge method: squash, as for the recent PRs on main.

🤖 Generated with Claude Code