[MaxGhenis]

Stacked on #599 (the base build and fit comparison), so its base is w3-fit-comparison; retarget to main once #599 lands. Do not merge on its own. Once approved on its gates, this lands with the d833 uk-data batch on Max's go. It is part of workstream W3 of the Autumn Budget 2026 CI plan: the single-age and HBAI checks.

Why

Calibration fits grouped targets: five-year national sex-age bands, ten-year regional age bands, income bands and totals. These checks measure what the groups hide, on the built dataset:

• the income distribution DWP publishes in HBAI (Median income runs 8-10% above HBAI FYE 2025; add an HBAI validation suite #487);
• the population by single year of age (Calibration leaves the teenage age mix and education status without direct targets: 35% too many 16-year-olds, half the 18- and 19-year-olds #563, which reported ages 15 to 19).

What changes

policyengine_uk_data/utils/distribution_validation.py covers two areas.

HBAI, at period 2024, which is FRS 2024/25 and HBAI FYE 2025. It measures:

• person-weighted medians, means and Gini coefficients of equivalised household income, before and after housing costs, in £ a week. As in HBAI, negative incomes before housing costs are reset to zero, and income after housing costs is derived from the adjusted figure. Weekly amounts use 365.25 / 7, as uk-data's FRS loader annualises them;
• the share of individuals below 60% of the median, for everyone, HBAI dependent children, working-age adults and those at State Pension age.

Each is compared with DWP's estimate and 95% confidence interval. These are in targets/reference_data/hbai_fye_2025.yaml, taken from the published uncertainty-hbai-2024-25-tables.ods, tables 8a and 8b. The independent review checked all 42 numbers against the tables; the tests spot-check three.

This is not an exact reproduction of HBAI. DWP adjusts incomes within the year for when each household was interviewed ("in-year deflation"), and adjusts "very rich" households to HMRC's survey of personal incomes. The second affects means and the Gini coefficient, not the median-based statistics. Neither is applied here. The FRS loader also clips self-employment losses at zero (datasets/frs.py, np.maximum(0, person.seincam2)), which HBAI keeps. So low-income households with losses can look richer here, which bears on the rates below 60% of the median.

Single ages, at the calibration year. It measures weighted people by sex and age 0 to 79, plus 80 and over, against the ONS 2024-based projection the calibration targets come from. Ages from 80 up are one band because the dataset's oldest records fall in 80–84 (#593).

In CI (.github/workflows/pull_request.yaml):

• the head's test job and Compare each pull request's calibration fit with its base build's #599's base-build job each measure their built dataset;
• the base is measured with the head's code;
• each uploads its report (distribution_report.json, base_distribution_report.json), which holds aggregates only. With Compare each pull request's calibration fit with its base build's #599, those reports and the fit summaries are the only artifacts. The raw calibration logs stay on the runner;
• the fit-comparison job compares the two and adds the result to the PR comment;
• the head's measurement runs after the test suite and the fit summary, whatever they did, with continue-on-error. It downloads the ONS projection, so an ONS outage can't skip the tests or the fit summary, or turn the test job red. The missing report fails the comparison instead, under its own FAIL heading;
• if the distribution comparison crashes, the comment says so, rather than showing only the fit result.

The comparison flags a measure that leaves its interval between base and head. It fails only on a broken, mismatched or missing report. That includes a report missing because the ONS projection couldn't be downloaded. The same-code spread of these measures is unmeasured (#598), so a no-worsening allowance can't yet be set on evidence. These runs measure it.

What it finds on uk-data 1.58.1

Measured on the released enhanced_frs_2024_25.h5 with policyengine-uk 2.122.2.

Four of 14 HBAI measures are outside their 95% intervals:

Measure	PE	HBAI	95% interval
Gini BHC (%)	30.75	32.74	31.89–33.98
Gini AHC (%)	35.62	36.78	35.90–38.29
Children below 60% of median, BHC (%)	25.28	20.52	17.36–23.96
Children below 60% of median, AHC (%)	31.92	27.41	23.49–30.64

The medians are inside their intervals: BHC 732.08 against 719.48, and AHC 636.65 against 623.14 (upper bound 637.07). The +8.4% BHC gap #487 found on 1.56.16 has narrowed. The lower Gini is partly expected, because the very-rich adjustment is not applied.

Single ages 0–79 differ from the ONS projection by 19.5% on average and by up to 89% for one age. Totals by sex are within 1–2% (men 33.7m against 34.0m, women 35.8m against 35.5m). Calibrating to five-year bands leaves single years lumpy across every age, not only 15–19.

Invariants (tested)

• Weighted median. It lies between the lower and upper medians of the integer-weight expanded sample, and is invariant to weight scaling (Hypothesis).
• Weighted Gini. It equals the mean-difference Gini of the expanded sample (Hypothesis, a differential against an independent formula), and is 0 for equal incomes.
• HBAI measures are checked on a known population.
• The reference file: three spot values match the published table, and every estimate lies within its interval.
• Negative incomes: DWP's treatment, on Sol's example (BHC −100 to 400, £50 housing costs): means £200 and £150, Gini 40% and 53.3%.
• Incomplete reports: a report missing a reference measure, or with a non-finite age summary, fails, and so does a missing distribution report. An empty group, such as no children in the data, has no rate (None, not a NaN), so the comparison fails on it.
• Single ages split by sex, and 80+ is banded.
• Comparison.
  • A self-comparison flags nothing.
  • Leaving an interval flags but doesn't fail.
  • A malformed report, mismatched periods or a non-finite measure fails.
• Data protection. The workflow uploads exactly the two fit summaries and the two distribution reports, and uses no actions/cache step (tested).

Tests

pytest -W error::RuntimeWarning policyengine_uk_data/tests/test_distribution_validation.py policyengine_uk_data/tests/test_fit_report.py: 39 passed.

axiom: n/a: data validation CI; no policy change

🤖 Generated with Claude Code