Repository navigation
Compare each build's national aggregates with its base's, model fixed - #602
Merged
Merged
Conversation
Pull request CI now also builds the PR's base, in the same way as the head (TESTING, one clone) and with the head's lockfile, so both run one model and environment. A new job compares the two final-epoch fits from their calibration logs. It fails when the share of national or local targets within 10% falls by more than 2pp, flags national targets whose error rises by more than 0.2, and posts the comparison on the PR. The thresholds (targets/fit_policy.yaml, adopted under d1344) come from five builds of the same data code: their shares moved by at most 0.93pp (national) and 0.69pp (local), and they flagged 0 to 6 targets per pair. Per-target moves only flag while #598 (two weight sets from one code) is open. Only calibration logs are uploaded, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibration fits grouped targets; these checks measure what the groups hide, on the built dataset. Each PR build and its base build report person-weighted equivalised income medians, means and Gini, and relative low income for everyone, children, working-age adults and pensioners, against DWP's HBAI FYE 2025 estimates and 95% intervals (tables 8a and 8b, read from the published ODS files), at period 2024. They also report people by sex and single year of age against the ONS 2024-based projection at the calibration year. The comparison job adds them to the PR comment and flags a measure that leaves its interval; it fails only on a broken or mismatched report until the same-code spread is measured (#598). On uk-data 1.58.1, 5 of 14 HBAI measures are outside their intervals (median AHC, both Gini coefficients, child low income BHC and AHC), and single ages 0 to 79 are off by 19.5% on average. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both builds are measured with policyengine-uk's aggregate-impact tool at a pinned commit (policyengine-uk#2425, stacked on #2335), under the head's locked model: receipts by tax head, spending and caseloads, poverty, inequality, deciles and the ten standard reform costings, 2025-26 to 2031-32. The comparison job runs `compare --axis data`, which fails unless both sides ran one model, runtime and set of definitions, and adds the result to the PR comment. It fails on a check the head breaks that the base passes. Only each side's result.json (aggregates) is uploaded; the tool's private household detail stays on the runner. Under the locked 2.122.2 the tool falls back to policyengine-uk main's partition, whose lists differ from that model's, so closure and two identities fail on both sides (pre-existing) until uk-data locks a model that ships its own partition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sol review of #600: - Negative incomes before housing costs are reset to zero, and income after housing costs is derived from the adjusted figure (DWP HBAI methodology FYE 2025, "Negative incomes"). Weekly amounts use 365.25 / 7, as the FRS loader annualises them. - A report missing any reference measure, or with a missing or non-finite single-age summary, fails the comparison. - A missing distribution report fails the job. On uk-data 1.58.1, four of 14 measures are now outside their intervals (both Gini coefficients, child low income BHC and AHC); median AHC moves inside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rint commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Raw calibration logs stay on the runner. Each build uploads a summary: whether every national and local target is within 10%, and the error of national targets supported by at least 10 households only (the constituency log gains a records column). An estimate from a few records, divided by a published count, could give one household's value back. The test job's raw-log uploads are replaced by the summary too. - The gate compares shares over the targets both builds have; added and removed targets are notes. - An empty national or local group, or a last epoch with fewer rows than the one before, fails. - The comment step writes a failure comment when the comparison never ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/pull_request.yaml # policyengine_uk_data/tests/test_fit_report.py
hbai_measures divided by a zero group weight, which warned and gave NaN. It now returns None for an empty group, the report marks it outside its interval, and compare fails on it as missing or not finite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/pull_request.yaml
…aggregate-data-comparison # Conflicts: # .github/workflows/pull_request.yaml # policyengine_uk_data/tests/test_fit_report.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aggregate-data-comparison
Calibration fit against the base build: pass
A drop of more than 2.0% fails. Notes
National targets in a worse error band on the head (0)None. Flags are for review: builds of the same code move 2 to 12 national targets to a worse band (#598). Bands are of |estimate / target - 1|; no target's estimate or error is published. Income distribution and single ages against the base build: passHBAI FYE 2025 (DWP), period 2024: equivalised income in £ a week, rates and Gini in per cent. A value outside HBAI's 95% interval is marked (outside).
FlagsNone. These measures report and flag but do not fail yet: their same-code spread is unmeasured (#598). Aggregate impact of the data change: passBase dataset Checks: 0 broken by this change, 4 failing on both sides, 0 fixed. Moves flagged for review: 10 (thresholds in Notes
Moves flagged for review
Receipts by tax head
Spending by benefit and service
Caseloads
Government balance, each component counted once
Model totals
Components the model counts twice
Household income totals
Population
Poverty rates
People in poverty
Inequality
Mean household net income in each side's own deciles
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileThe datasets hold different households, so households are not matched; compare the mean income in each side's own deciles. Checks failing on both base and head (4)
Counted-once checkCounted-once check not run: not requested. Provenance
Base dataset: Head dataset: Runtime differences:
Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Per-target errors could not be published safely: the dataset repeats households (output-area clones, SPI and CGT donors), so a row count is not a respondent count, and one household can dominate a target. The fit summary now holds only within-10% booleans, and a national target that leaves the band on the head is flagged in place of an error rise of more than 0.2. On the five same-code builds, 1 to 10 of 644 national targets leave the band per pair (mean 4.5). The records column in the calibration log is reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…son (delta review of #600) The measurement downloads the ONS projection, so it now runs after the test suite with if: !cancelled(): an ONS outage no longer skips the tests and the fit summary. A crashed distribution comparison writes a FAIL line into the comment, and the final error names both checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/pull_request.yaml
The within-10% boolean could not see a target worsening outside the band. The summary now holds each target's error band (within 10%, 10-25%, 25-50%, 50-100%, over 100%), still a bound and not a value, and flags a national target whose band is worse on the head: 2 to 12 per same-code pair (mean 7). Also: repeated local targets are refused; the fit summary runs after the tests with !cancelled(); data jobs restore but never save uv's cache; fit_policy.yaml records the run ids, the log folders and the status of the replacement rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view 2 of #600) The head's distribution measurement now has continue-on-error: a missing report already fails the comparison, so an ONS outage no longer turns the test job red. A missing report gets its own FAIL heading in the comment. A test pins the step order and conditions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ggregates independent (review of #602) A crashed aggregate comparison now sets the failure status first and writes its own FAIL heading; a missing result says so under a heading too. The base build's aggregate steps run whatever its distribution measurement did. A measurement that writes no result warns instead of failing at cp, and the final error names every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
That head records local installs and fingerprints the model at both ends of a measurement, so this PR's CI runs the identity checks the batch will. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eview of #600) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iring (delta review of #602) - The tool pin moves to 5d5558ed, which refuses any non-index install and fingerprints the model before the simulation is built. - The head's tool checkout and aggregate measurement have continue-on-error: a missing result fails the comparison instead. - A test pins one tool commit across all three checkouts, !cancelled() on every aggregate step, and the crash fallback's status line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5d5558ed published totals less their suppressed heads, which policyengine-uk#2335's re-reviews 6 and 7 showed can leak; 6406b2ce rebuilds them from their published parts and withholds reform figures from small cells. Every checkout of the tool still uses one full SHA (test_fit_report). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
56 of 78 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
mainbut contains #599 and #600: the pull request workflow runs only on PRs intomain, so this PR is their combined CI run. Review #599 and #600 separately; this PR's own change is its last commit. Do not merge on its own. Once approved on its gates, this lands with the d833 uk-data batch on Max's go. It also depends on PolicyEngine/policyengine-uk#2425 (the tool's data axis), which is stacked on #2335. The tool is pinned at #2425's approved head,6406b2ce(GPT-6.1 Sol delta review 11). That head includes #2335's current small-cell protection: totals rebuilt from their published parts, and reform figures withheld per record. The earlier pin,5d5558ed, still subtracted suppressed figures, a method re-reviews 6 and 7 showed can leak. Move the pin to #2425's merged commit when it lands; until then this commit stays reachable on #2425.This is W3 of the Autumn Budget 2026 CI plan: "run the comparison with the model fixed".
What changes
The head's
testjob and Compare each pull request's calibration fit with its base build's #599'sbase-buildjob each measure their builtenhanced_frs_2024_25.h5with policyengine-uk's aggregate-impact tool, copied from a pinned commit and run with the head's locked model. They measure, for 2025-26 to 2031-32:The guard is skipped: it tests the model, which is the same on both sides.
The head's aggregate steps run after the tests, and the base's after its other reports, all with
if: ${{ !cancelled() }}, so one failing measurement can't skip another or the tests.The
fit-comparisonjob runscompare --axis data. It fails unless both sides ran one policyengine-uk version, runtime (apart from policyengine-uk-data itself), partition and set of metric definitions. It fails on a check the head breaks that the base passes, or a missing result. It adds the tool's comment to the PR comment. If the comparison crashes, or one side has no result, the comment says so under its own FAIL heading, and the job fails.Data protection.
result.json(aggregates) is uploaded.private/household detail stays on the runner.actions/cachestep is used, and that onlyresult.jsonis copied out of the tool's folder.Tested locally
The tool runs under uk-data's locked policyengine-uk 2.122.2 on the released 1.58.1 file in 111 s, with reforms and without the guard.
compare --axis dataof that measurement against itself passes, with a note that the bytes are the same.On that file four checks fail:
gov_spendingandhousehold_benefitsidentities;sign:vat.Closure is structural to the model, so it fails on both builds of a PR alike. The two identities and
sign:vatdepend on the data. A self-comparison of one file doesn't show that both builds fail them the same way, and the comparator treats a newly failing year as a regression. A PR whose build first breaks one of them will fail.On the same file, policyengine-uk 2.130.1 passes closure and all seven identities, and still fails
sign:vat; it ships its partition. That is observed (pe-uk#2335's certified measurement) and doesn't separate the partition from the other model changes since 2.122.2.This PR's own CI is the first end-to-end run.
Tests
pytest -W error::RuntimeWarning policyengine_uk_data/tests/test_fit_report.py policyengine_uk_data/tests/test_distribution_validation.py: 29 passed, including the extended upload test.axiom: n/a: data comparison CI; no policy change
🤖 Generated with Claude Code