From 5865f0eec0316e8471cfae7650a17d107ebec008 Mon Sep 17 00:00:00 2001 From: Vahid Ahmadi Date: Wed, 27 May 2026 12:19:06 +0200 Subject: [PATCH 1/4] Add reform impacts dashboard planning doc (#806) #806 asks for a dashboard tracking high-importance reform budgetary impacts over time and across model / data versions. Auditing the existing infrastructure: - reforms_config.yaml already has 9 canonical regression fixtures (basic rate +1pp, UC taper to 20%, VAT +2pp, etc.) with expected impacts and tolerances. - test_parametric_reform_impacts.py runs each fixture in CI and fails on drift outside the tolerance. So the *fixtures* exist; what's missing is a published *time series* view onto them. The page proposes: - append-only CSV / Parquet storage of (timestamp, policyengine_uk_version, policyengine_uk_data_version, dataset_url, reform_name, expected_impact, observed_impact, gap), - a scheduled GitHub Actions runner (weekly + on tag) walking reforms_config.yaml, - three publication paths in order of effort: docs page Plotly chart, living JSON endpoint, policyengine.org dashboard tile, - annotations for EFO releases, dataset refreshes, methodology PRs. Cross-links to the nowcasting comparison and child poverty validation pages so the three together form a coherent "how is the model doing" view. --- changelog.d/806.md | 1 + .../reform-impacts-dashboard-plan.md | 149 ++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 changelog.d/806.md create mode 100644 docs/book/validation/reform-impacts-dashboard-plan.md diff --git a/changelog.d/806.md b/changelog.d/806.md new file mode 100644 index 0000000000..82657ea467 --- /dev/null +++ b/changelog.d/806.md @@ -0,0 +1 @@ +- Add a reform impacts dashboard planning docs page at `docs/book/validation/reform-impacts-dashboard-plan.md` covering how to build on the existing `reforms_config.yaml` fixtures and `test_parametric_reform_impacts.py` runner to publish a time series of canonical reform impacts across model and data versions, with annotations for EFO releases and dataset refreshes. diff --git a/docs/book/validation/reform-impacts-dashboard-plan.md b/docs/book/validation/reform-impacts-dashboard-plan.md new file mode 100644 index 0000000000..1a6cdda0a1 --- /dev/null +++ b/docs/book/validation/reform-impacts-dashboard-plan.md @@ -0,0 +1,149 @@ +# Reform impacts dashboard (planned) + +```{note} +**Planning page.** Tracks +[#806](https://github.com/PolicyEngine/policyengine-uk/issues/806), +which asks for a dashboard tracking the budgetary impact of +high-importance reforms over time and across model / data versions. +The repo already has the **fixtures** — what's missing is a published +view onto them. +``` + +## What exists today + +[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml) +is a list of canonical reform-impact regression fixtures with the +shape: + +```yaml +- name: Raise basic rate by 1pp + expected_impact: 7.8 # £bn fiscal impact at the reference year + parameters: + gov.hmrc.income_tax.rates.uk[0].rate: 0.21 +- name: Reduce Universal Credit taper rate to 20% + expected_impact: -18.1 + tolerance: 3.0 + parameters: + gov.dwp.universal_credit.means_test.reduction_rate: 0.2 +``` + +Each fixture is exercised by +[`test_parametric_reform_impacts.py`](../../../policyengine_uk/tests/test_parametric_reform_impacts.py) +on every PR, and any drift outside the `tolerance` fails CI. So CI +already catches *unintended* impact changes — but only via a binary +pass/fail signal that flips silently between PRs. + +What's missing is a **time series**: the same nine reforms run on +every commit / version, plotted against the published OBR/RF +benchmark for the same reform, with annotations for the relevant +infrastructure changes (model versions, dataset refreshes, EFO +updates). + +## Proposed shape + +### Storage + +Each scheduled run writes a row per fixture to a CSV / Parquet on +HuggingFace Hub or a small Postgres instance: + +``` +timestamp, policyengine_uk_version, policyengine_uk_data_version, +dataset_url, reform_name, expected_impact, observed_impact, gap +``` + +Append-only — no overwrites — so the history survives. + +### Computation + +A new entry point under `tests/microsimulation/` (or, better, a +GitHub Actions workflow that doesn't run inside the test suite) +walks `reforms_config.yaml`, runs each reform, and writes a row +per fixture. Schedule: weekly + on every release tag. + +### Publication + +Three publication paths, in order of effort: + +1. **Static page in the docs site** — the simplest. Convert the CSV + to a Plotly chart (matching the existing `validation/*.ipynb` + pattern) and render it under `docs/book/validation/`. Updates + when the docs site rebuilds. +2. **Living JSON endpoint** — push the latest aggregate JSON to a + stable URL (HuggingFace dataset or a `gh-pages` artefact) so the + policyengine.org app can render it without needing the docs build. +3. **policyengine.org dashboard tile** — a tile on the marketing site + that shows the latest impact alongside the OBR/RF benchmark. Most + work; right answer once the static page proves the data is stable. + +### What to put on the dashboard + +For each reform: + +- **Latest observed impact** (£bn, current model). +- **Published external benchmark** (OBR EFO costing where one + exists; RF *No half measures*-style costing as a comparator). Annotate + the publication date. +- **Time series** of observed impact across the last N model releases. +- **Annotations** for each known regime shift: EFO release, dataset + refresh, methodology PR. The reforms_config history is the natural + source of "what changed" annotations. + +## Initial reform list + +Use the existing `reforms_config.yaml` entries as the v1 dashboard +content: + +1. Raise basic rate by 1pp. +2. Raise higher rate by 1pp. +3. Raise personal allowance by ~£800/year. +4. Raise child benefit by £25/week per additional child. +5. Reduce Universal Credit taper to 20%. +6. Raise Class 1 main employee NI rate to 10%. +7. Raise VAT standard rate by 2pp. +8. Raise additional rate by 3pp. + +These cover three of the four big revenue heads (income tax, NI, VAT) +plus a UC parameter that matters disproportionately for distributional +analysis. + +Add to v2 over time as published OBR / RF reforms become available +benchmarks: + +- Remove the Universal Credit two-child limit (worked example in + [child-poverty-tcl.md](./child-poverty-tcl.md)). +- Freeze the Personal Allowance for an additional year. +- 1pp employer NI change. + +## What this doesn't replace + +This is a **regression-tracking** dashboard, not a forecast. The +numbers it plots are PolicyEngine UK's reading of a reform under +current model + data + EFO assumptions; they will move when any of +those move, by design. The whole point of plotting the time series is +to make those movements legible rather than hidden. + +For comparison with external nowcasters (RF, IFS, OBR scoring), the +relevant pages are: + +- [nowcasting-comparison.md](../assumptions/nowcasting-comparison.md) (PolicyEngine vs RF). +- [rf-nowcasting-methodology.md](../assumptions/rf-nowcasting-methodology.md) (RF methodology detail). +- [child-poverty-tcl.md](./child-poverty-tcl.md) (worked external comparison). + +## Open questions + +- Storage substrate: HuggingFace dataset CSV vs a small managed + database. CSV is simplest; database lets the policyengine.org app + query directly. +- Scheduling: every commit to main, every weekly tag, or both? + Recommend weekly + on tag, since per-commit runs cost + microsim time without proportional information gain. +- Should the dashboard include reform variants from the policyengine + blog series, or stick to the parametric fixtures? Recommend + parametric fixtures only for v1 to keep maintenance burden low. + +## References + +- Issue: [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806). +- Existing fixture list: [`reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml). +- Existing regression runner: [`test_parametric_reform_impacts.py`](../../../policyengine_uk/tests/test_parametric_reform_impacts.py). +- Visualisation helper used by neighbouring validation pages: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py). From 801a667d471af78bd141daf97b55fc553fc14e82 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Thu, 8 Oct 2026 15:52:22 +0100 Subject: [PATCH 2/4] Correct the reform impacts dashboard plan Name the test that runs the fixtures (microsimulation/test_reform_impacts.py, gov_balance in 2029, default tolerance 1bn), fix the fixture count to eight, describe update_reform_impacts.py correctly, link #1724 instead of a page that is not on main, and add why the Microcosm move makes the history useful. Type the changelog fragment. Co-Authored-By: Claude Opus 5.5 --- changelog.d/806.added.md | 1 + changelog.d/806.md | 1 - .../reform-impacts-dashboard-plan.md | 43 +++++++++++++------ 3 files changed, 31 insertions(+), 14 deletions(-) create mode 100644 changelog.d/806.added.md delete mode 100644 changelog.d/806.md diff --git a/changelog.d/806.added.md b/changelog.d/806.added.md new file mode 100644 index 0000000000..3a5e0d597d --- /dev/null +++ b/changelog.d/806.added.md @@ -0,0 +1 @@ +Added a plan for a reform impacts dashboard (#806) at `docs/book/validation/reform-impacts-dashboard-plan.md`, tracking the regression fixtures in `reforms_config.yaml` over model and dataset versions. diff --git a/changelog.d/806.md b/changelog.d/806.md deleted file mode 100644 index 82657ea467..0000000000 --- a/changelog.d/806.md +++ /dev/null @@ -1 +0,0 @@ -- Add a reform impacts dashboard planning docs page at `docs/book/validation/reform-impacts-dashboard-plan.md` covering how to build on the existing `reforms_config.yaml` fixtures and `test_parametric_reform_impacts.py` runner to publish a time series of canonical reform impacts across model and data versions, with annotations for EFO releases and dataset refreshes. diff --git a/docs/book/validation/reform-impacts-dashboard-plan.md b/docs/book/validation/reform-impacts-dashboard-plan.md index 1a6cdda0a1..38c1873c0d 100644 --- a/docs/book/validation/reform-impacts-dashboard-plan.md +++ b/docs/book/validation/reform-impacts-dashboard-plan.md @@ -6,7 +6,8 @@ which asks for a dashboard tracking the budgetary impact of high-importance reforms over time and across model / data versions. The repo already has the **fixtures** — what's missing is a published -view onto them. +view onto them. Updated October 2026: still not built; the dataset move +to Microcosm UK (below) makes it more useful. ``` ## What exists today @@ -27,13 +28,19 @@ shape: gov.dwp.universal_credit.means_test.reduction_rate: 0.2 ``` -Each fixture is exercised by -[`test_parametric_reform_impacts.py`](../../../policyengine_uk/tests/test_parametric_reform_impacts.py) -on every PR, and any drift outside the `tolerance` fails CI. So CI -already catches *unintended* impact changes — but only via a binary -pass/fail signal that flips silently between PRs. - -What's missing is a **time series**: the same nine reforms run on +Each fixture is run by +[`tests/microsimulation/test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py), +which compares the change in `gov_balance` in 2029 with +`expected_impact` (£bn), within `tolerance` (default £1bn). It runs in +CI's Test job on every PR, on the dataset CI is configured with, so CI +already catches *unintended* impact changes. But the signal is only +pass or fail, and when a change is intended, someone reruns +[`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py) +and the old value is overwritten. The fixtures are regression values +taken from the model's own output, not external costings (the file says +so at the top). + +What's missing is a **time series**: the same eight reforms run on every commit / version, plotted against the published OBR/RF benchmark for the same reform, with annotations for the relevant infrastructure changes (model versions, dataset refreshes, EFO @@ -109,11 +116,21 @@ analysis. Add to v2 over time as published OBR / RF reforms become available benchmarks: -- Remove the Universal Credit two-child limit (worked example in - [child-poverty-tcl.md](./child-poverty-tcl.md)). +- Remove the two-child limit (an external comparison is drafted in + [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724)). - Freeze the Personal Allowance for an additional year. - 1pp employer NI change. +## Why now + +The UK data pipeline is moving from the Enhanced FRS to +[Microcosm UK](https://github.com/PolicyEngine/microcosm), whose first +certified national release was published on 4 October 2026. Moving the +fixtures' dataset will shift every expected impact at once, and model +changes to VAT, fuel duty and other bases are landing at the same time. +A history that records the dataset alongside each value is the easiest +way to show which change moved which number. + ## What this doesn't replace This is a **regression-tracking** dashboard, not a forecast. The @@ -127,7 +144,7 @@ relevant pages are: - [nowcasting-comparison.md](../assumptions/nowcasting-comparison.md) (PolicyEngine vs RF). - [rf-nowcasting-methodology.md](../assumptions/rf-nowcasting-methodology.md) (RF methodology detail). -- [child-poverty-tcl.md](./child-poverty-tcl.md) (worked external comparison). +- [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724) (child poverty against OBR and Resolution Foundation benchmarks, in review). ## Open questions @@ -145,5 +162,5 @@ relevant pages are: - Issue: [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806). - Existing fixture list: [`reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml). -- Existing regression runner: [`test_parametric_reform_impacts.py`](../../../policyengine_uk/tests/test_parametric_reform_impacts.py). -- Visualisation helper used by neighbouring validation pages: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py). +- Existing regression runner: [`test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py). +- Script that rewrites the expected values: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py). From e2ceb6b042377698eb89331553dc66d9ba9d5233 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Mon, 12 Oct 2026 11:05:47 +0200 Subject: [PATCH 3/4] Bring the reform impacts dashboard plan up to date with CI and fixtures The microsimulation fixtures run only with dataset credentials and are required in the merge queue and on main. Describe the personal allowance and child benefit fixtures by what they set, drop the unverified Microcosm release date, and remove em dashes. Co-Authored-By: Claude Opus 5.5 --- .../reform-impacts-dashboard-plan.md | 36 +++++++++++-------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/book/validation/reform-impacts-dashboard-plan.md b/docs/book/validation/reform-impacts-dashboard-plan.md index 38c1873c0d..316ace197c 100644 --- a/docs/book/validation/reform-impacts-dashboard-plan.md +++ b/docs/book/validation/reform-impacts-dashboard-plan.md @@ -5,9 +5,9 @@ [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806), which asks for a dashboard tracking the budgetary impact of high-importance reforms over time and across model / data versions. -The repo already has the **fixtures** — what's missing is a published -view onto them. Updated October 2026: still not built; the dataset move -to Microcosm UK (below) makes it more useful. +The repo already has the **fixtures**; what's missing is a published +view onto them. Updated October 2026: still not built, and the planned +dataset move to Microcosm UK (below) makes it more useful. ``` ## What exists today @@ -31,9 +31,15 @@ shape: Each fixture is run by [`tests/microsimulation/test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py), which compares the change in `gov_balance` in 2029 with -`expected_impact` (£bn), within `tolerance` (default £1bn). It runs in -CI's Test job on every PR, on the dataset CI is configured with, so CI -already catches *unintended* impact changes. But the signal is only +`expected_impact` (£bn), within `tolerance` (default £1bn). The tests +are marked `microsimulation`, so they run in CI's test suite only when a +dataset is available: `policyengine_uk/tests/conftest.py` points +`POLICYENGINE_UK_DEFAULT_DATASET` at the Enhanced FRS 2023-24 file +(`enhanced_frs_2023_24.h5@1.40.3`) when a Hugging Face token is set. +The suite requires that data in the merge queue and on main, and skips +these tests on pull requests without credentials. So CI already +catches *unintended* impact changes before they reach main. But the +signal is only pass or fail, and when a change is intended, someone reruns [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py) and the old value is overwritten. The fixtures are regression values @@ -58,7 +64,7 @@ timestamp, policyengine_uk_version, policyengine_uk_data_version, dataset_url, reform_name, expected_impact, observed_impact, gap ``` -Append-only — no overwrites — so the history survives. +The table is append-only, with no overwrites, so the history survives. ### Computation @@ -71,14 +77,14 @@ per fixture. Schedule: weekly + on every release tag. Three publication paths, in order of effort: -1. **Static page in the docs site** — the simplest. Convert the CSV +1. **Static page in the docs site**: the simplest. Convert the CSV to a Plotly chart (matching the existing `validation/*.ipynb` pattern) and render it under `docs/book/validation/`. Updates when the docs site rebuilds. -2. **Living JSON endpoint** — push the latest aggregate JSON to a +2. **Living JSON endpoint**: push the latest aggregate JSON to a stable URL (HuggingFace dataset or a `gh-pages` artefact) so the policyengine.org app can render it without needing the docs build. -3. **policyengine.org dashboard tile** — a tile on the marketing site +3. **policyengine.org dashboard tile**: a tile on the marketing site that shows the latest impact alongside the OBR/RF benchmark. Most work; right answer once the static page proves the data is stable. @@ -102,8 +108,10 @@ content: 1. Raise basic rate by 1pp. 2. Raise higher rate by 1pp. -3. Raise personal allowance by ~£800/year. -4. Raise child benefit by £25/week per additional child. +3. Raise the personal allowance to £13,000 (the fixture's name says + "by ~800GBP/year", but the allowance is frozen at £12,570 to + 2030-31, so in 2029 the rise is £430). +4. Set the child benefit rate for each additional child to £25 a week. 5. Reduce Universal Credit taper to 20%. 6. Raise Class 1 main employee NI rate to 10%. 7. Raise VAT standard rate by 2pp. @@ -124,8 +132,8 @@ benchmarks: ## Why now The UK data pipeline is moving from the Enhanced FRS to -[Microcosm UK](https://github.com/PolicyEngine/microcosm), whose first -certified national release was published on 4 October 2026. Moving the +[Microcosm UK](https://github.com/PolicyEngine/microcosm), while the +fixtures still run on the Enhanced FRS 2023-24. Moving the fixtures' dataset will shift every expected impact at once, and model changes to VAT, fuel duty and other bases are landing at the same time. A history that records the dataset alongside each value is the easiest From 551399d02b264a11b0f22adb8ef6f718d9b4699d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 02:21:35 -0400 Subject: [PATCH 4/4] Bring the reform impacts dashboard plan up to date with main The page reached main through #2386 and its changelog line shipped in 2.143.0, so this drops the released .added fragment and corrects main's copy: - reforms_config.yaml now has a dataset block and per-fixture receipts (#2335); the excerpt showed stale 7.8 / -18.1 values. - The tests pin the certified Enhanced FRS 2024-25 (1.58.1) through certified_dataset.yaml, not enhanced_frs_2023_24.h5@1.40.3. - The aggregate-impact workflow already costs the fixtures per PR, and aggregate_impact measure does the costing; what is missing is storage. - "Raise higher rate by 1pp" sets 42%, a 2pp rise. - Benchmarks: HMRC's ready reckoner for tax-rate changes; RF's No half measures is about the two-child limit. - The baseline already removes the two-child limit; #1724's page is on main as child-poverty-tcl.md. - Notebooks need a myst.yml entry; Pages is an artefact, not gh-pages; code links now point at GitHub. Co-Authored-By: Claude Opus 5.5 --- ...m-impacts-dashboard-corrections.changed.md | 1 + changelog.d/806.added.md | 1 - .../reform-impacts-dashboard-plan.md | 138 ++++++++++++------ 3 files changed, 93 insertions(+), 47 deletions(-) create mode 100644 changelog.d/806-reform-impacts-dashboard-corrections.changed.md delete mode 100644 changelog.d/806.added.md diff --git a/changelog.d/806-reform-impacts-dashboard-corrections.changed.md b/changelog.d/806-reform-impacts-dashboard-corrections.changed.md new file mode 100644 index 0000000000..b2763a8274 --- /dev/null +++ b/changelog.d/806-reform-impacts-dashboard-corrections.changed.md @@ -0,0 +1 @@ +Updated the reform impacts dashboard plan (docs/book/validation/reform-impacts-dashboard-plan.md) for the fixture receipts, the certified Enhanced FRS 2024-25 pin and the per-pull-request aggregate-impact costings now on main, and corrected the higher-rate fixture, the external benchmarks and the two-child limit reform. diff --git a/changelog.d/806.added.md b/changelog.d/806.added.md deleted file mode 100644 index 3a5e0d597d..0000000000 --- a/changelog.d/806.added.md +++ /dev/null @@ -1 +0,0 @@ -Added a plan for a reform impacts dashboard (#806) at `docs/book/validation/reform-impacts-dashboard-plan.md`, tracking the regression fixtures in `reforms_config.yaml` over model and dataset versions. diff --git a/docs/book/validation/reform-impacts-dashboard-plan.md b/docs/book/validation/reform-impacts-dashboard-plan.md index 316ace197c..529b198808 100644 --- a/docs/book/validation/reform-impacts-dashboard-plan.md +++ b/docs/book/validation/reform-impacts-dashboard-plan.md @@ -4,53 +4,86 @@ **Planning page.** Tracks [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806), which asks for a dashboard tracking the budgetary impact of -high-importance reforms over time and across model / data versions. -The repo already has the **fixtures**; what's missing is a published -view onto them. Updated October 2026: still not built, and the planned -dataset move to Microcosm UK (below) makes it more useful. +high-importance reforms over time and across model versions. +The repo already has the **fixtures**, and each one now records its +earlier values with the model and data that produced them; what's +missing is a published view onto them. Updated October 2026: still not +built, and the planned dataset move to Microcosm UK (below) makes it more +useful. ``` ## What exists today -[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml) -is a list of canonical reform-impact regression fixtures with the -shape: +[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml) +names the dataset the fixtures were measured on and lists the canonical +reform-impact regression fixtures. Each fixture keeps its earlier values +under `receipts`, with the model and data that produced each. Abridged, +at policyengine-uk 2.148.2: ```yaml +dataset: + data_version: 1.58.1 + path: enhanced_frs_2024_25.h5 + revision: 2526edbbb687a3b54dbb11272e5389e4b62d2ac2 + sha256: 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784 +reforms: - name: Raise basic rate by 1pp - expected_impact: 7.8 # £bn fiscal impact at the reference year + expected_impact: 7.1 parameters: gov.hmrc.income_tax.rates.uk[0].rate: 0.21 + receipts: + - impact: 7.8 + data: enhanced_frs_2022_23, unpinned (latest at the time) + model: 2.34.5 (a1832dc77, #1205, 2025-07-11) + # ... four more receipts ... + - impact: 7.115 + data: 1.58.1 enhanced_frs_2024_25 + model: 2.135.7 (merged 7717438bf, main 7ad002f54, 2026-10-18) +# ... - name: Reduce Universal Credit taper rate to 20% - expected_impact: -18.1 + expected_impact: -22.0 tolerance: 3.0 parameters: gov.dwp.universal_credit.means_test.reduction_rate: 0.2 + receipts: + # ... ``` Each fixture is run by -[`tests/microsimulation/test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py), +[`policyengine_uk/tests/microsimulation/test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py), which compares the change in `gov_balance` in 2029 with `expected_impact` (£bn), within `tolerance` (default £1bn). The tests are marked `microsimulation`, so they run in CI's test suite only when a dataset is available: `policyengine_uk/tests/conftest.py` points -`POLICYENGINE_UK_DEFAULT_DATASET` at the Enhanced FRS 2023-24 file -(`enhanced_frs_2023_24.h5@1.40.3`) when a Hugging Face token is set. -The suite requires that data in the merge queue and on main, and skips -these tests on pull requests without credentials. So CI already -catches *unintended* impact changes before they reach main. But the -signal is only -pass or fail, and when a change is intended, someone reruns -[`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py) -and the old value is overwritten. The fixtures are regression values -taken from the model's own output, not external costings (the file says -so at the top). - -What's missing is a **time series**: the same eight reforms run on -every commit / version, plotted against the published OBR/RF -benchmark for the same reform, with annotations for the relevant -infrastructure changes (model versions, dataset refreshes, EFO -updates). +`POLICYENGINE_UK_DEFAULT_DATASET` at the certified dataset pinned in +`policyengine_uk/tests/certified_dataset.yaml` when a Hugging Face token +is set. That dataset is the Enhanced FRS 2024-25 +(`enhanced_frs_2024_25.h5` from policyengine-uk-data 1.58.1), pinned by +Hugging Face commit and checked against its SHA-256. The suite requires +that data in the merge queue and on main, and skips these tests on pull +requests without credentials. So CI already catches *unintended* impact +changes before they reach main. But the test itself is only pass or fail. +The `Aggregate impact` workflow +(`.github/workflows/aggregate-impact.yaml`) also costs these reforms on +each pull request's base and head for 2025-26 to 2031-32 and posts the +comparison as a pull request comment, but nothing keeps those results +once the pull request is done. When a change is intended, someone reruns +[`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py), +which overwrites `expected_impact`. The earlier values survive only +because each fixture also keeps a `receipts` list: the script does not +add to it, but a test requires the latest receipt to be on the pinned +dataset and to match `expected_impact`. The fixtures are regression +values taken from the model's own output, not external costings (the +file says so at the top). + +What's missing is a **time series** that keeps itself: the receipts hold +only the values someone chose to record, for 2029, and the +aggregate-impact comments aren't stored. The plan is the same eight +reforms run on every commit or version, plotted against a published +external costing where one exists (for the tax-rate changes, HMRC's +[ready reckoner](https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes)), +with annotations for the relevant infrastructure changes (model +versions, dataset refreshes, EFO updates). ## Proposed shape @@ -72,17 +105,21 @@ A new entry point under `tests/microsimulation/` (or, better, a GitHub Actions workflow that doesn't run inside the test suite) walks `reforms_config.yaml`, runs each reform, and writes a row per fixture. Schedule: weekly + on every release tag. +(`python -m policyengine_uk.utils.aggregate_impact measure` already does +the costing, for the eight fixtures and two more reforms; the new part is +running it on a schedule and keeping the results.) ### Publication Three publication paths, in order of effort: 1. **Static page in the docs site**: the simplest. Convert the CSV - to a Plotly chart (matching the existing `validation/*.ipynb` - pattern) and render it under `docs/book/validation/`. Updates - when the docs site rebuilds. + to a Plotly chart (as `validation/validation.ipynb` does) and render + it under `docs/book/validation/`. A notebook must be added to + `myst.yml` by name, since only Markdown pages are picked up by + folder. Updates when the docs site rebuilds. 2. **Living JSON endpoint**: push the latest aggregate JSON to a - stable URL (HuggingFace dataset or a `gh-pages` artefact) so the + stable URL (a HuggingFace dataset or a GitHub Pages artefact) so the policyengine.org app can render it without needing the docs build. 3. **policyengine.org dashboard tile**: a tile on the marketing site that shows the latest impact alongside the OBR/RF benchmark. Most @@ -93,9 +130,13 @@ Three publication paths, in order of effort: For each reform: - **Latest observed impact** (£bn, current model). -- **Published external benchmark** (OBR EFO costing where one - exists; RF *No half measures*-style costing as a comparator). Annotate - the publication date. +- **Published external benchmark** where one exists, with its + publication date: HMRC's ready reckoner (*Direct effects of + illustrative tax changes*) for the income tax, National Insurance and + VAT changes, and OBR or Resolution Foundation costings (such as RF's + *No half measures* on the two-child limit) for the policies they + cover. Any published figure must follow the aggregate-impact privacy + rules (`policyengine_uk/utils/aggregate_impact/privacy.py`). - **Time series** of observed impact across the last N model releases. - **Annotations** for each known regime shift: EFO release, dataset refresh, methodology PR. The reforms_config history is the natural @@ -107,7 +148,8 @@ Use the existing `reforms_config.yaml` entries as the v1 dashboard content: 1. Raise basic rate by 1pp. -2. Raise higher rate by 1pp. +2. Raise the higher rate to 42% (the fixture's name says "by 1pp", but + the rate is 40%, so the rise is 2pp). 3. Raise the personal allowance to £13,000 (the fixture's name says "by ~800GBP/year", but the allowance is frozen at £12,570 to 2030-31, so in 2029 the rise is £430). @@ -117,23 +159,26 @@ content: 7. Raise VAT standard rate by 2pp. 8. Raise additional rate by 3pp. -These cover three of the four big revenue heads (income tax, NI, VAT) -plus a UC parameter that matters disproportionately for distributional -analysis. +These cover income tax (four fixtures), National Insurance and VAT, +plus two benefit parameters: the Child Benefit rate for additional +children and the Universal Credit taper. Add to v2 over time as published OBR / RF reforms become available benchmarks: -- Remove the two-child limit (an external comparison is drafted in - [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724)). -- Freeze the Personal Allowance for an additional year. +- Reinstate the two-child limit. The baseline removes it from 6 April + 2026, and [child-poverty-tcl.md](child-poverty-tcl.md) compares its + removal with the DWP, Budget 2025 and Resolution Foundation estimates. +- Freeze the Personal Allowance for an additional year (to 2031-32, so + it needs a later reference year than 2029). - 1pp employer NI change. ## Why now The UK data pipeline is moving from the Enhanced FRS to [Microcosm UK](https://github.com/PolicyEngine/microcosm), while the -fixtures still run on the Enhanced FRS 2023-24. Moving the +fixtures still run on the Enhanced FRS (2024-25, from +policyengine-uk-data 1.58.1). Moving the fixtures' dataset will shift every expected impact at once, and model changes to VAT, fuel duty and other bases are landing at the same time. A history that records the dataset alongside each value is the easiest @@ -152,7 +197,7 @@ relevant pages are: - [nowcasting-comparison.md](../assumptions/nowcasting-comparison.md) (PolicyEngine vs RF). - [rf-nowcasting-methodology.md](../assumptions/rf-nowcasting-methodology.md) (RF methodology detail). -- [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724) (child poverty against OBR and Resolution Foundation benchmarks, in review). +- [child-poverty-tcl.md](child-poverty-tcl.md) (the child-poverty impact and cost of removing the two-child limit, against DWP, Budget 2025 and Resolution Foundation estimates). ## Open questions @@ -169,6 +214,7 @@ relevant pages are: ## References - Issue: [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806). -- Existing fixture list: [`reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml). -- Existing regression runner: [`test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py). -- Script that rewrites the expected values: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py). +- Existing fixture list: [`reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml). +- Existing regression runner: [`test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py). +- Script that rewrites the expected values: [`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py). +- Per-pull-request costings: [`.github/workflows/aggregate-impact.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/.github/workflows/aggregate-impact.yaml) and [`policyengine_uk/utils/aggregate_impact/`](https://github.com/PolicyEngine/policyengine-uk/tree/main/policyengine_uk/utils/aggregate_impact).