Update the reform impacts dashboard docs page (#806)

changelog.d/806-reform-impacts-dashboard-corrections.changed.md

@@ -0,0 +1 @@
+Updated the reform impacts dashboard plan (docs/book/validation/reform-impacts-dashboard-plan.md) for the fixture receipts, the certified Enhanced FRS 2024-25 pin and the per-pull-request aggregate-impact costings now on main, and corrected the higher-rate fixture, the external benchmarks and the two-child limit reform.

docs/book/validation/reform-impacts-dashboard-plan.md

@@ -4,53 +4,86 @@
 **Planning page.** Tracks
 [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806),
 which asks for a dashboard tracking the budgetary impact of
-high-importance reforms over time and across model / data versions.
-The repo already has the **fixtures**; what's missing is a published
-view onto them. Updated October 2026: still not built, and the planned
-dataset move to Microcosm UK (below) makes it more useful.
+high-importance reforms over time and across model versions.
+The repo already has the **fixtures**, and each one now records its
+earlier values with the model and data that produced them; what's
+missing is a published view onto them. Updated October 2026: still not
+built, and the planned dataset move to Microcosm UK (below) makes it more
+useful.
 ```
 
 ## What exists today
 
-[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml)
-is a list of canonical reform-impact regression fixtures with the
-shape:
+[`policyengine_uk/tests/microsimulation/reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml)
+names the dataset the fixtures were measured on and lists the canonical
+reform-impact regression fixtures. Each fixture keeps its earlier values
+under `receipts`, with the model and data that produced each. Abridged,
+at policyengine-uk 2.148.2:
 
 ```yaml
+dataset:
+  data_version: 1.58.1
+  path: enhanced_frs_2024_25.h5
+  revision: 2526edbbb687a3b54dbb11272e5389e4b62d2ac2
+  sha256: 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784
+reforms:
 - name: Raise basic rate by 1pp
-  expected_impact: 7.8     # £bn fiscal impact at the reference year
+  expected_impact: 7.1
   parameters:
     gov.hmrc.income_tax.rates.uk[0].rate: 0.21
+  receipts:
+  - impact: 7.8
+    data: enhanced_frs_2022_23, unpinned (latest at the time)
+    model: 2.34.5 (a1832dc77, #1205, 2025-07-11)
+  # ... four more receipts ...
+  - impact: 7.115
+    data: 1.58.1 enhanced_frs_2024_25
+    model: 2.135.7 (merged 7717438bf, main 7ad002f54, 2026-10-18)
+# ...
 - name: Reduce Universal Credit taper rate to 20%
-  expected_impact: -18.1
+  expected_impact: -22.0
   tolerance: 3.0
   parameters:
     gov.dwp.universal_credit.means_test.reduction_rate: 0.2
+  receipts:
+  # ...
 ```
 
 Each fixture is run by
-[`tests/microsimulation/test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py),
+[`policyengine_uk/tests/microsimulation/test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py),
 which compares the change in `gov_balance` in 2029 with
 `expected_impact` (£bn), within `tolerance` (default £1bn). The tests
 are marked `microsimulation`, so they run in CI's test suite only when a
 dataset is available: `policyengine_uk/tests/conftest.py` points
-`POLICYENGINE_UK_DEFAULT_DATASET` at the Enhanced FRS 2023-24 file
-(`enhanced_frs_2023_24.h5@1.40.3`) when a Hugging Face token is set.
-The suite requires that data in the merge queue and on main, and skips
-these tests on pull requests without credentials. So CI already
-catches *unintended* impact changes before they reach main. But the
-signal is only
-pass or fail, and when a change is intended, someone reruns
-[`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py)
-and the old value is overwritten. The fixtures are regression values
-taken from the model's own output, not external costings (the file says
-so at the top).
-
-What's missing is a **time series**: the same eight reforms run on
-every commit / version, plotted against the published OBR/RF
-benchmark for the same reform, with annotations for the relevant
-infrastructure changes (model versions, dataset refreshes, EFO
-updates).
+`POLICYENGINE_UK_DEFAULT_DATASET` at the certified dataset pinned in
+`policyengine_uk/tests/certified_dataset.yaml` when a Hugging Face token
+is set. That dataset is the Enhanced FRS 2024-25
+(`enhanced_frs_2024_25.h5` from policyengine-uk-data 1.58.1), pinned by
+Hugging Face commit and checked against its SHA-256. The suite requires
+that data in the merge queue and on main, and skips these tests on pull
+requests without credentials. So CI already catches *unintended* impact
+changes before they reach main. But the test itself is only pass or fail.
+The `Aggregate impact` workflow
+(`.github/workflows/aggregate-impact.yaml`) also costs these reforms on
+each pull request's base and head for 2025-26 to 2031-32 and posts the
+comparison as a pull request comment, but nothing keeps those results
+once the pull request is done. When a change is intended, someone reruns
+[`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py),
+which overwrites `expected_impact`. The earlier values survive only
+because each fixture also keeps a `receipts` list: the script does not
+add to it, but a test requires the latest receipt to be on the pinned
+dataset and to match `expected_impact`. The fixtures are regression
+values taken from the model's own output, not external costings (the
+file says so at the top).
+
+What's missing is a **time series** that keeps itself: the receipts hold
+only the values someone chose to record, for 2029, and the
+aggregate-impact comments aren't stored. The plan is the same eight
+reforms run on every commit or version, plotted against a published
+external costing where one exists (for the tax-rate changes, HMRC's
+[ready reckoner](https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes)),
+with annotations for the relevant infrastructure changes (model
+versions, dataset refreshes, EFO updates).
 
 ## Proposed shape
 
@@ -72,17 +105,21 @@ A new entry point under `tests/microsimulation/` (or, better, a
 GitHub Actions workflow that doesn't run inside the test suite)
 walks `reforms_config.yaml`, runs each reform, and writes a row
 per fixture. Schedule: weekly + on every release tag.
+(`python -m policyengine_uk.utils.aggregate_impact measure` already does
+the costing, for the eight fixtures and two more reforms; the new part is
+running it on a schedule and keeping the results.)
 
 ### Publication
 
 Three publication paths, in order of effort:
 
 1. **Static page in the docs site**: the simplest. Convert the CSV
-   to a Plotly chart (matching the existing `validation/*.ipynb`
-   pattern) and render it under `docs/book/validation/`. Updates
-   when the docs site rebuilds.
+   to a Plotly chart (as `validation/validation.ipynb` does) and render
+   it under `docs/book/validation/`. A notebook must be added to
+   `myst.yml` by name, since only Markdown pages are picked up by
+   folder. Updates when the docs site rebuilds.
 2. **Living JSON endpoint**: push the latest aggregate JSON to a
-   stable URL (HuggingFace dataset or a `gh-pages` artefact) so the
+   stable URL (a HuggingFace dataset or a GitHub Pages artefact) so the
    policyengine.org app can render it without needing the docs build.
 3. **policyengine.org dashboard tile**: a tile on the marketing site
    that shows the latest impact alongside the OBR/RF benchmark. Most
@@ -93,9 +130,13 @@ Three publication paths, in order of effort:
 For each reform:
 
 - **Latest observed impact** (£bn, current model).
-- **Published external benchmark** (OBR EFO costing where one
-  exists; RF *No half measures*-style costing as a comparator). Annotate
-  the publication date.
+- **Published external benchmark** where one exists, with its
+  publication date: HMRC's ready reckoner (*Direct effects of
+  illustrative tax changes*) for the income tax, National Insurance and
+  VAT changes, and OBR or Resolution Foundation costings (such as RF's
+  *No half measures* on the two-child limit) for the policies they
+  cover. Any published figure must follow the aggregate-impact privacy
+  rules (`policyengine_uk/utils/aggregate_impact/privacy.py`).
 - **Time series** of observed impact across the last N model releases.
 - **Annotations** for each known regime shift: EFO release, dataset
   refresh, methodology PR. The reforms_config history is the natural
@@ -107,7 +148,8 @@ Use the existing `reforms_config.yaml` entries as the v1 dashboard
 content:
 
 1. Raise basic rate by 1pp.
-2. Raise higher rate by 1pp.
+2. Raise the higher rate to 42% (the fixture's name says "by 1pp", but
+   the rate is 40%, so the rise is 2pp).
 3. Raise the personal allowance to £13,000 (the fixture's name says
    "by ~800GBP/year", but the allowance is frozen at £12,570 to
    2030-31, so in 2029 the rise is £430).
@@ -117,23 +159,26 @@ content:
 7. Raise VAT standard rate by 2pp.
 8. Raise additional rate by 3pp.
 
-These cover three of the four big revenue heads (income tax, NI, VAT)
-plus a UC parameter that matters disproportionately for distributional
-analysis.
+These cover income tax (four fixtures), National Insurance and VAT,
+plus two benefit parameters: the Child Benefit rate for additional
+children and the Universal Credit taper.
 
 Add to v2 over time as published OBR / RF reforms become available
 benchmarks:
 
-- Remove the two-child limit (an external comparison is drafted in
-  [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724)).
-- Freeze the Personal Allowance for an additional year.
+- Reinstate the two-child limit. The baseline removes it from 6 April
+  2026, and [child-poverty-tcl.md](child-poverty-tcl.md) compares its
+  removal with the DWP, Budget 2025 and Resolution Foundation estimates.
+- Freeze the Personal Allowance for an additional year (to 2031-32, so
+  it needs a later reference year than 2029).
 - 1pp employer NI change.
 
 ## Why now
 
 The UK data pipeline is moving from the Enhanced FRS to
 [Microcosm UK](https://github.com/PolicyEngine/microcosm), while the
-fixtures still run on the Enhanced FRS 2023-24. Moving the
+fixtures still run on the Enhanced FRS (2024-25, from
+policyengine-uk-data 1.58.1). Moving the
 fixtures' dataset will shift every expected impact at once, and model
 changes to VAT, fuel duty and other bases are landing at the same time.
 A history that records the dataset alongside each value is the easiest
@@ -152,7 +197,7 @@ relevant pages are:
 
 - [nowcasting-comparison.md](../assumptions/nowcasting-comparison.md) (PolicyEngine vs RF).
 - [rf-nowcasting-methodology.md](../assumptions/rf-nowcasting-methodology.md) (RF methodology detail).
-- [#1724](https://github.com/PolicyEngine/policyengine-uk/pull/1724) (child poverty against OBR and Resolution Foundation benchmarks, in review).
+- [child-poverty-tcl.md](child-poverty-tcl.md) (the child-poverty impact and cost of removing the two-child limit, against DWP, Budget 2025 and Resolution Foundation estimates).
 
 ## Open questions
 
@@ -169,6 +214,7 @@ relevant pages are:
 ## References
 
 - Issue: [#806](https://github.com/PolicyEngine/policyengine-uk/issues/806).
-- Existing fixture list: [`reforms_config.yaml`](../../../policyengine_uk/tests/microsimulation/reforms_config.yaml).
-- Existing regression runner: [`test_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/test_reform_impacts.py).
-- Script that rewrites the expected values: [`update_reform_impacts.py`](../../../policyengine_uk/tests/microsimulation/update_reform_impacts.py).
+- Existing fixture list: [`reforms_config.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/reforms_config.yaml).
+- Existing regression runner: [`test_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/test_reform_impacts.py).
+- Script that rewrites the expected values: [`update_reform_impacts.py`](https://github.com/PolicyEngine/policyengine-uk/blob/main/policyengine_uk/tests/microsimulation/update_reform_impacts.py).
+- Per-pull-request costings: [`.github/workflows/aggregate-impact.yaml`](https://github.com/PolicyEngine/policyengine-uk/blob/main/.github/workflows/aggregate-impact.yaml) and [`policyengine_uk/utils/aggregate_impact/`](https://github.com/PolicyEngine/policyengine-uk/tree/main/policyengine_uk/utils/aggregate_impact).