[DO NOT MERGE] Batch-3 fit attribution: release with #571 but no CGT prior raking, production settings

.github/workflows/push.yaml

@@ -123,6 +123,10 @@ jobs:
           fetch-depth: 0 # Fetch all history for all tags and branches
       - name: Install uv
         uses: astral-sh/setup-uv@v8.1.0
+        # Restore uv's package cache but never save one from a job that
+        # holds the data: its location can be moved (UV_CACHE_DIR).
+        with:
+          save-cache: false
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -139,17 +143,11 @@ jobs:
           HUGGING_FACE_TOKEN: ${{ secrets.HUGGING_FACE_TOKEN }}
       - name: Build datasets
         run: uv run --frozen make data
-      - name: Save calibration log (constituencies)
-        uses: actions/upload-artifact@v7
-        with:
-          name: constituency_calibration_log.csv
-          path: constituency_calibration_log.csv
-
-      - name: Save calibration log (local authorities)
-        uses: actions/upload-artifact@v7
-        with:
-          name: la_calibration_log.csv
-          path: la_calibration_log.csv
+      # The raw calibration logs stay on the runner. They hold every target's
+      # weighted estimate at every logged epoch with no small-cell control, so a
+      # target supported by a few households, divided by a published count,
+      # could give one household's value back, and this repository's
+      # artifacts are public.
       - name: Run tests
         run: uv run --frozen make test
       - name: Upload data

.gitignore

@@ -18,6 +18,7 @@
 !policyengine_uk_data/datasets/local_areas/**/*.csv
 **/_build
 !policyengine_uk_data/storage/*.csv
+!policyengine_uk_data/tests/fixtures/ons_uk_population_2025.csv
 **/version.json
 # Build output: household weights by local area, derived from FRS microdata
 policyengine_uk_data/storage/local_geography_weights.csv.gz
@@ -26,3 +27,6 @@ policyengine_uk_data/storage/local_geography_weights.csv.gz
 
 # Hypothesis example database (property-based tests)
 .hypothesis/
+
+# Local build-job notes, caches and hand-off artefacts
+.job/

changelog.d/468.fixed.md

@@ -0,0 +1,10 @@
+Align local Universal Credit targets by area code, preserve constituencies' 2024 boundaries, and use each matched constituency's country for its children split.
+
+Restore the correct 2024 code order for the mapped income, age and housing targets of all 57 Scottish constituencies. This is a pure reorder; Scotland's totals are unchanged. Target moves from the previous positional assignment are:
+
+| Target | Rows moving >5% | Rows moving >25% | Maximum move |
+|---|---:|---:|---:|
+| Age population | 44 | 11 | 269.7% |
+| Housing anchor | 44 | 11 | 269.7% |
+| Employment income amount | 53 | 26 | 245% |
+| Self-employment income amount | 53 | 37 | 799% |

changelog.d/593-age-topcode.fixed.md

@@ -0,0 +1 @@
+Matched national and regional ONS age targets to the FRS top code with open-ended 80+ bands, including the projection's oldest age groups.

changelog.d/aggregate-data-comparison.added.md

@@ -0,0 +1 @@
+Compare each pull request build's national aggregates (receipts, spending, caseloads, poverty, deciles and the standard reform costings) with its base build's, measured with the locked model by policyengine-uk's aggregate-impact tool, and report them in the pull request comment.

changelog.d/capital-gains-hmrc-2024-25.fixed.md

@@ -0,0 +1 @@
+- Calibrate capital gains to HMRC's 2024-25 outturn for individuals (Capital Gains Tax statistics, 2026 release: 548,000 taxpayers with £119.3bn of gains above the annual exempt amount) and start the calibration on it. The 1.58.0 build targeted the 2023-24 all-taxpayer headline (£65.9bn, including trusts) and ended 19% under even that, with £53bn of gains above the exempt amount and a further -£15bn of losses on 1.4m people. Its calibration started 17 times over on CGT taxpayers, because the has-gains copy of each zero-weight SPI household received a full prior weight. Capital gains are now the last imputation, so a has-gains copy matches its original in everything but the gain. Each household splits its calibration prior into complementary original and twin parts using the Advani-Summers propensity for its income band before rejecting small draws, zero-weight households included; both parts are carried through geography cloning and uprating into calibration; draws at or under the exempt amount, including losses, carry no gains; donors cover only the size bands from £500,000 up; and the calibration scales its starting weights so every HMRC size band's taxpayer count begins on target. The aggregate targets are now the sums of the size-band targets.

changelog.d/deterministic-calibration.fixed.md

@@ -0,0 +1 @@
+- Seed the weight dropout in `calibrate_local_areas` with its own generator (`seed`, default 0). The dropout masks were drawn from torch's global generator, which torch seeds differently in every process, so two builds of the same commit with the same inputs gave different household weights and aggregates.

changelog.d/distribution-checks.added.md

@@ -0,0 +1 @@
+Measure each pull request build's HBAI income distribution (medians, means, Gini and relative low income by group against DWP's HBAI FYE 2025 estimates and intervals) and single-age population (against the ONS 2024-based projection), and report them against the base build in the pull request comment. They flag but do not fail yet.

changelog.d/final-calibration-weights.fixed.md

@@ -0,0 +1 @@
+Save and return the calibrated weights from the final optimiser step. Weights were saved only every tenth epoch, so a 512-epoch build shipped the state after epoch 510 of 0–511 and dropped the last update (#456).

changelog.d/fit-comparison.added.md

@@ -0,0 +1 @@
+Build each pull request's base the same way as its head, with the head's lockfile, and fail when the share of national or local targets within 10% falls by more than 2 percentage points over the targets both builds have (`targets/fit_policy.yaml`). National targets in a worse error band on the head are flagged in a pull request comment. The raw calibration logs stay on the runner; only each target's error band (within 10%, 10-25%, 25-50%, 50-100%, over 100%) is uploaded, with no estimate or error (`python -m policyengine_uk_data.utils.fit_report summarise` and `compare`).

changelog.d/frs-approved-training-trainee.fixed.md

@@ -0,0 +1 @@
+Code FRS dependants in unwaged government training as in approved training, from the TRAINEE flag on child-table records. The build read `train`, which is the question name of `TRAIN2` and not a column, so nobody was in approved training. In FRS 2024-25 that left all 20 dependants aged 16 to 19 outside full-time education (33.0k grossed, every one a trainee) failing the education-or-training condition of the UC, Pension Credit, Child Benefit and Child Tax Credit qualifying-young-person tests.

changelog.d/obr-ni-total-target.fixed.md

@@ -0,0 +1 @@
+Stop calibrating OBR total NICs (Table 3.8, £200bn in 2025-26) on `ni_employee`, which the Class 1 employee NICs target (£50bn) also used; the two targets on one matrix column could not both be met. The Table 3.4 class targets (employee, employer, self-employed) already cover every NIC class the data populates. Tests now pin the variable of each OBR receipts target and check that no two calibration targets sharing a plain-variable column disagree on a year's value.

changelog.d/obr-source-manifest.changed.md

@@ -0,0 +1 @@
+Build the OBR targets only from the committed EFO workbooks named, with their SHA-256 and title, in a vintage manifest (`targets/source_manifests/`). Read each table's financial years from its header row rather than fixed column letters, and fail the build when a manifest target or year is missing instead of logging and dropping it. Add `python -m policyengine_uk_data.targets.obr_sources ingest` for adopting a new EFO vintage, which prints every target's change; its `--dry-run` reports, and exits non-zero on, an ingested vintage, an interrupted ingest's leftovers or a download page off obr.uk, which the real ingest refuses.

changelog.d/scotland-council-tax-workbook.fixed.md

@@ -0,0 +1 @@
+Read the Scottish council tax band targets from the committed September 2025 chargeable-dwellings workbook, verified by its SHA-256, instead of downloading it and substituting a fixed distribution when gov.scot blocked the download (#596).

changelog.d/spi-tax-year-start.fixed.md

@@ -0,0 +1 @@
+Map the HMRC Survey of Personal Incomes 2023-24 outturn to PolicyEngine year 2023, the tax year it describes, instead of 2024. Return each SPI income band as one target that holds both the base year and the projected years, so calibration uses the projected values rather than the base-year outturn carried forward. Regenerate the SPI projections from the corrected base year with the current uprating factors. Defer the CGT year mapping and aggregate/band feasibility to a follow-up on top of #571.

changelog.d/stop-raw-calibration-log-artifacts.fixed.md

@@ -0,0 +1 @@
+Pull request and push builds no longer upload the raw constituency and local authority calibration logs as Actions artifacts, which are public on this repository; the logs stay on the runner.

changelog.d/target-coverage-gate.changed.md

@@ -0,0 +1 @@
+Fail the calibration build when a national target is dropped (its name already taken, no value for the year, no column, an error), has a non-finite value, or enters with no household contributing, unless `targets/known_gaps.yaml` lists it with a reason and an issue, for every build or only for testing or release builds. Add `python -m policyengine_uk_data.targets.coverage` to report every target's status on a dataset.

docs/imputations.md

@@ -11,9 +11,9 @@ The imputations are applied in this order (dependencies noted):
 3. **VAT** (from ETB)
 4. **Public Services** (from ETB)
 5. **Income** (from SPI)
-6. **Capital Gains** (from Advani-Summers data)
-7. **Salary Sacrifice** (from FRS subsample)
-8. **Student Loan Plan** (rule-based, from age)
+6. **Salary Sacrifice** (from FRS subsample)
+7. **Student Loan Plan** (rule-based, from age)
+8. **Capital Gains** (from Advani-Summers data and HMRC statistics) — last, after Pension Credit take-up is assigned, because it stacks a second record for each household with gains
 
 ---
 
@@ -160,14 +160,15 @@ Imputes detailed income components to create "synthetic taxpayers" with higher i
 
 ## Capital Gains Imputation
 
-**Source:** Advani-Summers capital gains distribution data
+**Sources:** Advani-Summers capital gains distribution data; HMRC Capital Gains Tax statistics, Table 2.1a (individuals, by size of gain)
 
-Uses a gradient-based optimization approach rather than QRF. The dataset is doubled, with one half receiving imputed capital gains amounts. Weights are then optimized to match the empirical relationship between total income and capital gains incidence.
+Not a QRF. Advani and Summers give, for each band of total income, the share of people with gains and the percentiles of those gains. HMRC gives the number of individuals with a CGT liability, and their gains, by size of gain. The imputation takes who holds gains from the first and how many hold how much from the second.
 
 ### Method
-1. Double the dataset (original + clone)
-2. Assign capital gains to one adult per household in the cloned half
-3. Optimize blend weights to match income-band capital gains incidence from Advani-Summers data
+1. Each household's first adult draws a gain from their income band's Advani-Summers distribution.
+2. Where the draw is above the lowest HMRC size band limit (the £3,000 annual exempt amount), the household is stacked again as a has-gains twin carrying the gain. The split propensity adjusts Advani-Summers' share of all people in the income band for assignment to first adults only. This is the propensity before rejecting draws of £3,000 or less; rejecting those draws lowers realised incidence. For a zero-weight (SPI synthetic) household, complementary original and twin calibration priors are carried into calibration. The prior-equivalent weight sets only the twin's household weight, while the original's household weight remains zero. Smaller draws, including losses, get no twin: HMRC publishes liable taxpayers only, so nothing would anchor them.
+3. The spline stops at each band's 95th percentile, so few draws exceed £500,000 and none exceeds about £2m. For each HMRC size band from £500,000 up, 30 donor households are stacked carrying the band's published mean gain.
+4. Calibration targets the taxpayer count and the gains of each HMRC size band, and their totals. Before the first epoch it scales the starting weights so that each band's taxpayer count is on its target.
 
 ---
 

policyengine_uk_data/calibration/clone_and_assign.py

@@ -80,8 +80,8 @@ def clone_and_assign(
     The OA's region, LA and constituency therefore never
     contradict the household's ``region``.
 
-    Household weights are divided by n_clones so aggregate
-    population totals are preserved.
+    Household weights and any carried calibration priors are divided by
+    n_clones so their aggregate totals are preserved.
 
     Args:
         dataset: Input FRS dataset (post-imputation).
@@ -195,6 +195,10 @@ def clone_and_assign(
             hh_clone["source_year"] = dataset.time_period
         hh_clone["household_id"] = new_hh_ids
         hh_clone["household_weight"] = hh["household_weight"].values / n_clones
+        if "household_calibration_prior" in hh:
+            hh_clone["household_calibration_prior"] = (
+                hh["household_calibration_prior"].values / n_clones
+            )
 
         # Add geography columns from assignment
         # Initialise with empty strings for all households

policyengine_uk_data/calibration/matrix_builder.py

@@ -189,14 +189,22 @@ def _load_area_targets(
         get_uk_total_population,
     )
 
-    y = pd.DataFrame()
     time_period = dataset.time_period
 
     # ── Income targets ──────────────────────────────────────────────
     if area_type == "constituency":
         incomes = get_constituency_income_targets()
+        age_targets = get_constituency_age_targets()
     else:
         incomes = get_la_income_targets()
+        age_targets = get_la_age_targets()
+
+    # SPI income and age inputs share this source order. Keep the codes
+    # through any boundary mapping before selecting the requested rows.
+    assert incomes["code"].equals(age_targets["code"]), (
+        "Income and age code order differs"
+    )
+    y = pd.DataFrame(index=pd.Index(age_targets["code"], name="code"))
 
     national_incomes = get_national_income_projections(int(time_period))
 
@@ -215,11 +223,6 @@ def _load_area_targets(
         )
 
     # ── Age targets ─────────────────────────────────────────────────
-    if area_type == "constituency":
-        age_targets = get_constituency_age_targets()
-    else:
-        age_targets = get_la_age_targets()
-
     uk_total_population = get_uk_total_population(int(time_period))
     age_cols = [c for c in age_targets.columns if c.startswith("age/")]
 
@@ -231,31 +234,40 @@ def _load_area_targets(
     for col in age_cols:
         y[col] *= uk_total_population / targets_total_pop * 0.9
 
+    # ── Boundary mapping (constituency 2010 → 2024 only) ───────────
+    if area_type == "constituency":
+        from policyengine_uk_data.datasets.local_areas.constituencies.boundary_changes.mapping_matrix import (
+            mapping_matrix,
+            old_codes,
+            new_codes,
+        )
+
+        y = pd.DataFrame(
+            mapping_matrix @ y.loc[old_codes].to_numpy(),
+            index=pd.Index(new_codes, name="code"),
+            columns=y.columns,
+        )
+
+    # Both the OA assignment matrix and the LA extras use area_codes order.
+    y = y.loc[area_codes["code"]]
+
     # ── UC targets ──────────────────────────────────────────────────
+    # Constituency UC counts are already on 2024 boundaries and must stay
+    # outside the mapping applied to the older income and age sources.
     if area_type == "constituency":
         from policyengine_uk_data.targets.sources.local_uc import (
             get_constituency_uc_targets,
             get_constituency_uc_by_children_targets,
         )
 
-        y["uc_households"] = get_constituency_uc_targets().values
-        uc_by_children = get_constituency_uc_by_children_targets()
+        y["uc_households"] = get_constituency_uc_targets().loc[y.index]
+        uc_by_children = get_constituency_uc_by_children_targets().loc[y.index]
         for col in uc_by_children.columns:
-            y[col] = uc_by_children[col].values
+            y[col] = uc_by_children[col]
     else:
         from policyengine_uk_data.targets.sources.local_uc import get_la_uc_targets
 
-        y["uc_households"] = get_la_uc_targets().values
-
-    # ── Boundary mapping (constituency 2010 → 2024 only) ───────────
-    if area_type == "constituency":
-        from policyengine_uk_data.datasets.local_areas.constituencies.boundary_changes.mapping_matrix import (
-            mapping_matrix,
-        )
-
-        y_columns = list(y.columns)
-        y_values = mapping_matrix @ y.values
-        y = pd.DataFrame(y_values, columns=y_columns)
+        y["uc_households"] = get_la_uc_targets().loc[y.index]
 
     # ── LA extras: ONS income, tenure, rent ─────────────────────────
     if area_type == "la":
@@ -264,8 +276,7 @@ def _load_area_targets(
             load_household_counts,
             load_tenure_data,
             load_private_rents,
-            UPRATING_NET_INCOME_BHC_2020_TO_2025,
-            UPRATING_HOUSING_COSTS_2020_TO_2025,
+            get_ons_income_uprating_factors,
         )
 
         original_weights = sim.calculate("household_weight", 2025).values
@@ -288,16 +299,19 @@ def _load_area_targets(
             "equiv_hbai_household_net_income_ahc"
         ).values
         housing_costs = hbai_net_income - hbai_net_income_ahc
+        income_bhc_uprating_factor, housing_costs_uprating_factor = (
+            get_ons_income_uprating_factors(int(time_period))
+        )
 
         ons_merged["equiv_net_income_bhc_target"] = (
             ons_merged["net_income_bhc"]
             * ons_merged["households"]
-            * UPRATING_NET_INCOME_BHC_2020_TO_2025
+            * income_bhc_uprating_factor
         )
         ons_merged["equiv_housing_costs_target"] = (
             ons_merged["net_income_bhc"] * ons_merged["households"]
             - ons_merged["net_income_ahc"] * ons_merged["households"]
-        ) * UPRATING_HOUSING_COSTS_2020_TO_2025
+        ) * housing_costs_uprating_factor
         ons_merged["equiv_net_income_ahc_target"] = (
             ons_merged["equiv_net_income_bhc_target"]
             - ons_merged["equiv_housing_costs_target"]
@@ -385,7 +399,7 @@ def _load_area_targets(
             national_rent * la_household_share,
         )
 
-    return y
+    return y.reset_index(drop=True)
 
 
 def create_cloned_target_matrix(