[DO NOT MERGE] Batch-3 fit attribution: seeded release without #571 at production settings (512 epochs)

.github/workflows/push.yaml

@@ -123,6 +123,10 @@ jobs:
           fetch-depth: 0 # Fetch all history for all tags and branches
       - name: Install uv
         uses: astral-sh/setup-uv@v8.1.0
+        # Restore uv's package cache but never save one from a job that
+        # holds the data: its location can be moved (UV_CACHE_DIR).
+        with:
+          save-cache: false
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -139,17 +143,11 @@ jobs:
           HUGGING_FACE_TOKEN: ${{ secrets.HUGGING_FACE_TOKEN }}
       - name: Build datasets
         run: uv run --frozen make data
-      - name: Save calibration log (constituencies)
-        uses: actions/upload-artifact@v7
-        with:
-          name: constituency_calibration_log.csv
-          path: constituency_calibration_log.csv
-
-      - name: Save calibration log (local authorities)
-        uses: actions/upload-artifact@v7
-        with:
-          name: la_calibration_log.csv
-          path: la_calibration_log.csv
+      # The raw calibration logs stay on the runner. They hold every target's
+      # weighted estimate at every logged epoch with no small-cell control, so a
+      # target supported by a few households, divided by a published count,
+      # could give one household's value back, and this repository's
+      # artifacts are public.
       - name: Run tests
         run: uv run --frozen make test
       - name: Upload data

.gitignore

@@ -18,6 +18,7 @@
 !policyengine_uk_data/datasets/local_areas/**/*.csv
 **/_build
 !policyengine_uk_data/storage/*.csv
+!policyengine_uk_data/tests/fixtures/ons_uk_population_2025.csv
 **/version.json
 # Build output: household weights by local area, derived from FRS microdata
 policyengine_uk_data/storage/local_geography_weights.csv.gz
@@ -26,3 +27,6 @@ policyengine_uk_data/storage/local_geography_weights.csv.gz
 
 # Hypothesis example database (property-based tests)
 .hypothesis/
+
+# Local build-job notes, caches and hand-off artefacts
+.job/

changelog.d/468.fixed.md

@@ -0,0 +1,10 @@
+Align local Universal Credit targets by area code, preserve constituencies' 2024 boundaries, and use each matched constituency's country for its children split.
+
+Restore the correct 2024 code order for the mapped income, age and housing targets of all 57 Scottish constituencies. This is a pure reorder; Scotland's totals are unchanged. Target moves from the previous positional assignment are:
+
+| Target | Rows moving >5% | Rows moving >25% | Maximum move |
+|---|---:|---:|---:|
+| Age population | 44 | 11 | 269.7% |
+| Housing anchor | 44 | 11 | 269.7% |
+| Employment income amount | 53 | 26 | 245% |
+| Self-employment income amount | 53 | 37 | 799% |

changelog.d/593-age-topcode.fixed.md

@@ -0,0 +1 @@
+Matched national and regional ONS age targets to the FRS top code with open-ended 80+ bands, including the projection's oldest age groups.

changelog.d/aggregate-data-comparison.added.md

@@ -0,0 +1 @@
+Compare each pull request build's national aggregates (receipts, spending, caseloads, poverty, deciles and the standard reform costings) with its base build's, measured with the locked model by policyengine-uk's aggregate-impact tool, and report them in the pull request comment.

changelog.d/deterministic-calibration.fixed.md

@@ -0,0 +1 @@
+- Seed the weight dropout in `calibrate_local_areas` with its own generator (`seed`, default 0). The dropout masks were drawn from torch's global generator, which torch seeds differently in every process, so two builds of the same commit with the same inputs gave different household weights and aggregates.

changelog.d/distribution-checks.added.md

@@ -0,0 +1 @@
+Measure each pull request build's HBAI income distribution (medians, means, Gini and relative low income by group against DWP's HBAI FYE 2025 estimates and intervals) and single-age population (against the ONS 2024-based projection), and report them against the base build in the pull request comment. They flag but do not fail yet.

changelog.d/final-calibration-weights.fixed.md

@@ -0,0 +1 @@
+Save and return the calibrated weights from the final optimiser step. Weights were saved only every tenth epoch, so a 512-epoch build shipped the state after epoch 510 of 0–511 and dropped the last update (#456).

changelog.d/fit-comparison.added.md

@@ -0,0 +1 @@
+Build each pull request's base the same way as its head, with the head's lockfile, and fail when the share of national or local targets within 10% falls by more than 2 percentage points over the targets both builds have (`targets/fit_policy.yaml`). National targets in a worse error band on the head are flagged in a pull request comment. The raw calibration logs stay on the runner; only each target's error band (within 10%, 10-25%, 25-50%, 50-100%, over 100%) is uploaded, with no estimate or error (`python -m policyengine_uk_data.utils.fit_report summarise` and `compare`).

changelog.d/frs-approved-training-trainee.fixed.md

@@ -0,0 +1 @@
+Code FRS dependants in unwaged government training as in approved training, from the TRAINEE flag on child-table records. The build read `train`, which is the question name of `TRAIN2` and not a column, so nobody was in approved training. In FRS 2024-25 that left all 20 dependants aged 16 to 19 outside full-time education (33.0k grossed, every one a trainee) failing the education-or-training condition of the UC, Pension Credit, Child Benefit and Child Tax Credit qualifying-young-person tests.

changelog.d/obr-ni-total-target.fixed.md

@@ -0,0 +1 @@
+Stop calibrating OBR total NICs (Table 3.8, £200bn in 2025-26) on `ni_employee`, which the Class 1 employee NICs target (£50bn) also used; the two targets on one matrix column could not both be met. The Table 3.4 class targets (employee, employer, self-employed) already cover every NIC class the data populates. Tests now pin the variable of each OBR receipts target and check that no two calibration targets sharing a plain-variable column disagree on a year's value.

changelog.d/obr-source-manifest.changed.md

@@ -0,0 +1 @@
+Build the OBR targets only from the committed EFO workbooks named, with their SHA-256 and title, in a vintage manifest (`targets/source_manifests/`). Read each table's financial years from its header row rather than fixed column letters, and fail the build when a manifest target or year is missing instead of logging and dropping it. Add `python -m policyengine_uk_data.targets.obr_sources ingest` for adopting a new EFO vintage, which prints every target's change; its `--dry-run` reports, and exits non-zero on, an ingested vintage, an interrupted ingest's leftovers or a download page off obr.uk, which the real ingest refuses.

changelog.d/scotland-council-tax-workbook.fixed.md

@@ -0,0 +1 @@
+Read the Scottish council tax band targets from the committed September 2025 chargeable-dwellings workbook, verified by its SHA-256, instead of downloading it and substituting a fixed distribution when gov.scot blocked the download (#596).

changelog.d/spi-tax-year-start.fixed.md

@@ -0,0 +1 @@
+Map the HMRC Survey of Personal Incomes 2023-24 outturn to PolicyEngine year 2023, the tax year it describes, instead of 2024. Return each SPI income band as one target that holds both the base year and the projected years, so calibration uses the projected values rather than the base-year outturn carried forward. Regenerate the SPI projections from the corrected base year with the current uprating factors. Defer the CGT year mapping and aggregate/band feasibility to a follow-up on top of #571.

changelog.d/stop-raw-calibration-log-artifacts.fixed.md

@@ -0,0 +1 @@
+Pull request and push builds no longer upload the raw constituency and local authority calibration logs as Actions artifacts, which are public on this repository; the logs stay on the runner.

changelog.d/target-coverage-gate.changed.md

@@ -0,0 +1 @@
+Fail the calibration build when a national target is dropped (its name already taken, no value for the year, no column, an error), has a non-finite value, or enters with no household contributing, unless `targets/known_gaps.yaml` lists it with a reason and an issue, for every build or only for testing or release builds. Add `python -m policyengine_uk_data.targets.coverage` to report every target's status on a dataset.

policyengine_uk_data/calibration/matrix_builder.py

@@ -189,14 +189,22 @@ def _load_area_targets(
         get_uk_total_population,
     )
 
-    y = pd.DataFrame()
     time_period = dataset.time_period
 
     # ── Income targets ──────────────────────────────────────────────
     if area_type == "constituency":
         incomes = get_constituency_income_targets()
+        age_targets = get_constituency_age_targets()
     else:
         incomes = get_la_income_targets()
+        age_targets = get_la_age_targets()
+
+    # SPI income and age inputs share this source order. Keep the codes
+    # through any boundary mapping before selecting the requested rows.
+    assert incomes["code"].equals(age_targets["code"]), (
+        "Income and age code order differs"
+    )
+    y = pd.DataFrame(index=pd.Index(age_targets["code"], name="code"))
 
     national_incomes = get_national_income_projections(int(time_period))
 
@@ -215,11 +223,6 @@ def _load_area_targets(
         )
 
     # ── Age targets ─────────────────────────────────────────────────
-    if area_type == "constituency":
-        age_targets = get_constituency_age_targets()
-    else:
-        age_targets = get_la_age_targets()
-
     uk_total_population = get_uk_total_population(int(time_period))
     age_cols = [c for c in age_targets.columns if c.startswith("age/")]
 
@@ -231,31 +234,40 @@ def _load_area_targets(
     for col in age_cols:
         y[col] *= uk_total_population / targets_total_pop * 0.9
 
+    # ── Boundary mapping (constituency 2010 → 2024 only) ───────────
+    if area_type == "constituency":
+        from policyengine_uk_data.datasets.local_areas.constituencies.boundary_changes.mapping_matrix import (
+            mapping_matrix,
+            old_codes,
+            new_codes,
+        )
+
+        y = pd.DataFrame(
+            mapping_matrix @ y.loc[old_codes].to_numpy(),
+            index=pd.Index(new_codes, name="code"),
+            columns=y.columns,
+        )
+
+    # Both the OA assignment matrix and the LA extras use area_codes order.
+    y = y.loc[area_codes["code"]]
+
     # ── UC targets ──────────────────────────────────────────────────
+    # Constituency UC counts are already on 2024 boundaries and must stay
+    # outside the mapping applied to the older income and age sources.
     if area_type == "constituency":
         from policyengine_uk_data.targets.sources.local_uc import (
             get_constituency_uc_targets,
             get_constituency_uc_by_children_targets,
         )
 
-        y["uc_households"] = get_constituency_uc_targets().values
-        uc_by_children = get_constituency_uc_by_children_targets()
+        y["uc_households"] = get_constituency_uc_targets().loc[y.index]
+        uc_by_children = get_constituency_uc_by_children_targets().loc[y.index]
         for col in uc_by_children.columns:
-            y[col] = uc_by_children[col].values
+            y[col] = uc_by_children[col]
     else:
         from policyengine_uk_data.targets.sources.local_uc import get_la_uc_targets
 
-        y["uc_households"] = get_la_uc_targets().values
-
-    # ── Boundary mapping (constituency 2010 → 2024 only) ───────────
-    if area_type == "constituency":
-        from policyengine_uk_data.datasets.local_areas.constituencies.boundary_changes.mapping_matrix import (
-            mapping_matrix,
-        )
-
-        y_columns = list(y.columns)
-        y_values = mapping_matrix @ y.values
-        y = pd.DataFrame(y_values, columns=y_columns)
+        y["uc_households"] = get_la_uc_targets().loc[y.index]
 
     # ── LA extras: ONS income, tenure, rent ─────────────────────────
     if area_type == "la":
@@ -264,8 +276,7 @@ def _load_area_targets(
             load_household_counts,
             load_tenure_data,
             load_private_rents,
-            UPRATING_NET_INCOME_BHC_2020_TO_2025,
-            UPRATING_HOUSING_COSTS_2020_TO_2025,
+            get_ons_income_uprating_factors,
         )
 
         original_weights = sim.calculate("household_weight", 2025).values
@@ -288,16 +299,19 @@ def _load_area_targets(
             "equiv_hbai_household_net_income_ahc"
         ).values
         housing_costs = hbai_net_income - hbai_net_income_ahc
+        income_bhc_uprating_factor, housing_costs_uprating_factor = (
+            get_ons_income_uprating_factors(int(time_period))
+        )
 
         ons_merged["equiv_net_income_bhc_target"] = (
             ons_merged["net_income_bhc"]
             * ons_merged["households"]
-            * UPRATING_NET_INCOME_BHC_2020_TO_2025
+            * income_bhc_uprating_factor
         )
         ons_merged["equiv_housing_costs_target"] = (
             ons_merged["net_income_bhc"] * ons_merged["households"]
             - ons_merged["net_income_ahc"] * ons_merged["households"]
-        ) * UPRATING_HOUSING_COSTS_2020_TO_2025
+        ) * housing_costs_uprating_factor
         ons_merged["equiv_net_income_ahc_target"] = (
             ons_merged["equiv_net_income_bhc_target"]
             - ons_merged["equiv_housing_costs_target"]
@@ -385,7 +399,7 @@ def _load_area_targets(
             national_rent * la_household_share,
         )
 
-    return y
+    return y.reset_index(drop=True)
 
 
 def create_cloned_target_matrix(

policyengine_uk_data/datasets/frs.py

@@ -112,8 +112,11 @@
     "UPPER_SECONDARY",
     "POST_SECONDARY",
 )
-# FRS government-training question variants use 10 or 13 for "None of these".
-FRS_APPROVED_TRAINING_CODES = tuple(range(1, 10))
+# FRS TRAINEE, "Whether on a government scheme for employment training". It
+# shares the FRS dictionary's value format with EDUCFT, where 1 is yes; in each
+# of FRS 2020-21, 2022-23, 2023-24 and 2024-25 the child-table records aged 16
+# to 19 outside full-time education are exactly the child-table records with 1.
+FRS_TRAINEE_YES = 1
 UNKNOWN_QUALIFYING_EDUCATION_OR_TRAINING_ENTRY_AGE = 1000
 # FRS RENTPROF: whether ROYYR1 is a profit (1) or a loss (2).
 FRS_RENTPROF_LOSS = 2
@@ -366,14 +369,34 @@ def derive_is_in_non_advanced_education(
 
 def derive_is_in_approved_training_from_frs_person(
     person: pd.DataFrame,
+    adult_person_ids,
 ) -> pd.Series:
-    """Identify reported government training scheme participation in FRS."""
-
-    if "train" not in person.columns:
-        return pd.Series(False, index=person.index)
+    """Identify FRS dependants in unwaged government training.
+
+    The FRS counts a 16- to 19-year-old as a dependent child when they are in
+    full-time non-advanced education or unwaged government training. In each
+    of FRS 2020-21, 2022-23, 2023-24 and 2024-25, every child-table record
+    aged 16 to 19 outside full-time education carries TRAINEE = 1.
+
+    Leaving adult-table records out is a proxy. The FRS has already decided
+    they are not dependants, and their TRAINEE answer does not say the training
+    is unwaged, so it may be training under a contract of employment, which is
+    not approved training (UC Regulations 2013, reg. 5(3)). TRAIN2 names the
+    scheme but only the adult table has it.
+
+    The build fails when TRAINEE is missing rather than marking nobody as in
+    training: an earlier version read TRAIN2's question name, ``train``, which
+    is not a column, so every record was coded as not in training.
+    """
 
-    train = pd.to_numeric(person.train, errors="coerce").fillna(0)
-    return train.isin(FRS_APPROVED_TRAINING_CODES)
+    if "trainee" not in person.columns:
+        raise KeyError(
+            "The FRS person table has no TRAINEE column, so approved training "
+            "cannot be derived."
+        )
+    is_trainee = pd.to_numeric(person.trainee, errors="coerce").eq(FRS_TRAINEE_YES)
+    is_adult_record = person.person_id.isin(np.asarray(adult_person_ids))
+    return is_trainee & ~is_adult_record
 
 
 def derive_age_started_or_accepted_current_education_or_training(
@@ -1055,7 +1078,7 @@ def determine_education_level(fted_val, typeed2_val, age_val):
         pe_person.current_education
     )
     pe_person["is_in_approved_training"] = (
-        derive_is_in_approved_training_from_frs_person(person)
+        derive_is_in_approved_training_from_frs_person(person, frs["adult"].person_id)
     )
     pe_person["age_started_or_accepted_current_education_or_training"] = (
         derive_age_started_or_accepted_current_education_or_training(

policyengine_uk_data/datasets/local_areas/constituencies/boundary_changes/mapping_matrix.py

@@ -5,6 +5,19 @@
 # 1. Read the boundary change data into a DataFrame
 df = pd.read_csv(Path(__file__).parent / "boundary_changes.csv")
 
+# Five unchanged Scottish constituencies retain their 2010 codes in the
+# boundary-change source. Use their 2024 codes from constituencies_2024.csv
+# so the mapped rows can be joined to the other 2024 targets by code.
+df["code_2024"] = df["code_2024"].replace(
+    {
+        "S14000006": "S14000107",  # Ayr, Carrick and Cumnock
+        "S14000008": "S14000108",  # Berwickshire, Roxburgh and Selkirk
+        "S14000010": "S14000109",  # Central Ayrshire
+        "S14000040": "S14000110",  # Kilmarnock and Loudoun
+        "S14000058": "S14000111",  # West Aberdeenshire and Kincardine
+    }
+)
+
 # 2. Get unique constituency codes for 2010 and 2024, sorted alphabetically
 old_codes = sorted(df["code_2010"].unique())
 new_codes = sorted(df["code_2024"].unique())