diff --git a/changelog.d/1063-release-cut-blockers.fixed.md b/changelog.d/1063-release-cut-blockers.fixed.md index 7bce8c6ba..dc80298c5 100644 --- a/changelog.d/1063-release-cut-blockers.fixed.md +++ b/changelog.d/1063-release-cut-blockers.fixed.md @@ -1 +1 @@ -Clear the UK release-cut certifier's blockers found by the rehearsal on the 2026-09-30 national build (microcosm#1063 c9). The candidate label is `microcosm_uk_2024_25`; the certifier and the full-build gate evidence read the export surface through `uk_export_candidate_columns`, which strips the structural id columns and carries `household.household_weight` as the enhanced-FRS reference does, so ids are no longer reported as unreviewed extras nor the weight as missing. Every declared stage output the incumbent never carried is allow-listed (ruling 2026-10-02): the source and support-channel lineage of the three entities, the council tax family, the reported-benefit inputs the take-up stages read, the SPI channel's `hmrc_spi_*` leaves, the FRS education and housing fields, the two fenced raw FRS codes and `other_investment_income`. The one reviewed export exclusion, `person.incapacity_benefit_reported`, and the five internal disability carriers (`attendance_allowance_reported`, `dla_sc_reported`, `dla_m_reported`, `pip_m_reported`, `pip_dl_reported`, stage-internal inputs the export registers keep off the release) now leave at the release boundary: `uk_release_export_frame` drops it from the written candidate, the graph readback expects the same frame, and the spine keeps the column for the SPI income stage. The FRS 2024-25 vintage's recoded training variable (`TRAIN2`, where 9 is "None of these" and 10 the Kickstart Scheme) is read, so `is_in_approved_training` carries signal again. The LCFS donor's diary consumption is floored at the declared `floor` (0) before the imputation, the support-clip ranges and the rake see it, with a receipt of the rows raised; `uk_stage_lcfs_consumption_support` declares the floor and requires the receipt, no remaining negative donor row and no clip range below it (250 release rows had inherited a minimum of -14,165 in housing, water and electricity consumption). The two FRS education inputs the enhanced-FRS reference barely carries (`adult_ema`, `dfe_education_spending`) enter the per-reference input-mass register, and the sparse charity-gifts leaf re-enters the QRF-tail register, each a signed receipt expiring 2027-04-02. `uk_stage_child_benefit_take_up` declares `population_fact_check` like the Pension Credit gate, so a synthetic smoke build records its failure and continues, and the anchor composition check allows the absolute rounding tolerance the other mass identities use. +Clear the UK release-cut certifier's blockers found by the rehearsal on the 2026-09-30 national build (microcosm#1063 c9). The candidate label is `microcosm_uk_2024_25`; the certifier and the full-build gate evidence read the export surface through `uk_export_candidate_columns`, which strips the structural id columns and carries `household.household_weight` as the enhanced-FRS reference does, so ids are no longer reported as unreviewed extras nor the weight as missing. Every declared stage output the incumbent never carried is allow-listed (ruling 2026-10-02): the source and support-channel lineage of the three entities, the council tax family, the reported-benefit inputs the take-up stages read, the SPI channel's `hmrc_spi_*` leaves, the FRS education and housing fields, the two fenced raw FRS codes and `other_investment_income`. The one reviewed export exclusion, `person.incapacity_benefit_reported`, and the five internal disability carriers (`attendance_allowance_reported`, `dla_sc_reported`, `dla_m_reported`, `pip_m_reported`, `pip_dl_reported`, stage-internal inputs the export registers keep off the release) now leave at the release boundary: `uk_release_export_frame` drops it from the written candidate, the graph readback expects the same frame, and the spine keeps the column for the SPI income stage. The FRS 2024-25 vintage's recoded training variable (`TRAIN2`, where 9 is "None of these" and 10 the Kickstart Scheme) is read, so `is_in_approved_training` carries signal again. The LCFS donor's diary consumption is floored at the declared `floor` (0) before the imputation, the support-clip ranges and the rake see it, with a receipt of the rows raised; `uk_stage_lcfs_consumption_support` declares the floor and requires the receipt, no remaining negative donor row and no clip range below it (250 release rows had inherited negative housing, water and electricity consumption). The two FRS education inputs the enhanced-FRS reference barely carries (`adult_ema`, `dfe_education_spending`) enter the per-reference input-mass register, and the sparse charity-gifts leaf re-enters the QRF-tail register, each a signed receipt expiring 2027-04-02. `uk_stage_child_benefit_take_up` declares `population_fact_check` like the Pension Credit gate, so a synthetic smoke build records its failure and continues, and the anchor composition check allows the absolute rounding tolerance the other mass identities use. diff --git a/changelog.d/1110-prepayment-meter-input.added.md b/changelog.d/1110-prepayment-meter-input.added.md new file mode 100644 index 000000000..8e6777b20 --- /dev/null +++ b/changelog.d/1110-prepayment-meter-input.added.md @@ -0,0 +1 @@ +The UK `lcfs_consumption` stage draws `uses_energy_prepayment_meter`, the household input policyengine-uk reads from 2.123.3 for the Energy Price Guarantee prepayment discount, July 2023 to March 2024 (microcosm#1110; PolicyEngine/policyengine-uk#2098; uk-data#539 tracks the same input). An LCFS donor prepays when it last put money on an electricity, gas or combined key or meter (b1751, b1701, b233, which the LCF asks only of prepayment-meter households) or its diary records a slot-meter payment (c45114, c45214): electricity-only prepayment counts, and a household with no meter payment, including one that did not answer, does not. The declared `impute_prepayment_meter` step fits that flag with a weighted regime-gated QRF on the chain's predictors and draws it at uniforms keyed on the household id, so no other draw moves; its receipt records the donor's and the frame's weighted shares and the Great Britain households flagged. The release exports the column: policyengine-uk reads it from 2.123.3 and an older engine, the locked one among them, ignores it; it is a reviewed extra column of the export surface until the incumbent carries it (uk-data#539), and an engine-lane test checks an engine that reads it reads a household boolean. On the K25 spine the frame flags about 3.0 million Great Britain households (10.5 %) at prior weights, against the around three million prepayment households the government counted in July 2023, when the discount began: a different reference period, and households in both, where Ofgem and DESNZ count meters or customers. diff --git a/changelog.d/1164-etb-vat-disposable-income.fixed.md b/changelog.d/1164-etb-vat-disposable-income.fixed.md new file mode 100644 index 000000000..d741500f4 --- /dev/null +++ b/changelog.d/1164-etb-vat-disposable-income.fixed.md @@ -0,0 +1 @@ +The UK `etb_vat` stage draws the full-rate VAT expenditure rate on disposable income on the ETB's own concept, at uniforms keyed on the household id (microcosm#1164; the predictor item microcosm#1225 left out). The donor's income is ETB `disinc`, gross income less income tax, employees' National Insurance and council tax, before indirect taxes; the recipient's was the engine's `household_net_income`, which also takes off the indirect taxes and keeps the in-kind NHS and education spending, and which read recipients about a quarter richer than the donor at the median on the K25 spine. The recipient's income is now its gross cash income (HBAI net income with policyengine-uk's HBAI deductions added back) less the same direct taxes, from `uk_runtime.household_income.etb_disposable_income`, whose distribution sits closest to the donor's of the concepts tried. A declared `uprate_donor_columns` step carries the 2023 donor's income to the 2024 base year on the engine's average earnings index, as `etb_services` carries the same `disinc`. The QRF now draws at quantile and sign uniforms keyed on the household id under the declared `salt`, where the model's generator drew in row order, so a household keeps its rate however the frame is ordered. diff --git a/changelog.d/1164-lcfs-2024-25.changed.md b/changelog.d/1164-lcfs-2024-25.changed.md new file mode 100644 index 000000000..203fa583c --- /dev/null +++ b/changelog.d/1164-lcfs-2024-25.changed.md @@ -0,0 +1 @@ +The UK `lcfs_consumption` stage reads the Living Costs and Food Survey 2024-25 (UK Data Service SN 9603, 2nd edition) in place of 2023-24 (SN 9468), so every UK survey stage is now on its latest microdata (microcosm#1164). The diary covers April 2024 to March 2025, the FRS 2024-25 spine's year: the CPI bridge to the calendar-2024 basis steps back a quarter instead of forward three, and the income predictors need no step. The donor has about 5,000 households against about 4,200. The 2nd edition's household tab (`dvhh_uk_anon_v3_2024.tab`) carries the annual weight as `cal_weight_annual`, the name `weighta` had, and its 1st-edition `dvhh_uk_anon_v2_2024.tab` carries no weight; no other variable the stage reads changed name or coding, and COICOP 2018 is not yet adopted. The tabs carry their study number and DOI in the spec, the fit is named `uk_lcfs_2024_25_consumption`, the support bounds are cut from the 2024-25 donor, the road-fuel level's other-fuels share is 0.46 percent of COICOP 07.2.2 on the new diary (0.37 on the old), and the road-fuel target's scope note says so. diff --git a/changelog.d/1164-lcfs-donor-floor-gaps.fixed.md b/changelog.d/1164-lcfs-donor-floor-gaps.fixed.md new file mode 100644 index 000000000..71f664abd --- /dev/null +++ b/changelog.d/1164-lcfs-donor-floor-gaps.fixed.md @@ -0,0 +1 @@ +The UK LCFS donor floor now covers every donor spend it reads (microcosm#1164). `bus_fare_spending`, summed from three diary lines a refund can also make negative, is floored with the fifteen diary consumption columns. The road-fuel level's other-fuels share reads the c72211–c72213 lines floored the same way. `tools/build_uk_e6_support_bounds.py` applies the stage's declared floor before reading donor ranges, so the committed housing, water and electricity bound starts at 0 like the stage's support; no other bound moves. diff --git a/changelog.d/1164-lcfs-energy-split-labelled-codes.fixed.md b/changelog.d/1164-lcfs-energy-split-labelled-codes.fixed.md new file mode 100644 index 000000000..d65b75f03 --- /dev/null +++ b/changelog.d/1164-lcfs-energy-split-labelled-codes.fixed.md @@ -0,0 +1 @@ +The UK LCFS stage splits the diary's fuel, light and power (p537) into electricity and gas by its labelled parts, from the LCF User Guide's derived-variable labels and P537 definition (microcosm#1164). The derivation it replaces, ported from the incumbent uk-data pipeline, read b226 as electricity paid by direct debit and b489 and b490 as prepayment-meter payments. The labels make b226 the gas component of a combined utilities bill (b227 is the electricity one) and b489 and b490 the electricity and gas account payments less rebates (b175 less b178, b170 less b173), so it gave combined-bill households, most of the donor, their gas as electricity; took gas away from electricity for households with two accounts; split an electricity account between the fuels; and folded oil and solid fuels into the drawn energy. Each fuel is now its account payment less its rebate (b489, b490 as the tab carries them; the stage refuses a donor where either equals its payment less its rebate for under 95 % of that fuel's account payers, which pins the labels and the rebate's sign), its prepayment key or meter payment, its part of a combined bill or meter payment, and the diary's second-dwelling account and slot-meter lines. A combined payment left unsplit, net of the combined bill's rebate (b174), goes by the household's own electricity share where it pays for both fuels and by the donor's mean share of dual-fuel households otherwise, since a combined payment means it pays for both; a donor with no dual-fuel household fails rather than taking a fixed share. Oil, bottled gas, solid fuels and heat stay in the housing total. A `donor_energy_split` receipt records the donor's electricity, gas and other fuels against p537, what reading refunds as zero adds, the weighted share of donors whose unclipped parts differ from their p537, and the account identity shares. Negative amounts (refunds) are read as zero. The NEED rake and the DESNZ level still set each fuel's level, but which households carry electricity and which gas now follows what they reported. diff --git a/changelog.d/1164-lcfs-gas-connection-connect.fixed.md b/changelog.d/1164-lcfs-gas-connection-connect.fixed.md new file mode 100644 index 000000000..3da49cac3 --- /dev/null +++ b/changelog.d/1164-lcfs-gas-connection-connect.fixed.md @@ -0,0 +1 @@ +The UK LCFS stage connects households to gas where its draw leaves a region below the published DESNZ share (microcosm#1164). The connection step only disconnected: it walked gas-positive households down to each region's share of gas meters over electricity meters. The old energy split gave gas to 98 % of LCFS donors, so every region could be cut to its share; the labelled split gives gas to about 80 %, against the about 85 % of homes DESNZ counts gas meters for, so nine of the eleven Great Britain regions fell short and the frame's gas-connected share fell to 79 %. The `price_domestic_energy` step now declares `connect_rule: gas_payer_probability_order`: two weighted fits on the chain's predictors give each zero-gas household the probability that a donor like it pays for gas (a gradient-boosted classifier) and the gas such a donor pays (a regime-gated QRF of the payers' gas spend), and in a region below its share the zero-gas households are connected in an identity-keyed order drawn in proportion to that probability, by the disconnection walk's weight-fitting rule, until the share is met. The NEED rake, the household ceilings and the DESNZ level then run as before. The `energy_rake` gate requires the receipt to name the declared rule and salt and waives a region's shortfall only where it ran out of households to connect. diff --git a/changelog.d/1164-lcfs-household-kwh-ceilings.fixed.md b/changelog.d/1164-lcfs-household-kwh-ceilings.fixed.md new file mode 100644 index 000000000..762065515 --- /dev/null +++ b/changelog.d/1164-lcfs-household-kwh-ceilings.fixed.md @@ -0,0 +1 @@ +The UK LCFS stage no longer leaves a household's electricity or gas above the domestic meter definitions behind the DESNZ total it levels to (microcosm#1164; the extreme energy values item of #1207; María's ruling of 2026-10-12). NEED's quality-assurance annex counts a gas meter as domestic under 73,200 kWh a year and an electricity meter under 50,000 kWh on profile classes 1 and 2. The energy columns are exempt from the donor support clip because the NEED rake and the DESNZ level set them, so a few drawn households sat far above any domestic use, and since #1167 the housing total contained them. The `price_domestic_energy` step declares `household_kwh_ceiling`; after the level, `cap_household_kwh` sets a household above it to the ceiling and re-rakes the households below it in its own NEED cells to the levelled targets with the capped ones held, then restores the DESNZ total. Handed to every household in proportion instead, the kWh moved the cells that hold most capped households (detached and high-income homes) off their NEED means. The `energy_rake` receipt counts the households capped and the share of the total they gave up, the NEED fit is read after the cap, and the `energy_rake` gate checks the declared ceilings, that no household is left above them and that the total held. diff --git a/changelog.d/1164-lcfs-recomposed-totals-donor-maximum.fixed.md b/changelog.d/1164-lcfs-recomposed-totals-donor-maximum.fixed.md new file mode 100644 index 000000000..7aa0b00fc --- /dev/null +++ b/changelog.d/1164-lcfs-recomposed-totals-donor-maximum.fixed.md @@ -0,0 +1 @@ +The UK LCFS stage holds each recomposed total, housing and transport, to the donor's realised maximum of that total, the support every drawn column is clipped to (microcosm#1164). The recomposition writes a total as its draw less the drawn parts plus the levelled parts, so a remainder near the top of the diary's range plus a levelled energy bill larger than the drawn one could leave the committed support bound the terminal `uk_support` gate holds the emitted totals to; on the 2024-25 diary, whose housing bound is far tighter than 2023-24's, some households did. The `recompose_from_remainder` step declares `maximum_total: donor_support`; a held total never falls below its parts, the excess comes out of its remainder, and the receipt counts the rows held and the weighted mass taken, which the `recomposed_totals` gate's identities include. diff --git a/changelog.d/1164-lcfs-unmapped-codes-refused.fixed.md b/changelog.d/1164-lcfs-unmapped-codes-refused.fixed.md new file mode 100644 index 000000000..66757f936 --- /dev/null +++ b/changelog.d/1164-lcfs-unmapped-codes-refused.fixed.md @@ -0,0 +1 @@ +The UK `lcfs_consumption` stage refuses an LCFS region (`gorx`), tenure (`a122`) or accommodation (`a121`) code it does not map, instead of dropping the donor (microcosm#1164). The donor table ended in a `.dropna()` that silently lost any household whose code fell outside the stage's maps, and the derive step's `logged_dropna_row_count` was never implemented, so a survey year that recoded one of these variables would have shrunk the donor without a trace. The derive step now declares `unmapped_codes: refuse`; the error names each unknown code, or a blank, without counting rows, and any other missing donor value stops the stage too. Every LCFS 2023-24 and 2024-25 code maps (the stage runs on both with the refusal in place), so neither donor loses a household. diff --git a/changelog.d/1164-need-band-gross-income.fixed.md b/changelog.d/1164-need-band-gross-income.fixed.md new file mode 100644 index 000000000..ba7beab8f --- /dev/null +++ b/changelog.d/1164-need-band-gross-income.fixed.md @@ -0,0 +1 @@ +The UK LCFS stage bands recipient households for the NEED energy rake on gross cash income, the concept the donor (LCFS p344p) and NEED band on (microcosm#1164). No spine stage wrote a gross income, so the recipient fell back to its HBAI net income and sat in too low a NEED income band. The engine's `household_gross_income` is not the concept either: it adds capital gains and the imputed value of NHS, education and transport spending, about £15,000 a household on average, which put the recipient in too high a band (a 7.8 % NEED residual on the K25 spine). The recipient's gross income is now its HBAI net income with the deductions policyengine-uk takes to reach it added back (council tax and domestic rates, income tax, National Insurance, student loan repayments, employee and personal pension contributions, maintenance and child payments, LVT), from the new `uk_runtime.household_income` helper; the stage materializes those for the band alone, and an engine-lane test fails if the engine's HBAI deduction list changes. On the K25 spine its distribution sits close to the donor's (weighted medians about £46k and £49k). The rake then converges at 2.40 % electricity and 2.82 % gas in the income margin, a cross-margin gap with NEED's other margins, so the `energy_rake` gate's residual rule moves from 2.5 % to 3.5 % on María's ruling of 2026-10-12. Electricity and gas by income move on every row; the DESNZ level holds. diff --git a/changelog.d/1164-nts-income-band-hbai-gross.fixed.md b/changelog.d/1164-nts-income-band-hbai-gross.fixed.md new file mode 100644 index 000000000..c5004f560 --- /dev/null +++ b/changelog.d/1164-nts-income-band-hbai-gross.fixed.md @@ -0,0 +1 @@ +The UK `nts_bus_travel` stage bands recipient households on gross cash income, the concept of the NTS household income band, rather than the engine's `household_gross_income` (microcosm#1164). The engine's figure adds capital gains and the imputed value of NHS, education and transport spending, which put about a fifth of households (pensioners, families with school-age children and capital-gains households above all) a band higher than their cash income, and the NTS shows bus use falling with income. The recipient's band income is now its HBAI net income with policyengine-uk's HBAI deductions added back, the same `uk_runtime.household_income` helper the LCFS NEED band uses, and the stage declares those engine variables. Bus-use bands, trips and the bus fares and bus support the LCFS and ETB services stages price from them move. diff --git a/changelog.d/1164-receipt-counts-banded.fixed.md b/changelog.d/1164-receipt-counts-banded.fixed.md new file mode 100644 index 000000000..536e89e9c --- /dev/null +++ b/changelog.d/1164-receipt-counts-banded.fixed.md @@ -0,0 +1 @@ +The UK LCFS and ETB VAT stages band the donor counts their receipts record at source, since a stage receipt is build output that can leave the machine (microcosm#1164, from the review of microcosm#1260). The new `uk_runtime.receipt_counts` rounds donor sizes and training rows, writing one to nine as "<10" and rounding to the nearest ten below a thousand and the nearest hundred above: the road-fuel and gas-payer training rows and the fuel-flag donor households. Counts of rows picked out by an extreme value are banded as "<10", "10-49", "50-99", then to the nearest hundred: the ETB VAT rows dropped and capped, and the LCFS donor floor's rows raised, per column and in total. The ETB VAT receipt records no donor size, which beside the donor year's sample size would give the dropped count back. An unweighted share of donors is floored to two decimals, since an exact share gives back its count, and a weighted share carried by fewer than ten donors is written "<10". A count a gate must read as an exact zero stays exact. diff --git a/changelog.d/1164-support-bounds-no-rerounding.fixed.md b/changelog.d/1164-support-bounds-no-rerounding.fixed.md new file mode 100644 index 000000000..c342fe9cc --- /dev/null +++ b/changelog.d/1164-support-bounds-no-rerounding.fixed.md @@ -0,0 +1 @@ +`tools/build_uk_e6_support_bounds.py` refuses to move a committed rounded support bound while the file's donor tab digests are unchanged (microcosm#1164). Each bound is a donor extreme rounded outward; re-derived from the same tabs after a change in the declared uprating, the cleaning or the floor, a bound that rounds to a different value narrows the extreme between the two thresholds. A lower bound may still move to zero, the floor, which reads no donor value. diff --git a/changelog.d/1225-etb-vat-rate-share-bounds.fixed.md b/changelog.d/1225-etb-vat-rate-share-bounds.fixed.md new file mode 100644 index 000000000..7fe69a7ce --- /dev/null +++ b/changelog.d/1225-etb-vat-rate-share-bounds.fixed.md @@ -0,0 +1 @@ +The UK `etb_vat` stage keeps `full_rate_vat_expenditure_rate` a share of expenditure after VAT, in [0, 1] (microcosm#1225, the ETB half of uk-data#572; shipped with microcosm#1164). The rate is `totvat × (1 − reduced-rate share) / standard rate` over `expdis − totvat`, and the stage refused only a zero denominator. So the ETB 2023 donor kept the households whose VAT exceeds their disposable expenditure, with negative rates, and the few whose little expenditure after VAT gives a rate above one, and the QRF drew both onto recipients. The derive step now declares `rate_bounds: [0, 1]`; it drops donors with no expenditure after VAT, caps the rest at one, and fails on a negative `totvat`. A `donor_rate` receipt counts the rows dropped and capped. The output is declared non-negative, and the committed support bound is the declared range rather than a rounded donor extreme. diff --git a/docs/evidence/uk-lcfs-2024-25-1164/README.md b/docs/evidence/uk-lcfs-2024-25-1164/README.md new file mode 100644 index 000000000..50671cc28 --- /dev/null +++ b/docs/evidence/uk-lcfs-2024-25-1164/README.md @@ -0,0 +1,75 @@ +# LCFS 2024-25 and the consumption-stage fixes: measurement scripts and outputs + +Evidence for microcosm#1164: the `lcfs_consumption` stage moved from the Living Costs and Food Survey 2023-24 (UKDS SN 9468) to 2024-25 (SN 9603, 2nd edition), with the fixes that ship with it. `receipts.json` carries the digests of the files here and of the spine the runs measured. + +## What was measured + +- **Stage-only runs** (`scripts/stage_dev_run.py`, `scripts/run_stage_dev_at.sh`). Each tree's own `lcfs_consumption` stage, on the LCFS tabs its own spec pins, re-run on one finished K25 spine (about 64,000 households, built on 2026-10-10 with policyengine-uk 2.122.2, before #1145's `property_components` stage), with every lcfs stage-health gate run on its receipts: + - `stage_dev_run.control-2023-24-final.json`: the base tree, #1200's head (a50ffba5e, which carries #1145 and #1198), on LCFS 2023-24; + - `stage_dev_run.treatment-2024-25-final.json`: this branch, on LCFS 2024-25. +- **The NTS-then-LCFS chain** (`scripts/nts_lcfs_chain_run.py`). `nts_bus_travel` then `lcfs_consumption` on the same spine, because the LCFS stage prices bus fares from the NTS trips: `nts_lcfs_chain_run.control.json` and `nts_lcfs_chain_run.treatment.json`. +- **The ETB VAT stage** (`scripts/etb_vat_dev_run.py`). `etb_vat` on the same spine, each tree's own stage on the ETB tab its spec pins, with the stage's support gate: `etb_vat_dev_run.control.json` and `etb_vat_dev_run.treatment.json`. + +Both arms run the same spine and policyengine-uk 2.123.0, so the comparison isolates this branch's stages. The spine already holds the stages after the LCFS stage, so compare the arms, not a run with the spine's stored columns (`before_spine`), which mixes in redraw drift. The R1 remeasure on fresh spines follows. + +## Results + +Every LCFS and NTS stage-health gate passes on both arms, and no household is left outside a committed support bound. Prior-weighted annual totals, £bn: + +| Column | Base (LCFS 2023-24) | This branch (LCFS 2024-25) | +|---|---|---| +| `food_and_non_alcoholic_beverages_consumption` | 109.6 | 113.8 | +| `alcohol_and_tobacco_consumption` | 21.8 | 19.7 | +| `clothing_and_footwear_consumption` | 32.2 | 30.1 | +| `housing_water_and_electricity_consumption` | 171.4 | 179.5 | +| `household_furnishings_consumption` | 94.9 | 80.2 | +| `health_consumption` | 19.5 | 23.5 | +| `transport_consumption` | 185.3 | 187.2 | +| `communication_consumption` | 24.6 | 25.5 | +| `recreation_consumption` | 139.5 | 183.0 | +| `education_consumption` | 4.6 | 20.3 | +| `restaurants_and_hotels_consumption` | 83.1 | 85.4 | +| `miscellaneous_consumption` | 84.3 | 97.4 | +| Twelve divisions | 970.6 | 1,045.6 | +| `electricity_consumption` | 29.5 | 29.6 | +| `gas_consumption` | 19.2 | 19.2 | +| `petrol_spending` | 25.6 | 26.1 | +| `diesel_spending` | 11.3 | 10.7 | +| `bus_fare_spending` | 4.1 | 4.1 | + +- **The diary year.** The 2024-25 diary records more spending than 2023-24: its own weighted twelve-division total rises about 7 % at base-year prices, with health (+58 %), education (+74 %) and recreation (+14 %) leading in current prices. The division capture against ONS Consumer Trends 2024 (domestic concept less the classes a diary does not record) rises from 0.73 to 0.78 overall. Furnishings moves from 1.25 to 1.05; recreation from 0.85 to 1.12, above ONS, and education from 0.11 to 0.48. The chain's draws run above the donor's own weighted totals on several divisions (recreation and education about 1.5 times), the over-draw follow-up. +- **The bridge.** The diary columns now step a quarter back to calendar 2024 (factor below one) where they stepped three quarters forward, and income predictors and the litre proxies take a factor of one. The recomposition's uprating residue falls from £1.05bn (housing) and £1.91bn (transport) to −£0.46bn and −£0.24bn. +- **Energy.** The labelled energy split reads the donor's electricity at £30.1bn and gas at £20.7bn a year, with £3.2bn of other fuels (oil, bottled gas, solid fuels) left in the housing remainder. Its parts sum to 0.6 % above the diary's fuel, light and power (P537): 0.2 points are refunds read as zero, and 0.4 points are the unclipped parts differing from P537, for 8 % of donors (weighted) by more than a penny a week and for 1 % by more than 50p. The account amounts (b489, b490) equal the payment less the rebate for at least 98 % of each fuel's account payers. The old split gave gas to 98 % of donors and electricity to 92-94 %; the labelled one gives gas to about 80 % and electricity to 98 %. The DESNZ level holds the electricity and gas totals on both arms. +- **Gas connection.** With donors paying for gas less often than DESNZ counts gas meters, the declared connect leg fills every Great Britain region to its published share (the frame's gas-connected share is 83.7 % against the base's 84.6 %; Northern Ireland, with no published share, keeps its draw). +- **The NEED rake.** Recipients band on HBAI gross income. The rake converges at 2.40 % electricity and 2.63 % gas in the income margin (gas: accommodation 1.40 %, tenure 0.71 %, region 0.11 %) against the `energy_rake` rule of 3.5 % (María's ruling of 2026-10-12); the base, banded on HBAI net income, converges at 0.6 %. The household ceilings bind on 2.6 % of gas kWh and on no electricity. That share moves with the forest's draw of the gas tail, not with the donor: the previous head's split gives 0.5 % with this head's code, and this head with the chain seed moved by one gives 1.4 %, while the donor's raked gas moves by under 2 % for all but about 200 donors (those whose combined bill nets a rebate or whose account amount differs from the payment less the rebate). Every gate passes on all three draws, and the gas income-margin residual stays between 2.6 % and 2.8 %. +- **The NTS band.** On HBAI gross income the recipients' person-weighted NTS income bands (under £25k, £25-50k, £50k and over) move from 6 / 21 / 73 % to 13 / 30 / 57 %; London bus trips rise 1.6 % and other local bus trips fall 1.0 %, and bus fares are within 1 %. +- **Prepayment meters** (microcosm#1110). The donor's weighted prepayment share is 11.0 %; the frame's is 11.5 %, and 10.5 % in Great Britain, about 3.0 million households. The government counted around three million prepayment households in Great Britain when the Energy Price Guarantee prepayment discount began in July 2023: a different reference period, households in both (Ofgem and DESNZ count meters or customers). +- **Road fuel.** The incidence redraw leaves no flagged household at zero, and the level holds ONS 07.2.2 less the donor's other-fuels share. +- **The ETB VAT rate.** Drawn on the ETB's own disposable-income concept, at identity-keyed uniforms and within [0, 1] (microcosm#1225), the weighted mean full-rate VAT expenditure rate moves from 0.401 to 0.398, and its profile by quintile of HBAI net income from 0.358 / 0.412 / 0.413 / 0.416 / 0.405 to 0.354 / 0.398 / 0.404 / 0.415 / 0.419: it now rises with income, where the engine's net income, inflated by in-kind spending, dipped in the top quintile. No recipient keeps a negative or zero rate. The support gate passes on both trees. + +## Decisions behind the energy changes + +- **Which income bands the NEED rake.** The engine's `household_gross_income` adds capital gains and the imputed value of NHS, education and transport spending, about £15,000 a household on average, and missed NEED by 7.8 %. HBAI gross income (the HBAI net income with policyengine-uk's HBAI deductions added back) has a distribution close to the LCFS donor's gross income (weighted medians about £46k and £49k). HBAI net income leaves the two top NEED bands thin (about 6 % of households against about 14 % on HBAI gross), so the rake drives about 2 % of gas kWh above the domestic meter limit and the ceilings then leave those cells 3.2 % short. +- **How the ceilings hand out what they take.** Handed to every household in proportion, the kWh capped households give up moved the cells holding most of them off their NEED means (3.2 % gas on the HBAI-gross band); re-raked within the capped households' own cells, it moves the fit 0.06 points. +- **Which income the ETB VAT rate is drawn on.** On the K25 spine, against the ETB 2023 donor's `disinc` carried to 2024 on average earnings, the engine's `household_net_income` puts 12.8 % of recipients' mass in the wrong donor income decile (median about £47k against the donor's £39k), HBAI net income 4.1 % and the ETB's own concept, gross cash income less income tax, National Insurance and council tax, 3.0 %. +- **How the shortfall is connected.** A gradient-boosted classifier of the donor's gas payers orders the zero-gas households a region connects (a weighted identity-keyed order, so likelier payers go first), and a regime-gated QRF of the payers' gas spend gives each its gas. Without the connect leg nine of the eleven Great Britain regions fell short of their share and the frame's gas-connected share was 79.2 %. + +## What is not here + +- **Unit records.** The household dumps the runs write (`*.households.parquet`) stay outside the repository, with the diagnostic runs behind the decisions above. +- **Licensed inputs.** The LCFS and NTS tabs and the spine. +- **Small cells, exact counts and extremes.** `scripts/sanitize_outputs.py` drops exact donor extremes, remainder minima and the donor-floor negative mass. It writes a count of one to nine households "<10" and drops the weighted quantity those households carry, rounds every other count (to the nearest 10 under 1,000, the nearest 100 above), bands a count of rows picked out by an extreme value ("10-49", "50-99", then the nearest 100), and keeps each NEED margin's largest deviation rather than its cells. The stage receipts band their own donor counts too (`uk_runtime/receipt_counts.py`). + +## Re-running + +The runs need the licensed LCFS 2023-24 and 2024-25 tabs, the NTS 2002-2024 tabs, the ETB 1977-2024 household tab and the spine: + +```bash +scripts/run_stage_dev_at.sh