UK: lcfs_consumption on LCFS 2024-25, with the energy, income-band and prepayment fixes (#1164)

changelog.d/1063-release-cut-blockers.fixed.md

@@ -1 +1 @@
-Clear the UK release-cut certifier's blockers found by the rehearsal on the 2026-09-30 national build (microcosm#1063 c9). The candidate label is `microcosm_uk_2024_25`; the certifier and the full-build gate evidence read the export surface through `uk_export_candidate_columns`, which strips the structural id columns and carries `household.household_weight` as the enhanced-FRS reference does, so ids are no longer reported as unreviewed extras nor the weight as missing. Every declared stage output the incumbent never carried is allow-listed (ruling 2026-10-02): the source and support-channel lineage of the three entities, the council tax family, the reported-benefit inputs the take-up stages read, the SPI channel's `hmrc_spi_*` leaves, the FRS education and housing fields, the two fenced raw FRS codes and `other_investment_income`. The one reviewed export exclusion, `person.incapacity_benefit_reported`, and the five internal disability carriers (`attendance_allowance_reported`, `dla_sc_reported`, `dla_m_reported`, `pip_m_reported`, `pip_dl_reported`, stage-internal inputs the export registers keep off the release) now leave at the release boundary: `uk_release_export_frame` drops it from the written candidate, the graph readback expects the same frame, and the spine keeps the column for the SPI income stage. The FRS 2024-25 vintage's recoded training variable (`TRAIN2`, where 9 is "None of these" and 10 the Kickstart Scheme) is read, so `is_in_approved_training` carries signal again. The LCFS donor's diary consumption is floored at the declared `floor` (0) before the imputation, the support-clip ranges and the rake see it, with a receipt of the rows raised; `uk_stage_lcfs_consumption_support` declares the floor and requires the receipt, no remaining negative donor row and no clip range below it (250 release rows had inherited a minimum of -14,165 in housing, water and electricity consumption). The two FRS education inputs the enhanced-FRS reference barely carries (`adult_ema`, `dfe_education_spending`) enter the per-reference input-mass register, and the sparse charity-gifts leaf re-enters the QRF-tail register, each a signed receipt expiring 2027-04-02. `uk_stage_child_benefit_take_up` declares `population_fact_check` like the Pension Credit gate, so a synthetic smoke build records its failure and continues, and the anchor composition check allows the absolute rounding tolerance the other mass identities use.
+Clear the UK release-cut certifier's blockers found by the rehearsal on the 2026-09-30 national build (microcosm#1063 c9). The candidate label is `microcosm_uk_2024_25`; the certifier and the full-build gate evidence read the export surface through `uk_export_candidate_columns`, which strips the structural id columns and carries `household.household_weight` as the enhanced-FRS reference does, so ids are no longer reported as unreviewed extras nor the weight as missing. Every declared stage output the incumbent never carried is allow-listed (ruling 2026-10-02): the source and support-channel lineage of the three entities, the council tax family, the reported-benefit inputs the take-up stages read, the SPI channel's `hmrc_spi_*` leaves, the FRS education and housing fields, the two fenced raw FRS codes and `other_investment_income`. The one reviewed export exclusion, `person.incapacity_benefit_reported`, and the five internal disability carriers (`attendance_allowance_reported`, `dla_sc_reported`, `dla_m_reported`, `pip_m_reported`, `pip_dl_reported`, stage-internal inputs the export registers keep off the release) now leave at the release boundary: `uk_release_export_frame` drops it from the written candidate, the graph readback expects the same frame, and the spine keeps the column for the SPI income stage. The FRS 2024-25 vintage's recoded training variable (`TRAIN2`, where 9 is "None of these" and 10 the Kickstart Scheme) is read, so `is_in_approved_training` carries signal again. The LCFS donor's diary consumption is floored at the declared `floor` (0) before the imputation, the support-clip ranges and the rake see it, with a receipt of the rows raised; `uk_stage_lcfs_consumption_support` declares the floor and requires the receipt, no remaining negative donor row and no clip range below it (250 release rows had inherited negative housing, water and electricity consumption). The two FRS education inputs the enhanced-FRS reference barely carries (`adult_ema`, `dfe_education_spending`) enter the per-reference input-mass register, and the sparse charity-gifts leaf re-enters the QRF-tail register, each a signed receipt expiring 2027-04-02. `uk_stage_child_benefit_take_up` declares `population_fact_check` like the Pension Credit gate, so a synthetic smoke build records its failure and continues, and the anchor composition check allows the absolute rounding tolerance the other mass identities use.

changelog.d/1110-prepayment-meter-input.added.md

@@ -0,0 +1 @@
+The UK `lcfs_consumption` stage draws `uses_energy_prepayment_meter`, the household input policyengine-uk reads from 2.123.3 for the Energy Price Guarantee prepayment discount, July 2023 to March 2024 (microcosm#1110; PolicyEngine/policyengine-uk#2098; uk-data#539 tracks the same input). An LCFS donor prepays when it last put money on an electricity, gas or combined key or meter (b1751, b1701, b233, which the LCF asks only of prepayment-meter households) or its diary records a slot-meter payment (c45114, c45214): electricity-only prepayment counts, and a household with no meter payment, including one that did not answer, does not. The declared `impute_prepayment_meter` step fits that flag with a weighted regime-gated QRF on the chain's predictors and draws it at uniforms keyed on the household id, so no other draw moves; its receipt records the donor's and the frame's weighted shares and the Great Britain households flagged. The release exports the column: policyengine-uk reads it from 2.123.3 and an older engine, the locked one among them, ignores it; it is a reviewed extra column of the export surface until the incumbent carries it (uk-data#539), and an engine-lane test checks an engine that reads it reads a household boolean. On the K25 spine the frame flags about 3.0 million Great Britain households (10.5 %) at prior weights, against the around three million prepayment households the government counted in July 2023, when the discount began: a different reference period, and households in both, where Ofgem and DESNZ count meters or customers.

changelog.d/1164-etb-vat-disposable-income.fixed.md

@@ -0,0 +1 @@
+The UK `etb_vat` stage draws the full-rate VAT expenditure rate on disposable income on the ETB's own concept, at uniforms keyed on the household id (microcosm#1164; the predictor item microcosm#1225 left out). The donor's income is ETB `disinc`, gross income less income tax, employees' National Insurance and council tax, before indirect taxes; the recipient's was the engine's `household_net_income`, which also takes off the indirect taxes and keeps the in-kind NHS and education spending, and which read recipients about a quarter richer than the donor at the median on the K25 spine. The recipient's income is now its gross cash income (HBAI net income with policyengine-uk's HBAI deductions added back) less the same direct taxes, from `uk_runtime.household_income.etb_disposable_income`, whose distribution sits closest to the donor's of the concepts tried. A declared `uprate_donor_columns` step carries the 2023 donor's income to the 2024 base year on the engine's average earnings index, as `etb_services` carries the same `disinc`. The QRF now draws at quantile and sign uniforms keyed on the household id under the declared `salt`, where the model's generator drew in row order, so a household keeps its rate however the frame is ordered.

changelog.d/1164-lcfs-2024-25.changed.md

@@ -0,0 +1 @@
+The UK `lcfs_consumption` stage reads the Living Costs and Food Survey 2024-25 (UK Data Service SN 9603, 2nd edition) in place of 2023-24 (SN 9468), so every UK survey stage is now on its latest microdata (microcosm#1164). The diary covers April 2024 to March 2025, the FRS 2024-25 spine's year: the CPI bridge to the calendar-2024 basis steps back a quarter instead of forward three, and the income predictors need no step. The donor has about 5,000 households against about 4,200. The 2nd edition's household tab (`dvhh_uk_anon_v3_2024.tab`) carries the annual weight as `cal_weight_annual`, the name `weighta` had, and its 1st-edition `dvhh_uk_anon_v2_2024.tab` carries no weight; no other variable the stage reads changed name or coding, and COICOP 2018 is not yet adopted. The tabs carry their study number and DOI in the spec, the fit is named `uk_lcfs_2024_25_consumption`, the support bounds are cut from the 2024-25 donor, the road-fuel level's other-fuels share is 0.46 percent of COICOP 07.2.2 on the new diary (0.37 on the old), and the road-fuel target's scope note says so.

changelog.d/1164-lcfs-donor-floor-gaps.fixed.md

@@ -0,0 +1 @@
+The UK LCFS donor floor now covers every donor spend it reads (microcosm#1164). `bus_fare_spending`, summed from three diary lines a refund can also make negative, is floored with the fifteen diary consumption columns. The road-fuel level's other-fuels share reads the c72211–c72213 lines floored the same way. `tools/build_uk_e6_support_bounds.py` applies the stage's declared floor before reading donor ranges, so the committed housing, water and electricity bound starts at 0 like the stage's support; no other bound moves.

changelog.d/1164-lcfs-energy-split-labelled-codes.fixed.md

@@ -0,0 +1 @@
+The UK LCFS stage splits the diary's fuel, light and power (p537) into electricity and gas by its labelled parts, from the LCF User Guide's derived-variable labels and P537 definition (microcosm#1164). The derivation it replaces, ported from the incumbent uk-data pipeline, read b226 as electricity paid by direct debit and b489 and b490 as prepayment-meter payments. The labels make b226 the gas component of a combined utilities bill (b227 is the electricity one) and b489 and b490 the electricity and gas account payments less rebates (b175 less b178, b170 less b173), so it gave combined-bill households, most of the donor, their gas as electricity; took gas away from electricity for households with two accounts; split an electricity account between the fuels; and folded oil and solid fuels into the drawn energy. Each fuel is now its account payment less its rebate (b489, b490 as the tab carries them; the stage refuses a donor where either equals its payment less its rebate for under 95 % of that fuel's account payers, which pins the labels and the rebate's sign), its prepayment key or meter payment, its part of a combined bill or meter payment, and the diary's second-dwelling account and slot-meter lines. A combined payment left unsplit, net of the combined bill's rebate (b174), goes by the household's own electricity share where it pays for both fuels and by the donor's mean share of dual-fuel households otherwise, since a combined payment means it pays for both; a donor with no dual-fuel household fails rather than taking a fixed share. Oil, bottled gas, solid fuels and heat stay in the housing total. A `donor_energy_split` receipt records the donor's electricity, gas and other fuels against p537, what reading refunds as zero adds, the weighted share of donors whose unclipped parts differ from their p537, and the account identity shares. Negative amounts (refunds) are read as zero. The NEED rake and the DESNZ level still set each fuel's level, but which households carry electricity and which gas now follows what they reported.

changelog.d/1164-lcfs-gas-connection-connect.fixed.md

@@ -0,0 +1 @@
+The UK LCFS stage connects households to gas where its draw leaves a region below the published DESNZ share (microcosm#1164). The connection step only disconnected: it walked gas-positive households down to each region's share of gas meters over electricity meters. The old energy split gave gas to 98 % of LCFS donors, so every region could be cut to its share; the labelled split gives gas to about 80 %, against the about 85 % of homes DESNZ counts gas meters for, so nine of the eleven Great Britain regions fell short and the frame's gas-connected share fell to 79 %. The `price_domestic_energy` step now declares `connect_rule: gas_payer_probability_order`: two weighted fits on the chain's predictors give each zero-gas household the probability that a donor like it pays for gas (a gradient-boosted classifier) and the gas such a donor pays (a regime-gated QRF of the payers' gas spend), and in a region below its share the zero-gas households are connected in an identity-keyed order drawn in proportion to that probability, by the disconnection walk's weight-fitting rule, until the share is met. The NEED rake, the household ceilings and the DESNZ level then run as before. The `energy_rake` gate requires the receipt to name the declared rule and salt and waives a region's shortfall only where it ran out of households to connect.

changelog.d/1164-lcfs-household-kwh-ceilings.fixed.md

@@ -0,0 +1 @@
+The UK LCFS stage no longer leaves a household's electricity or gas above the domestic meter definitions behind the DESNZ total it levels to (microcosm#1164; the extreme energy values item of #1207; María's ruling of 2026-10-12). NEED's quality-assurance annex counts a gas meter as domestic under 73,200 kWh a year and an electricity meter under 50,000 kWh on profile classes 1 and 2. The energy columns are exempt from the donor support clip because the NEED rake and the DESNZ level set them, so a few drawn households sat far above any domestic use, and since #1167 the housing total contained them. The `price_domestic_energy` step declares `household_kwh_ceiling`; after the level, `cap_household_kwh` sets a household above it to the ceiling and re-rakes the households below it in its own NEED cells to the levelled targets with the capped ones held, then restores the DESNZ total. Handed to every household in proportion instead, the kWh moved the cells that hold most capped households (detached and high-income homes) off their NEED means. The `energy_rake` receipt counts the households capped and the share of the total they gave up, the NEED fit is read after the cap, and the `energy_rake` gate checks the declared ceilings, that no household is left above them and that the total held.

changelog.d/1164-lcfs-recomposed-totals-donor-maximum.fixed.md

@@ -0,0 +1 @@
+The UK LCFS stage holds each recomposed total, housing and transport, to the donor's realised maximum of that total, the support every drawn column is clipped to (microcosm#1164). The recomposition writes a total as its draw less the drawn parts plus the levelled parts, so a remainder near the top of the diary's range plus a levelled energy bill larger than the drawn one could leave the committed support bound the terminal `uk_support` gate holds the emitted totals to; on the 2024-25 diary, whose housing bound is far tighter than 2023-24's, some households did. The `recompose_from_remainder` step declares `maximum_total: donor_support`; a held total never falls below its parts, the excess comes out of its remainder, and the receipt counts the rows held and the weighted mass taken, which the `recomposed_totals` gate's identities include.

changelog.d/1164-lcfs-unmapped-codes-refused.fixed.md

@@ -0,0 +1 @@
+The UK `lcfs_consumption` stage refuses an LCFS region (`gorx`), tenure (`a122`) or accommodation (`a121`) code it does not map, instead of dropping the donor (microcosm#1164). The donor table ended in a `.dropna()` that silently lost any household whose code fell outside the stage's maps, and the derive step's `logged_dropna_row_count` was never implemented, so a survey year that recoded one of these variables would have shrunk the donor without a trace. The derive step now declares `unmapped_codes: refuse`; the error names each unknown code, or a blank, without counting rows, and any other missing donor value stops the stage too. Every LCFS 2023-24 and 2024-25 code maps (the stage runs on both with the refusal in place), so neither donor loses a household.

changelog.d/1164-need-band-gross-income.fixed.md

@@ -0,0 +1 @@
+The UK LCFS stage bands recipient households for the NEED energy rake on gross cash income, the concept the donor (LCFS p344p) and NEED band on (microcosm#1164). No spine stage wrote a gross income, so the recipient fell back to its HBAI net income and sat in too low a NEED income band. The engine's `household_gross_income` is not the concept either: it adds capital gains and the imputed value of NHS, education and transport spending, about £15,000 a household on average, which put the recipient in too high a band (a 7.8 % NEED residual on the K25 spine). The recipient's gross income is now its HBAI net income with the deductions policyengine-uk takes to reach it added back (council tax and domestic rates, income tax, National Insurance, student loan repayments, employee and personal pension contributions, maintenance and child payments, LVT), from the new `uk_runtime.household_income` helper; the stage materializes those for the band alone, and an engine-lane test fails if the engine's HBAI deduction list changes. On the K25 spine its distribution sits close to the donor's (weighted medians about £46k and £49k). The rake then converges at 2.40 % electricity and 2.82 % gas in the income margin, a cross-margin gap with NEED's other margins, so the `energy_rake` gate's residual rule moves from 2.5 % to 3.5 % on María's ruling of 2026-10-12. Electricity and gas by income move on every row; the DESNZ level holds.

changelog.d/1164-nts-income-band-hbai-gross.fixed.md

@@ -0,0 +1 @@
+The UK `nts_bus_travel` stage bands recipient households on gross cash income, the concept of the NTS household income band, rather than the engine's `household_gross_income` (microcosm#1164). The engine's figure adds capital gains and the imputed value of NHS, education and transport spending, which put about a fifth of households (pensioners, families with school-age children and capital-gains households above all) a band higher than their cash income, and the NTS shows bus use falling with income. The recipient's band income is now its HBAI net income with policyengine-uk's HBAI deductions added back, the same `uk_runtime.household_income` helper the LCFS NEED band uses, and the stage declares those engine variables. Bus-use bands, trips and the bus fares and bus support the LCFS and ETB services stages price from them move.

changelog.d/1164-receipt-counts-banded.fixed.md

@@ -0,0 +1 @@
+The UK LCFS and ETB VAT stages band the donor counts their receipts record at source, since a stage receipt is build output that can leave the machine (microcosm#1164, from the review of microcosm#1260). The new `uk_runtime.receipt_counts` rounds donor sizes and training rows, writing one to nine as "<10" and rounding to the nearest ten below a thousand and the nearest hundred above: the road-fuel and gas-payer training rows and the fuel-flag donor households. Counts of rows picked out by an extreme value are banded as "<10", "10-49", "50-99", then to the nearest hundred: the ETB VAT rows dropped and capped, and the LCFS donor floor's rows raised, per column and in total. The ETB VAT receipt records no donor size, which beside the donor year's sample size would give the dropped count back. An unweighted share of donors is floored to two decimals, since an exact share gives back its count, and a weighted share carried by fewer than ten donors is written "<10". A count a gate must read as an exact zero stays exact.

changelog.d/1164-support-bounds-no-rerounding.fixed.md

@@ -0,0 +1 @@
+`tools/build_uk_e6_support_bounds.py` refuses to move a committed rounded support bound while the file's donor tab digests are unchanged (microcosm#1164). Each bound is a donor extreme rounded outward; re-derived from the same tabs after a change in the declared uprating, the cleaning or the floor, a bound that rounds to a different value narrows the extreme between the two thresholds. A lower bound may still move to zero, the floor, which reads no donor value.