Fix the annual allowance on survey data, the UC benefit cap earnings threshold, relevant UK earnings and Scottish water uprating

.github/test_outcome_allowlist.toml

@@ -64,3 +64,10 @@ nodeid = "policyengine_uk/tests/test_dwp_published_rates.py::test_projection_con
 outcome = "skipped"
 reason = '^(not uprated|series ends before 2026-27|Universal Credit Act 2025 s\.1 departs from CPI uprating)$'
 why = "The test is parametrised over every DWP rate parameter and skips those it does not apply to: parameters with no uprating, series that end before 2026-27, and the four UC standard allowances, whose minimum the Universal Credit Act 2025 s.1 sets above CPI uprating (#1925). The LCWRA element, frozen by s.3, is checked against its own uprating index (#2284). 64 cases on 12 October 2026."
+
+[[allow]]
+nodeid = "policyengine_uk/tests/microsimulation/test_annual_allowance_against_hmrc.py::test_2024_annual_allowance_charge_and_payers_are_near_hmrc"
+outcome = "xfailed"
+reason = "policyengine-uk-data#606"
+data = ["present"]
+why = "Known data gaps, chiefly PolicyEngine/policyengine-uk-data#606: the certified Enhanced FRS imputes employer pension contributions as three times employee contributions, which puts the 2024 annual allowance count, charge and excess several times above HMRC's; with a third of those contributions all three fall within the band. Pinned by a strict xfail until the data is fixed (#2523)."

changelog.d/relevant-uk-earnings.fixed.md

@@ -0,0 +1 @@
+- Use one definition of relevant UK earnings (Finance Act 2004 s. 189(2)) for pension contribution relief, the annual allowance income tests and the pension deduction in adjusted total income: employment income after allowable expenses but before pension deductions, and trading profits after capital allowances and the trading allowance but before losses brought forward. Relief previously counted pay before expenses and trading profits before capital allowances and the trading allowance.

changelog.d/scottish-water-uprating.fixed.md

@@ -0,0 +1 @@
+- Uprate water and sewerage charges by country when a dataset is projected: England and Wales by Ofwat's average bills, Scotland by a new Scottish Water series (8.8% in 2024-25, 9.9% in 2025-26 and 8.67% in 2026-27, then the CPI + 2% cap in WICS's draft 2027-33 determination), and Northern Ireland, which has no domestic water charges, not at all. The charges were previously held flat to 2030 and then grown by the England and Wales series everywhere.

changelog.d/survey-pension-input.added.md

@@ -0,0 +1 @@
+- Add `defined_benefit_pension_share`, `defined_benefit_pension_contributions` and `defined_benefit_pension_input_amount`, so data can say which workplace pension contributions go to defined-benefit arrangements and supply their statutory pension input amount (Finance Act 2004 s. 234) for the annual allowance. Where the input amount is not supplied it is those contributions times `gov.simulation.pensions.defined_benefit_input_per_contribution`, set at 1 from the accrual and contribution rates of the main public service schemes.

changelog.d/survey-pension-input.changed.md

@@ -0,0 +1 @@
+- Estimate unused annual allowance carried forward (Finance Act 2004 s. 228A) from 2020-21 where none is supplied, instead of nil: each of the five previous years' pension input is this year's times `gov.simulation.pensions.prior_year_input_ratio` (1), with incomes and input deflated by average earnings growth, against that year's tapered allowance, used earliest first (s. 228A(7)).

changelog.d/uc-benefit-cap-earnings-threshold.fixed.md

@@ -0,0 +1 @@
+- Derive the Universal Credit benefit cap's earnings exception threshold each year from the national living wage: the pay for 16 hours a week, times 52 and divided by 12, with any fraction of a pound disregarded (Universal Credit Regulations 2013 regs. 82(1)(a) and 6(1A)(za)), and a fixed £430 a month before April 2017. It was fixed at the 2025-26 figure of £10,152 a year for every year; it is now £9,516 in 2024-25 and £10,572 in 2026-27. The national living wage parameter ends at its April 2026 rate, so the threshold stays at £10,572 in later years until that parameter is projected.

docs/book/assumptions/water-bills.md

@@ -81,3 +81,28 @@ The model uses these year-on-year growth rates. Each year is the charging year s
 ```{warning}
 The 2027-2029 projections cannot be reproduced from the repository. They take the CCW table, which reflects Ofwat's PR24 *draft* determinations rather than the final ones, weight each company by customer numbers, and add a hardcoded CPIH path. The customer weights were in `ofwat_increases.csv`, which is gitignored and was never committed.
 ```
+
+## Scotland and Northern Ireland
+
+Ofwat's bills cover England and Wales. When a dataset is projected forward, each household's `water_and_sewerage_charges` grows by its own country's series (`uprate_water_and_sewerage_charges` in `policyengine_uk/data/economic_assumptions.py`):
+
+- England and Wales, and households with an unknown region, take `ofwat.water_bills`.
+- Scotland takes `scottish_water.water_bills`. Scottish Water sets household charges within limits from the [Water Industry Commission for Scotland](https://wics.scot/) (WICS), and every council tax band's charge is a fixed fraction of Band D, so each band rises by the same percentage.
+- Northern Ireland has no domestic water charges, so any charge a dataset gives its households is held flat.
+
+This holds to the end of the dataset projection (2032). After it, policyengine-core grows every household's charge by the index the variable declares, Ofwat's England and Wales series, as it did before this split.
+
+| Year | Scottish Water growth | Source |
+|------|-------------|--------|
+| 2022 | 4.2% | [WICS, charges scheme 2022-23](https://wics.scot/publications/price-setting/strategic-review-charges-2021-27/scottish-waters-charges-scheme/charges-scheme-2022-23) |
+| 2023 | 5.0% | [WICS, charges scheme 2023-24](https://wics.scot/publications/price-setting/strategic-review-charges-2021-27/scottish-waters-charges-scheme/charges-scheme-2023-24) |
+| 2024 | 8.8% | [WICS, charges scheme 2024-25](https://wics.scot/publications/price-setting/strategic-review-charges-2021-27/scottish-waters-charges-scheme/charges-scheme-2024-25) |
+| 2025 | 9.9% | [Scottish Water, 28 January 2025](https://www.scottishwater.co.uk/About-Us/News-and-Views/2025/01/Charges-2025-to-2026-Announcement) |
+| 2026 | 8.67% | [Scottish Water, 22 January 2026](https://www.scottishwater.co.uk/about-us/news-and-views/2026/01/230126-charges-2026) |
+| 2027 | 3.9% | [WICS Draft Determination 2027-33](https://wics.scot/system/files/2026-06/Draft-Determination-2027-33.pdf): the CPI + 2% core cap (para. 9.1.3) with WICS's CPI forecast of 1.9% (Table 68) |
+| 2028 | 3.9% | As above |
+| 2029 | 4.1% | As above, with CPI at 2.1% |
+
+Each year is the charging year starting in April, so 2025 is 2025-26. Later years to 2032 hold the 2029-30 rate. [WICS publishes its Final Determination for 2027-33](https://wics.scot/latest/wics-publishes-2027-33-draft-determination) on 29 October 2026; the 2027-2029 values should then be replaced with it.
+
+`policyengine_uk/utils/create_ahc_deflator.py` still reads the England and Wales index. It is a standalone script that the model does not run.

docs/book/validation/hbai.md

@@ -108,6 +108,6 @@ Each of the variables above is either derived from other variables, or is a dire
 | `maintenance_expenses` | gov.obr.consumer_price_index | Uprated based on CPI inflation |
 | `student_loan_repayments` | gov.economic_assumptions.indices.obr.average_earnings | Uprated based on OBR average earnings projections |
 | `rent` | gov.obr.rent | Social rents uprated by CPI+1%, private rents uprated with outturn data by region to 2025, then with backed-out private rent level growth from OBR aggregate rent forecasts (assuming social rent grows at CPI+1%) |
-| `water_and_sewerage_charges` | gov.obr.consumer_price_index | Uprated based on CPI as proxy |
+| `water_and_sewerage_charges` | gov.economic_assumptions.yoy_growth.ofwat.water_bills (England and Wales), gov.economic_assumptions.yoy_growth.scottish_water.water_bills (Scotland) | Uprated by each country's published bill increases, then projections; Northern Ireland has no domestic water charges and is held flat |
 | `mortgage_interest_repayment` | gov.obr.mortgage_interest | Uprated based on outturn Ofwat data to 2025, then Ofwat projections onwards |
 | `housing_service_charges` | gov.economic_assumptions.indices.obr.lagged_average_earnings | Uprated based on lagged average earnings as proxy |

docs/engineering/pension-annual-allowance.md

@@ -76,6 +76,17 @@ net pay. Threshold income deducts [FA 2004 ss. 188–191 pension
 relief](https://www.legislation.gov.uk/ukpga/2004/12/section/188) on actual
 earnings and contributions.
 
+Ordinary pension relief, the ordinary `adjusted_total_income` and both income
+tests take relevant UK earnings
+([s. 189(2)](https://www.legislation.gov.uk/ukpga/2004/12/section/189)) from
+one helper, `relevant_uk_earnings`: employment income chargeable to income
+tax, which is net taxable earnings ([ITEPA 2003 ss. 9(2) and
+11](https://www.legislation.gov.uk/ukpga/2003/1/section/11)), so pay and
+employer-paid taxable statutory pay less allowable expenses; and trading
+profits after capital allowances and the trading allowance but before losses
+brought forward. The pension deductions that s. 327(5) also lists are the
+relief these earnings limit, so they are not taken off.
+
 An explicitly supplied `adjusted_net_income` is authoritative. For this input
 path, the helper restores gross Gift Aid and pension relief recomputed on
 actual contributions and earnings. It does not remove an actual supplied
@@ -117,12 +128,11 @@ for the whole year according to age on that date; payments are not split at
 the birthday. Contributions before 75 that exceed the earnings relief limit
 still count. Adjusted income subtracts all actual member contributions under
 s. 228ZA(4)(d)(ii), including those excluded from the post-75 DC input.
-For defined-benefit
-arrangements, supply the statutory total pension input amount, including
-valued DB accrual, through this variable. Cash employer funding is not a
-valuation of DB rights. Survey estimates retain the existing cash proxy;
-this correction does not reconstruct DB rights, overseas adjustments, unused
-allowance histories or anti-avoidance arrangements under s. 228ZB.
+Defined-benefit
+arrangements are counted separately; see the last section. Supplying
+`pension_contributions_for_annual_allowance` itself overrides the whole total.
+The model does not reconstruct overseas adjustments or anti-avoidance
+arrangements under s. 228ZB.
 
 [The NICs Act 2026](https://www.legislation.gov.uk/ukpga/2026/15/enacted)
 changes NICs treatment without changing s. 228ZA. From 2029, the model includes
@@ -148,3 +158,77 @@ Pension relief retains the model's aggregate earnings cap and allocation
 conventions. Historical years use calendar-year proxies for the named tax
 years; the 2015 transitional pension-input alignment rules are outside this
 taper calculation.
+
+## Survey data: defined benefits and carry forward
+
+Surveys record contributions paid this year. Two statutory amounts need more:
+the pension input amount of a defined-benefit arrangement, and unused
+allowance carried forward from earlier years.
+
+**Defined benefits.** A defined-benefit arrangement's pension input amount is
+the increase in the value of the member's rights over the year
+([s. 234](https://www.legislation.gov.uk/ukpga/2004/12/section/234)): 16 times
+the annual pension, plus any separate lump sum, at the end less the same at the
+start, with the start raised by CPI
+([s. 235](https://www.legislation.gov.uk/ukpga/2004/12/section/235)). It is not
+the cash paid in.
+
+- `defined_benefit_pension_share` (an input from 0 to 1, default nil) is the
+  share of the person's workplace contributions (employer, net pay and salary
+  sacrifice) paid into defined-benefit arrangements. Personal (relief at
+  source) contributions are money purchase.
+- `defined_benefit_pension_input_amount` replaces those contributions in
+  `pension_contributions_for_annual_allowance`. Supply it where it is known,
+  as on a pension savings statement, with the share of the contributions that
+  fund it: contributions outside the share also count as money purchase
+  input. Otherwise it is the contributions times
+  `gov.simulation.pensions.defined_benefit_input_per_contribution`, which is
+  1. In the four largest public service schemes in 2024-25 the input of an
+  active member with ten years' service is 0.8 to 1.2 times employer and
+  member contributions together (the parameter file gives the figures), and
+  survey employer contributions drawn from regular contribution rates leave
+  out deficit repair payments.
+- With the default share of nil, every contribution is counted as money
+  purchase input, as before. With the ratio at 1, a supplied share alone
+  changes the input only for net-pay contributions to a defined-benefit
+  scheme from age 75, which now count, since the defined-benefit input has
+  no age limit. Results move only once data supplies a valued input amount.
+
+**Carry forward.** [s. 228A](https://www.legislation.gov.uk/ukpga/2004/12/section/228A)
+adds the allowance left unused in each of the three previous tax years in
+which the person was a scheme member, earliest first. `unused_pension_annual_allowance`
+can be supplied for a year. A supplied or stored value applies to that year
+only: the window of years moves on, so dataset projection does not copy it
+into later years (`YEAR_SPECIFIC_INPUTS` in `data/economic_assumptions.py`),
+which use the estimate. From 2020-21 it otherwise defaults to an estimate:
+
+- In each of the five previous years, the pension input amount was this
+  year's times `gov.simulation.pensions.prior_year_input_ratio` (1), and the
+  incomes tested for that year's taper were this year's, each deflated by
+  average earnings growth.
+- Each year's allowance follows that year's parameters, tapered on those
+  incomes. Each year's excess uses the unused allowance of the three years
+  before it, earliest first (s. 228A(7)), and the unused allowance of the
+  three years before this one is carried forward.
+- A person with pension input this year is taken to have been a scheme member
+  in each of those years; with none, nothing is carried forward.
+
+With a ratio of 1 the estimate matters mainly where the allowance has risen,
+as it did to £60,000 in 2023-24, or where pay has grown. A ratio of 0, no input
+in earlier years, gives the most there could be. The ratio of 1 is a
+convention, since HMRC publishes no statistics on carry forward. Real
+excesses are often lumpy (a defined-benefit promotion, a one-off
+contribution), so a steady history probably understates carry forward.
+
+**Against HMRC.** HMRC's Private pension statistics (July 2026) count 30,440
+individuals reporting contributions above their allowance through Self
+Assessment in 2024-25 (provisional, Table 7), after carry forward, and
+estimate annual allowance charges of £0.3bn (Table 6). On the certified
+Enhanced FRS the model's 2024 count and charge are several times these,
+because the dataset imputes employer contributions as three times employee
+contributions ([policyengine-uk-data#606](https://github.com/PolicyEngine/policyengine-uk-data/issues/606)).
+With employer contributions at a third of their stored values, the count,
+the charge and the chargeable amount (against Table 7's £672m of excess) all
+fall within half to one and a half times HMRC's figures.
+`tests/microsimulation/test_annual_allowance_against_hmrc.py` checks all three
+against that band, as a strict expected failure until the data is fixed.