From b6d8af5b556d5c64848a0d9035b39bef298eab09 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 24 Oct 2026 07:59:11 -0400 Subject: [PATCH 01/11] Re-derive VAT survey coverage on the certified a133f37a cut survey_consumption_coverage was derived in #2187 on Microcosm f9d1922c. The 6cba5d71 and a133f37a cuts record about 10% more standard-rated consumption, so with 0.67 total vat ran 7.5% to 10.3% above OBR receipts in 2024-25 to 2026-27. The doc's own calculation on a133f37a gives 0.739 for the latest outturn year; it is used as 0.74 (product 0.518), inside the 0.720 to 0.756 bound from receipts and the ready reckoner. Adds Hypothesis properties for the grossing factor and for vat being statutory liability over the shipped product. Totals at the new product and the reform-regression fixture wait on runs in the hub's next reserved window. Fixes #2633 Co-Authored-By: Claude Opus 5.5 --- changelog.d/2633.fixed.md | 1 + docs/book/assumptions/mp30k-uk-switch-plan.md | 11 ++- docs/engineering/vat-coverage.md | 79 +++++++++++-------- .../simulation/microdata_vat_coverage.yaml | 4 +- .../vat/survey_consumption_coverage.yaml | 34 ++++---- .../baseline/gov/hmrc/vat/vat_grossing.yaml | 6 +- .../tests/test_vat_grossing_legacy.py | 2 +- .../tests/test_vat_grossing_properties.py | 77 ++++++++++++++++++ 8 files changed, 154 insertions(+), 60 deletions(-) create mode 100644 changelog.d/2633.fixed.md create mode 100644 policyengine_uk/tests/test_vat_grossing_properties.py diff --git a/changelog.d/2633.fixed.md b/changelog.d/2633.fixed.md new file mode 100644 index 0000000000..1bc303aa32 --- /dev/null +++ b/changelog.d/2633.fixed.md @@ -0,0 +1 @@ +- Re-derived the VAT survey consumption coverage on the certified Microcosm UK 2024-25 cut a133f37a (0.67 to 0.74, product 0.469 to 0.518). The earlier value came from an older cut that recorded about 10% less standard-rated consumption, which put VAT 7.5% to 10.3% above OBR receipts. diff --git a/docs/book/assumptions/mp30k-uk-switch-plan.md b/docs/book/assumptions/mp30k-uk-switch-plan.md index 374e9209f3..b7bbe44954 100644 --- a/docs/book/assumptions/mp30k-uk-switch-plan.md +++ b/docs/book/assumptions/mp30k-uk-switch-plan.md @@ -117,12 +117,11 @@ A documented gate must pass before any environment's default changes: which dataset is loaded. One parameter is tied to the dataset, though: `gov.simulation.vat.survey_consumption_coverage` is calibrated to one Microcosm UK 2024-25 national cut only - (`microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68`; policyengine.py - 6.10.0 pins the later cut - `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), puts 2024-25 - VAT 38% (policyengine-uk-data 1.56.16) or 44% (1.58.0) above receipts on - the Enhanced FRS 2024-25, and has to be re-derived when the dataset it - should match changes (`docs/engineering/vat-coverage.md`). + (`microcosm-uk-2024-25-national-20261023T075039Z-a133f37a`; the earlier + `6cba5d71` cut gives the same value), puts VAT above receipts on the + Enhanced FRS 2024-25 (whose implied coverage is 0.90 to 0.97), and has to + be re-derived when the dataset it should match changes + (`docs/engineering/vat-coverage.md`). 2. While both are in use, CI runs the dataset-dependent tests on both. 3. The default is set per environment: for direct use of this package through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index 49c467898a..b6169c2261 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -11,7 +11,7 @@ factors, so that total VAT matches receipts: theoretical liability; the rest falls on government, charities, VAT-exempt businesses and new housing. This is a published quantity and does not depend on the dataset. -- `gov.simulation.vat.survey_consumption_coverage` (0.67): the share of +- `gov.simulation.vat.survey_consumption_coverage` (0.74): the share of household VAT liability that the dataset's consumption captures. This depends on the dataset's consumption imputation and must be re-derived when the default dataset changes. @@ -22,33 +22,41 @@ to households in proportion to their own VAT-liable spending. These two factors replace `gov.simulation.microdata_vat_coverage`, a single undated 0.38 added in January 2023 against an older LCFS-based consumption -imputation, with no derivation recorded. Its product with today's datasets put -2024-25 VAT 24% above OBR receipts on Microcosm UK 2024-25 and 71% above on -Enhanced FRS 2024-25 (policyengine-uk-data 1.56.16). +imputation, with no derivation recorded. `gov.simulation.microdata_vat_coverage` is kept as a deprecated parameter so that existing reforms to it still work. Its value is now the product of the two -factors (0.469). If a reform sets it to any other value, `vat` and +factors (0.518). If a reform sets it to any other value, `vat` and `baseline_vat` use that value in place of the two factors (`policyengine_uk/utils/vat.py`); otherwise they use the two factors. New reforms should change the two factors instead. ## Which datasets the factor applies to -The survey coverage of 0.67 is calibrated to one dataset: the certified -Microcosm UK 2024-25 national release. It does not carry over to other -datasets, because their consumption imputations capture a different share of -household VAT. On Enhanced FRS 2024-25 it puts VAT 35% to 44% above receipts, -depending on the release and year (see the tables below). Re-derive the coverage, with -the calculation below, whenever the dataset it should match changes. +The survey coverage of 0.74 is calibrated to one dataset: the certified +Microcosm UK 2024-25 national release `a133f37a`. It does not carry over to +other datasets, because their consumption imputations capture a different +share of household VAT: on Enhanced FRS 2024-25 its implied coverage is +0.90 to 0.97 (see the tables below). Re-derive the coverage, with the calculation below, whenever the dataset it +should match changes. -## Derivation (October 2026) +## Derivation (24 October 2026) + +The first derivation (#2187, 0.67) used the Microcosm release `f9d1922c`. +That release was replaced by the `6cba5d71` and `a133f37a` cuts, which record +about 10% more standard-rated consumption. With 0.67, total `vat` on +`a133f37a` ran 7.5% to 10.3% above receipts in 2024-25 to 2026-27, and on +`6cba5d71` a reform run of one point on the standard rate cost £9.94bn in +2026-27, 12.9% above HMRC's ready reckoner. This derivation repeats the same calculation on `a133f37a` +(PolicyEngine/policyengine-uk#2633). ### Inputs | Input | Value | Source | |---|---|---| -| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68` (revision `f9d1922c`), SHA-256 `aa31bdf67c977927ea2b325567d1cf7a79d94381239bc79918a0a0fc9c9588af` | Private repository | +| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261023T075039Z-a133f37a`, SHA-256 `64f83900ca3a1242fb017585234844ce89f743d32e41cd2b6558057777f914aa` | Private repository | +| Microcosm UK 2024-25 national release, cross-check | tag `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`, SHA-256 `810ba980005a8b907cb51464279fa5e0125464b7af9a4871b7f6744c30951ece` | Private repository | +| Microcosm UK 2024-25 national release, first derivation | tag `microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68` (revision `f9d1922c`), SHA-256 `aa31bdf67c977927ea2b325567d1cf7a79d94381239bc79918a0a0fc9c9588af` | Private repository | | Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.56.16 (revision `a9e52499`, SHA-256 `e433e532b17bd8ce76030156285816e33d44e93edabd2204adbef71d19a68712`) and 1.58.0 (revision `47bbb58d`, SHA-256 `5ba399ea5ab9a6c7d4e9aaf6c623080beb34d9ad5f330f6244dbad72b8d960b6`) | Private repository | | VAT receipts | £170.994bn (2024-25 outturn), £180.169bn (2025-26), £187.704bn (2026-27) | OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8, cash-basis VAT | | Household share of VAT liabilities | 0.7 | HMRC VAT gap estimates (around 70% of the VAT total theoretical liability); `household_share_of_receipts` | @@ -56,8 +64,9 @@ the calculation below, whenever the dataset it should match changes. | Statutory rates | 20% standard, 5% reduced | `gov.hmrc.vat` | Model year 2024 is the 2024-25 fiscal year. All dataset totals are weighted -sums from `Microsimulation`, computed with policyengine-uk at this branch and -policyengine-core 3.33.2. +sums from `Microsimulation`. The `a133f37a` and `6cba5d71` rows were computed +with policyengine-uk at this branch's base (main `c0252415a`) and +policyengine-core 3.33.2; the other rows come from the first derivation. ### Formula @@ -75,6 +84,12 @@ equals receipts when the product matches the dataset. | Dataset | Year | Standard-rated base | Reduced-rated base | Pre-scaling VAT | Receipts | Product | Survey coverage | |---|---|---|---|---|---|---|---| +| Microcosm UK 2024-25 (`a133f37a`) | 2024-25 | £435.60bn | £26.76bn | £88.46bn | £171.0bn | 0.517 | 0.739 | +| Microcosm UK 2024-25 (`a133f37a`) | 2025-26 | £453.65bn | £27.87bn | £92.12bn | £180.2bn | 0.511 | 0.730 | +| Microcosm UK 2024-25 (`a133f37a`) | 2026-27 | £465.85bn | £28.62bn | £94.60bn | £187.7bn | 0.504 | 0.720 | +| Microcosm UK 2024-25 (`6cba5d71`) | 2024-25 | | | £88.51bn | £171.0bn | 0.518 | 0.739 | +| Microcosm UK 2024-25 (`6cba5d71`) | 2025-26 | | | £92.18bn | £180.2bn | 0.512 | 0.731 | +| Microcosm UK 2024-25 (`6cba5d71`) | 2026-27 | | | £94.66bn | £187.7bn | 0.504 | 0.720 | | Microcosm UK 2024-25 (`f9d1922c`) | 2024-25 | £396.15bn | £25.02bn | £80.48bn | £171.0bn | 0.471 | 0.672 | | Microcosm UK 2024-25 (`f9d1922c`) | 2025-26 | £412.57bn | £26.06bn | £83.82bn | £180.2bn | 0.465 | 0.665 | | Microcosm UK 2024-25 (`f9d1922c`) | 2026-27 | £423.66bn | £26.76bn | £86.07bn | £187.7bn | 0.459 | 0.655 | @@ -85,34 +100,32 @@ equals receipts when the product matches the dataset. | Enhanced FRS 2024-25, uk-data 1.58.0 | 2025-26 | £593.92bn | £34.54bn | £120.51bn | £180.2bn | 0.669 | 0.956 | | Enhanced FRS 2024-25, uk-data 1.58.0 | 2026-27 | £609.89bn | £35.47bn | £123.75bn | £187.7bn | 0.659 | 0.942 | -The parameter takes the Microcosm 2024-25 value, 0.67 (product 0.469), the -latest outturn year. +The parameter takes the `a133f37a` value for the latest outturn year, 0.739, +rounded to 0.74 (product 0.518). -With that product, total `vat` against receipts is: +With that product, total `vat` against receipts, from runs at this branch: | Dataset | 2024-25 | 2025-26 | 2026-27 | |---|---|---|---| -| Microcosm UK 2024-25 (`f9d1922c`) | £171.6bn (+0.4%) | £178.7bn (-0.8%) | £183.5bn (-2.2%) | -| Enhanced FRS 2024-25, uk-data 1.56.16 | £236.7bn (+38.4%) | £246.5bn (+36.8%) | £253.1bn (+34.8%) | -| Enhanced FRS 2024-25, uk-data 1.58.0 | £246.7bn (+44.3%) | £257.0bn (+42.6%) | £263.9bn (+40.6%) | +| Microcosm UK 2024-25 (`a133f37a`) | PENDING | PENDING | PENDING | +| Enhanced FRS 2024-25, uk-data 1.58.1 | PENDING | PENDING | PENDING | ### Bound Total VAT depends only on the product, so the household share being a rounded "around 70%" changes how the product is split between the two factors, -not the totals. On Microcosm the product is bounded by: +not the totals. On `a133f37a` the product is bounded by: -- the receipts route across 2024-25 to 2026-27: 0.459 to 0.471 (coverage 0.655 - to 0.672); +- the receipts route across 2024-25 to 2026-27: 0.504 to 0.517 (coverage 0.720 + to 0.739); - the ready reckoner, which does not use the level of receipts: one point on - the standard rate raises £4.24bn before scaling in 2026-27 (1% of - £423.66bn), so matching HMRC's £8.8bn needs a product of 0.481 (coverage - 0.688). - -So on Microcosm the product lies between 0.459 and 0.481, and the coverage -between 0.655 and 0.688 at a household share of 0.7. The value used, 0.469 -(0.67), sits inside that range, and the totals it gives are within 2.2% of -receipts from 2024-25 to 2026-27. The implied product moves by less than 3% + the standard rate raises £4.66bn before scaling in 2026-27 (1% of + £465.85bn), so matching HMRC's £8.8bn needs a product of 0.529 (coverage + 0.756). With the product used, a reform run of one point costs PENDING. + +So on `a133f37a` the product lies between 0.504 and 0.529, and the coverage +between 0.720 and 0.756 at a household share of 0.7. The value used, 0.518 +(0.74), sits inside that range. The implied product moves by less than 3% over those years, so the factors are not dated by year. ### Reproducing @@ -129,7 +142,7 @@ path = hf_hub_download( "policyengine/populace-uk-private", "microcosm_uk_2024_25.h5", repo_type="dataset", - revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88", + revision="microcosm-uk-2024-25-national-20261023T075039Z-a133f37a", ) sim = Microsimulation(dataset=path) # OBR March 2026 EFO, table 3.8, cash-basis VAT receipts diff --git a/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml b/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml index 54e8d87667..bfc2ffb8dd 100644 --- a/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml @@ -1,4 +1,4 @@ -description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.67 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead. +description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.74 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead. metadata: unit: /1 period: year @@ -8,4 +8,4 @@ metadata: - title: docs/engineering/vat-coverage.md (derivation of the two factors this combines) href: https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/engineering/vat-coverage.md values: - 2010-01-01: 0.469 + 2010-01-01: 0.518 diff --git a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml index e5a16038db..9f0f250ca7 100644 --- a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml @@ -6,24 +6,28 @@ metadata: derivation: >- Calibrated to the certified Microcosm UK 2024-25 national release only (policyengine/populace-uk-private, tag - microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68, revision - f9d1922c). Pre-scaling VAT (standard rate times full_rate_vat_consumption - plus reduced rate times reduced_rate_vat_consumption) on that release is - 0.20 x £396.15bn + 0.05 x £25.02bn = £80.48bn in 2024. OBR VAT receipts - for 2024-25 (outturn, March 2026 EFO table 3.8, cash basis) are £170.994bn. - 80.48 / 170.994 = 0.471 for the product of the two factors; divided by the - household share of 0.7 gives 0.67. Across 2024-25 to 2026-27 the product is - 0.459 to 0.471, and the HMRC ready reckoner (£8.8bn for one point on the - standard rate in 2026-27) implies 0.481, so the coverage lies between 0.655 - and 0.688. On Enhanced FRS 2024-25 the same calculation gives a coverage of - 0.927 (policyengine-uk-data 1.56.16) or 0.967 (1.58.0), so this value puts - VAT on that dataset 38% or 44% above 2024-25 receipts. Re-derive when the - dataset it should match changes, with docs/engineering/vat-coverage.md, - which lists the inputs, file hashes and per-dataset results. + microcosm-uk-2024-25-national-20261023T075039Z-a133f37a, SHA-256 + 64f83900ca3a1242...). Pre-scaling VAT (standard rate times + full_rate_vat_consumption plus reduced rate times + reduced_rate_vat_consumption) on that release is 0.20 x £435.60bn + 0.05 x + £26.76bn = £88.46bn in 2024. OBR VAT receipts for 2024-25 (outturn, March + 2026 EFO table 3.8, cash basis) are £170.994bn. 88.46 / 170.994 = 0.517 for + the product of the two factors; divided by the household share of 0.7 gives + 0.739, used as 0.74 (product 0.518). Across 2024-25 to 2026-27 the product + is 0.504 to 0.517, and the HMRC ready reckoner (£8.8bn for one point on the + standard rate in 2026-27) implies 0.529, so the coverage lies between 0.720 + and 0.756. The cut it replaces (tag + microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68) recorded about 10% + less standard-rated consumption and gave 0.67; the 6cba5d71 cut gives the + same value as a133f37a to three decimal places. Enhanced FRS 2024-25 + records more consumption (implied coverage 0.90 to 0.97), so this value + puts VAT on that dataset above receipts. Re-derive when the dataset it + should match changes, with docs/engineering/vat-coverage.md, which lists + the inputs, file hashes and per-dataset results. reference: - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8 href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ - title: HMRC, Direct effects of illustrative tax changes bulletin (June 2025) href: https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes/direct-effects-of-illustrative-tax-changes-bulletin-january-2025 values: - 2010-01-01: 0.67 + 2010-01-01: 0.74 diff --git a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml index 934f838e84..8e6eed4fd2 100644 --- a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml @@ -5,9 +5,9 @@ full_rate_vat_consumption: 10_000 reduced_rate_vat_consumption: 1_000 output: - # (10,000 x 20% + 1,000 x 5%) / (0.67 x 0.7) = 2,050 / 0.469 - vat: 4_371.00 - baseline_vat: 4_371.00 + # (10,000 x 20% + 1,000 x 5%) / (0.74 x 0.7) = 2,050 / 0.518 + vat: 3_957.53 + baseline_vat: 3_957.53 - name: With both factors at one, VAT is the statutory liability on recorded consumption period: 2025 diff --git a/policyengine_uk/tests/test_vat_grossing_legacy.py b/policyengine_uk/tests/test_vat_grossing_legacy.py index 32c98a28c7..d3f798de36 100644 --- a/policyengine_uk/tests/test_vat_grossing_legacy.py +++ b/policyengine_uk/tests/test_vat_grossing_legacy.py @@ -28,7 +28,7 @@ def test_legacy_default_is_the_product_of_the_two_factors(): year ) * p.vat.household_share_of_receipts(year) assert p.microdata_vat_coverage(year) == pytest.approx(product) - assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.469) + assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.518) def test_reform_dict_to_legacy_path_changes_vat(): diff --git a/policyengine_uk/tests/test_vat_grossing_properties.py b/policyengine_uk/tests/test_vat_grossing_properties.py new file mode 100644 index 0000000000..b14948e51c --- /dev/null +++ b/policyengine_uk/tests/test_vat_grossing_properties.py @@ -0,0 +1,77 @@ +"""Properties of VAT grossing that hold for every input (#2633). + +`vat` is the statutory liability on recorded consumption divided by one +grossing factor. The factor is the survey coverage times the household share +of receipts, unless a reform moves the deprecated combined parameter, in which +case that value is used. +""" + +from types import SimpleNamespace + +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from policyengine_uk import Simulation +from policyengine_uk.system import system +from policyengine_uk.utils.vat import LEGACY_COVERAGE_DEFAULT, vat_grossing_factor + +SHARE = st.floats(min_value=0.01, max_value=1, allow_nan=False) +SPEND = st.floats(min_value=0, max_value=1e6, allow_nan=False) + + +def params(legacy, coverage, share): + return SimpleNamespace( + microdata_vat_coverage=legacy, + vat=SimpleNamespace( + survey_consumption_coverage=coverage, + household_share_of_receipts=share, + ), + ) + + +@given(coverage=SHARE, share=SHARE) +def test_untouched_legacy_factor_uses_the_product_of_the_two(coverage, share): + factor = vat_grossing_factor(params(LEGACY_COVERAGE_DEFAULT, coverage, share)) + assert factor == pytest.approx(coverage * share) + + +@given(legacy=SHARE, coverage=SHARE, share=SHARE) +def test_moved_legacy_factor_overrides_the_two(legacy, coverage, share): + factor = vat_grossing_factor(params(legacy, coverage, share)) + if legacy == pytest.approx(LEGACY_COVERAGE_DEFAULT, rel=1e-5, abs=1e-8): + assert factor == pytest.approx(coverage * share) + else: + assert factor == legacy + + +@settings(max_examples=5, deadline=None) +@given(spend=st.lists(st.tuples(SPEND, SPEND), min_size=1, max_size=20)) +def test_vat_is_statutory_liability_over_the_shipped_product(spend): + # One simulation of many households, so the property is checked on a + # vector: VAT is linear in each household's recorded consumption, with + # no cross-household term. + p = system.parameters.gov + product = p.simulation.vat.survey_consumption_coverage( + 2025 + ) * p.simulation.vat.household_share_of_receipts(2025) + names = [f"h{i}" for i in range(len(spend))] + situation = { + "people": {f"p{i}": {"age": {2025: 40}} for i in range(len(spend))}, + "benunits": {f"b{i}": {"members": [f"p{i}"]} for i in range(len(spend))}, + "households": { + name: { + "members": [f"p{i}"], + "full_rate_vat_consumption": {2025: full}, + "reduced_rate_vat_consumption": {2025: reduced}, + } + for i, (name, (full, reduced)) in enumerate(zip(names, spend)) + }, + } + vat = Simulation(situation=situation).calculate("vat", 2025) + for value, (full, reduced) in zip(vat, spend): + expected = ( + full * p.hmrc.vat.standard_rate(2025) + + reduced * p.hmrc.vat.reduced_rate(2025) + ) / product + assert value == pytest.approx(expected, rel=1e-5, abs=1e-2) From ad53a004fe5a578ea991d7b898f2e9c9a9038595 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 24 Oct 2026 08:04:39 -0400 Subject: [PATCH 02/11] Pin 2024 VAT to the product and record the cuts' revisions The deprecated combined factor has one undated value, so every year, 2024 included, takes the two factors' product. A YAML case pins vat in 2024; adding a dated entry to the deprecated factor makes it fail. The doc's inputs now give the Hugging Face revisions of the a133f37a and 6cba5d71 tags, and the reproduction snippet pins a133f37a's revision. Refs #2633 Co-Authored-By: Claude Opus 5.5 --- docs/engineering/vat-coverage.md | 6 +++--- .../policy/baseline/gov/hmrc/vat/vat_grossing.yaml | 12 ++++++++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index b6169c2261..81eb6c62c1 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -54,8 +54,8 @@ about 10% more standard-rated consumption. With 0.67, total `vat` on | Input | Value | Source | |---|---|---| -| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261023T075039Z-a133f37a`, SHA-256 `64f83900ca3a1242fb017585234844ce89f743d32e41cd2b6558057777f914aa` | Private repository | -| Microcosm UK 2024-25 national release, cross-check | tag `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`, SHA-256 `810ba980005a8b907cb51464279fa5e0125464b7af9a4871b7f6744c30951ece` | Private repository | +| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261023T075039Z-a133f37a` (revision `c3a751d9`), SHA-256 `64f83900ca3a1242fb017585234844ce89f743d32e41cd2b6558057777f914aa` | Private repository | +| Microcosm UK 2024-25 national release, cross-check | tag `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71` (revision `bb225848`), SHA-256 `810ba980005a8b907cb51464279fa5e0125464b7af9a4871b7f6744c30951ece` | Private repository | | Microcosm UK 2024-25 national release, first derivation | tag `microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68` (revision `f9d1922c`), SHA-256 `aa31bdf67c977927ea2b325567d1cf7a79d94381239bc79918a0a0fc9c9588af` | Private repository | | Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.56.16 (revision `a9e52499`, SHA-256 `e433e532b17bd8ce76030156285816e33d44e93edabd2204adbef71d19a68712`) and 1.58.0 (revision `47bbb58d`, SHA-256 `5ba399ea5ab9a6c7d4e9aaf6c623080beb34d9ad5f330f6244dbad72b8d960b6`) | Private repository | | VAT receipts | £170.994bn (2024-25 outturn), £180.169bn (2025-26), £187.704bn (2026-27) | OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8, cash-basis VAT | @@ -142,7 +142,7 @@ path = hf_hub_download( "policyengine/populace-uk-private", "microcosm_uk_2024_25.h5", repo_type="dataset", - revision="microcosm-uk-2024-25-national-20261023T075039Z-a133f37a", + revision="c3a751d911ac82e482d298dda0b6f2c949f4ed4d", # tag ...-a133f37a ) sim = Microsimulation(dataset=path) # OBR March 2026 EFO, table 3.8, cash-basis VAT receipts diff --git a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml index 8e6eed4fd2..1701290963 100644 --- a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml @@ -9,6 +9,18 @@ vat: 3_957.53 baseline_vat: 3_957.53 +- name: VAT in 2024 is grossed up by the same product as later years + period: 2024 + absolute_error_margin: 0.01 + input: + full_rate_vat_consumption: 10_000 + reduced_rate_vat_consumption: 1_000 + output: + # The deprecated combined factor has one undated value, so 2024 takes the + # two factors' product too: 2,050 / 0.518 + vat: 3_957.53 + baseline_vat: 3_957.53 + - name: With both factors at one, VAT is the statutory liability on recorded consumption period: 2025 absolute_error_margin: 0.01 From e1ff49300f5a5e93e0c146dc7665fbf747c51405 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 24 Oct 2026 10:19:11 -0400 Subject: [PATCH 03/11] Cite the 6cba reform runs and align the legacy-factor property with np.isclose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review r1 of #2634 (Sol, REQUEST_CHANGES at b6d8af5) found: - The £9.94bn one-point figure was described as a reform run without the runs being cited. The doc now names them: reform minus baseline `vat` and `gov_balance` on 6cba5d71 (revision bb225848), policyengine-uk 2.135.7 and 2.152.6, £9.94bn in 2026-27 and £9.68bn in 2025-26. - The moved-legacy property classified closeness with pytest.approx, which takes the larger tolerance, while the helper uses np.isclose, which adds them. A legacy value of 0.518005185 failed the property. The band is now stated once (1e-8 + 1e-5 x the default), the moved property draws values outside it, a new property draws values inside it, and fixed cases sit at 0.99 and 1.01 bands either side. Changing the helper to exact equality fails the inside cases. - The helper properties kept Hypothesis's 200 ms deadline and flaked on a loaded host; they now have no deadline. - The doc attributed both cuts' rows to main and core 3.33.2; it now gives each cut's recorded versions. - The parameter said 6cba5d71 matches a133f37a to three decimal places; it now says the 2024-25 coverage (0.739) matches. Co-Authored-By: Claude Opus 5.5 --- docs/engineering/vat-coverage.md | 19 ++++++---- .../vat/survey_consumption_coverage.yaml | 2 +- .../tests/test_vat_grossing_properties.py | 36 ++++++++++++++++--- 3 files changed, 45 insertions(+), 12 deletions(-) diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index 81eb6c62c1..c08d6d326a 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -45,9 +45,13 @@ should match changes. The first derivation (#2187, 0.67) used the Microcosm release `f9d1922c`. That release was replaced by the `6cba5d71` and `a133f37a` cuts, which record about 10% more standard-rated consumption. With 0.67, total `vat` on -`a133f37a` ran 7.5% to 10.3% above receipts in 2024-25 to 2026-27, and on -`6cba5d71` a reform run of one point on the standard rate cost £9.94bn in -2026-27, 12.9% above HMRC's ready reckoner. This derivation repeats the same calculation on `a133f37a` +`a133f37a` ran 7.5% to 10.3% above receipts in 2024-25 to 2026-27. On +`6cba5d71` (revision `bb225848`), a reform run that raises the standard rate +from 20% to 21% changed total `vat` by £9.94bn in 2026-27 (£9.68bn in +2025-26), 12.9% above HMRC's ready reckoner. That is the reform run minus the +baseline run, with policyengine-uk 2.135.7 and again with 2.152.6, which give +the same figure; `gov_balance` changed by the same amount. This derivation +repeats the same calculation on `a133f37a` (PolicyEngine/policyengine-uk#2633). ### Inputs @@ -64,9 +68,12 @@ about 10% more standard-rated consumption. With 0.67, total `vat` on | Statutory rates | 20% standard, 5% reduced | `gov.hmrc.vat` | Model year 2024 is the 2024-25 fiscal year. All dataset totals are weighted -sums from `Microsimulation`. The `a133f37a` and `6cba5d71` rows were computed -with policyengine-uk at this branch's base (main `c0252415a`) and -policyengine-core 3.33.2; the other rows come from the first derivation. +sums from `Microsimulation`. The `a133f37a` rows were computed with +policyengine-uk 2.152.7 (this branch's base, main `c0252415a`) and +policyengine-core 3.33.2, and again with policyengine-uk 2.147.2 and +policyengine-core 3.33.3, which give the same totals. The `6cba5d71` rows were +computed with policyengine-uk 2.147.2 and policyengine-core 3.33.3. The other +rows come from the first derivation. ### Formula diff --git a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml index 9f0f250ca7..6ced1c592b 100644 --- a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml @@ -19,7 +19,7 @@ metadata: and 0.756. The cut it replaces (tag microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68) recorded about 10% less standard-rated consumption and gave 0.67; the 6cba5d71 cut gives the - same value as a133f37a to three decimal places. Enhanced FRS 2024-25 + same 2024-25 coverage, 0.739. Enhanced FRS 2024-25 records more consumption (implied coverage 0.90 to 0.97), so this value puts VAT on that dataset above receipts. Re-derive when the dataset it should match changes, with docs/engineering/vat-coverage.md, which lists diff --git a/policyengine_uk/tests/test_vat_grossing_properties.py b/policyengine_uk/tests/test_vat_grossing_properties.py index b14948e51c..e3eaf72e7f 100644 --- a/policyengine_uk/tests/test_vat_grossing_properties.py +++ b/policyengine_uk/tests/test_vat_grossing_properties.py @@ -9,7 +9,7 @@ from types import SimpleNamespace import pytest -from hypothesis import given, settings +from hypothesis import assume, given, settings from hypothesis import strategies as st from policyengine_uk import Simulation @@ -18,6 +18,9 @@ SHARE = st.floats(min_value=0.01, max_value=1, allow_nan=False) SPEND = st.floats(min_value=0, max_value=1e6, allow_nan=False) +# A legacy value counts as untouched when it is within NumPy's default +# closeness of the declared default: 1e-8 plus 1e-5 times the default. +BAND = 1e-8 + 1e-5 * abs(LEGACY_COVERAGE_DEFAULT) def params(legacy, coverage, share): @@ -30,19 +33,42 @@ def params(legacy, coverage, share): ) +# The helper properties check algebra and routing, not speed, so they have no +# deadline: a slow first example on a loaded host is not a failure. +@settings(deadline=None) @given(coverage=SHARE, share=SHARE) def test_untouched_legacy_factor_uses_the_product_of_the_two(coverage, share): factor = vat_grossing_factor(params(LEGACY_COVERAGE_DEFAULT, coverage, share)) assert factor == pytest.approx(coverage * share) +@settings(deadline=None) +@given( + offset=st.floats(min_value=-0.99, max_value=0.99), + coverage=SHARE, + share=SHARE, +) +def test_legacy_factor_within_the_band_counts_as_untouched(offset, coverage, share): + legacy = LEGACY_COVERAGE_DEFAULT + offset * BAND + factor = vat_grossing_factor(params(legacy, coverage, share)) + assert factor == pytest.approx(coverage * share) + + +@settings(deadline=None) @given(legacy=SHARE, coverage=SHARE, share=SHARE) def test_moved_legacy_factor_overrides_the_two(legacy, coverage, share): + assume(abs(legacy - LEGACY_COVERAGE_DEFAULT) > 1.01 * BAND) factor = vat_grossing_factor(params(legacy, coverage, share)) - if legacy == pytest.approx(LEGACY_COVERAGE_DEFAULT, rel=1e-5, abs=1e-8): - assert factor == pytest.approx(coverage * share) - else: - assert factor == legacy + assert factor == legacy + + +@pytest.mark.parametrize( + "multiple, moved", [(-1.01, True), (-0.99, False), (0.99, False), (1.01, True)] +) +def test_legacy_factor_either_side_of_the_band(multiple, moved): + legacy = LEGACY_COVERAGE_DEFAULT + multiple * BAND + factor = vat_grossing_factor(params(legacy, 0.5, 0.7)) + assert factor == (legacy if moved else pytest.approx(0.35)) @settings(max_examples=5, deadline=None) From 66533369d19501c99f81d69d53d0f7666e641f97 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 2 Nov 2026 16:09:42 +0000 Subject: [PATCH 04/11] Apply the school share in the VAT grossing property's oracle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vat takes private school fees' share of VAT receipts out of the grossed-up amount (from #2319, merged into this branch). The property's oracle divided the statutory liability by the shipped product without that step, so every example with more than about £0.56 of liability failed by the share (0.8353% in 2025). The oracle now reads the share from the parameters, asserts the households carry no energy-bill VAT, and always runs the minimal case as an explicit example. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LwQq6J9S5H2Fv9MFjnDPCr --- changelog.d/2633.fixed.md | 2 +- .../tests/test_vat_grossing_properties.py | 62 ++++++++++++------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/changelog.d/2633.fixed.md b/changelog.d/2633.fixed.md index 0e5e7d303f..1ada80c7d3 100644 --- a/changelog.d/2633.fixed.md +++ b/changelog.d/2633.fixed.md @@ -1 +1 @@ -- Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product on a vector of households), a test that 2024 is grossed by the same product as later years, and a record of how the shipped coverage performs on the Microcosm UK cuts certified since `f9d1922c`. The coverage value is unchanged pending re-derivation on the next certified cut (#2633). +Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, less private school fees' share of VAT receipts, on a vector of households), a test that 2024 is grossed by the same product as later years, and a record of how the shipped coverage performs on the Microcosm UK cuts certified since `f9d1922c`. The coverage value is unchanged pending re-derivation on the next certified cut (#2633). diff --git a/policyengine_uk/tests/test_vat_grossing_properties.py b/policyengine_uk/tests/test_vat_grossing_properties.py index e3eaf72e7f..3646610768 100644 --- a/policyengine_uk/tests/test_vat_grossing_properties.py +++ b/policyengine_uk/tests/test_vat_grossing_properties.py @@ -1,15 +1,16 @@ """Properties of VAT grossing that hold for every input (#2633). `vat` is the statutory liability on recorded consumption divided by one -grossing factor. The factor is the survey coverage times the household share -of receipts, unless a reform moves the deprecated combined parameter, in which +grossing factor, less private school fees' share of VAT receipts, plus VAT on +energy bills. The factor is the survey coverage times the household share of +receipts, unless a reform moves the deprecated combined parameter, in which case that value is used. """ from types import SimpleNamespace import pytest -from hypothesis import assume, given, settings +from hypothesis import assume, example, given, settings from hypothesis import strategies as st from policyengine_uk import Simulation @@ -71,33 +72,46 @@ def test_legacy_factor_either_side_of_the_band(multiple, moved): assert factor == (legacy if moved else pytest.approx(0.35)) -@settings(max_examples=5, deadline=None) -@given(spend=st.lists(st.tuples(SPEND, SPEND), min_size=1, max_size=20)) -def test_vat_is_statutory_liability_over_the_shipped_product(spend): - # One simulation of many households, so the property is checked on a - # vector: VAT is linear in each household's recorded consumption, with - # no cross-household term. - p = system.parameters.gov - product = p.simulation.vat.survey_consumption_coverage( - 2025 - ) * p.simulation.vat.household_share_of_receipts(2025) - names = [f"h{i}" for i in range(len(spend))] +def household_vat(spend, year): + """`vat` for one household per (full-rate, reduced-rate) spend pair.""" situation = { - "people": {f"p{i}": {"age": {2025: 40}} for i in range(len(spend))}, + "people": {f"p{i}": {"age": {year: 40}} for i in range(len(spend))}, "benunits": {f"b{i}": {"members": [f"p{i}"]} for i in range(len(spend))}, "households": { - name: { + f"h{i}": { "members": [f"p{i}"], - "full_rate_vat_consumption": {2025: full}, - "reduced_rate_vat_consumption": {2025: reduced}, + "full_rate_vat_consumption": {year: full}, + "reduced_rate_vat_consumption": {year: reduced}, } - for i, (name, (full, reduced)) in enumerate(zip(names, spend)) + for i, (full, reduced) in enumerate(spend) }, } - vat = Simulation(situation=situation).calculate("vat", 2025) + sim = Simulation(situation=situation) + # The households record no energy bills, so no energy-bill VAT is added. + assert (sim.calculate("domestic_energy_vat", year) == 0).all() + return sim.calculate("vat", year) + + +def expected_vat(full, reduced, year): + """Statutory liability over the shipped product, less the school share.""" + p = system.parameters(year).gov + product = ( + p.simulation.vat.survey_consumption_coverage + * p.simulation.vat.household_share_of_receipts + ) + liability = full * p.hmrc.vat.standard_rate + reduced * p.hmrc.vat.reduced_rate + return liability / product * (1 - p.simulation.vat.private_school_share_of_receipts) + + +# Each spend pair is (full_rate_vat_consumption, reduced_rate_vat_consumption). +@settings(max_examples=5, deadline=None) +@given(spend=st.lists(st.tuples(SPEND, SPEND), min_size=1, max_size=20)) +@example(spend=[(0.0, 15_438.0)]) +def test_vat_is_statutory_liability_over_the_shipped_product(spend): + # One simulation of many households, so the property is checked on a + # vector: VAT is linear in each household's recorded consumption, with + # no cross-household term. + vat = household_vat(spend, 2025) for value, (full, reduced) in zip(vat, spend): - expected = ( - full * p.hmrc.vat.standard_rate(2025) - + reduced * p.hmrc.vat.reduced_rate(2025) - ) / product + expected = expected_vat(full, reduced, 2025) assert value == pytest.approx(expected, rel=1e-5, abs=1e-2) From 36bbbf42f34a23968159406d1b6d8942ebcd6e1c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 2 Nov 2026 17:24:01 +0000 Subject: [PATCH 05/11] Type the VAT fragment as added and pin the standard rate in the example The fragment moves to 2633.added.md with its text unchanged: against main this branch adds tests and docs and changes no model output. The statutory property's fixed example gains a full-rate household, so a formula that charges standard-rated spending at another rate fails on every run, not only on random draws. The module and oracle docstrings say the school share multiplies the grossed-up VAT by one minus the share. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01N2tyHQ6oJEGs3igENBpYcM --- changelog.d/{2633.fixed.md => 2633.added.md} | 0 .../tests/test_vat_grossing_properties.py | 12 ++++++------ 2 files changed, 6 insertions(+), 6 deletions(-) rename changelog.d/{2633.fixed.md => 2633.added.md} (100%) diff --git a/changelog.d/2633.fixed.md b/changelog.d/2633.added.md similarity index 100% rename from changelog.d/2633.fixed.md rename to changelog.d/2633.added.md diff --git a/policyengine_uk/tests/test_vat_grossing_properties.py b/policyengine_uk/tests/test_vat_grossing_properties.py index 3646610768..654d5bdfcb 100644 --- a/policyengine_uk/tests/test_vat_grossing_properties.py +++ b/policyengine_uk/tests/test_vat_grossing_properties.py @@ -1,10 +1,10 @@ """Properties of VAT grossing that hold for every input (#2633). `vat` is the statutory liability on recorded consumption divided by one -grossing factor, less private school fees' share of VAT receipts, plus VAT on -energy bills. The factor is the survey coverage times the household share of -receipts, unless a reform moves the deprecated combined parameter, in which -case that value is used. +grossing factor, times one minus private school fees' share of VAT receipts, +plus VAT on energy bills, which is not grossed up. The factor is the survey +coverage times the household share of receipts, unless a reform moves the +deprecated combined parameter, in which case that value is used. """ from types import SimpleNamespace @@ -93,7 +93,7 @@ def household_vat(spend, year): def expected_vat(full, reduced, year): - """Statutory liability over the shipped product, less the school share.""" + """Statutory liability over the shipped product, times (1 - school share).""" p = system.parameters(year).gov product = ( p.simulation.vat.survey_consumption_coverage @@ -106,7 +106,7 @@ def expected_vat(full, reduced, year): # Each spend pair is (full_rate_vat_consumption, reduced_rate_vat_consumption). @settings(max_examples=5, deadline=None) @given(spend=st.lists(st.tuples(SPEND, SPEND), min_size=1, max_size=20)) -@example(spend=[(0.0, 15_438.0)]) +@example(spend=[(0.0, 15_438.0), (10_000.0, 0.0)]) def test_vat_is_statutory_liability_over_the_shipped_product(spend): # One simulation of many households, so the property is checked on a # vector: VAT is linear in each household's recorded consumption, with From 9419afbaadf833473f68c53582d83e6ccbc684c3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 09:51:37 +0000 Subject: [PATCH 06/11] Move the VAT survey coverage to 0.69 on Microcosm a6299504 Re-derive gov.simulation.vat.survey_consumption_coverage on the certified Microcosm UK 2024-25 national cut a6299504, whose fit holds obr.vat out: 81.48 / (0.7 x (170.994 - 2.33)) = 0.690 in 2024-25. The deprecated combined factor moves to the product, 0.483, and the parameter's reference adds the OBR receipts workbook's own URL. Tests that pin the product move with it: the 2024 and 2025 cases in vat_grossing.yaml, the legacy default in test_vat_grossing_legacy.py, FACTOR in test_private_school_vat_receipts.py, whose published receipts check now uses a6299504's rounded inputs, and a stale illustrative figure in test_scenario_baseline_parameters.py. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WR8LHapmX36bpbh3SMixXg --- changelog.d/2633.changed.md | 1 + .../simulation/microdata_vat_coverage.yaml | 4 +- .../vat/survey_consumption_coverage.yaml | 37 ++++++++++--------- .../baseline/gov/hmrc/vat/vat_grossing.yaml | 12 +++--- .../tests/test_private_school_vat_receipts.py | 14 +++---- .../test_scenario_baseline_parameters.py | 4 +- .../tests/test_vat_grossing_legacy.py | 2 +- 7 files changed, 39 insertions(+), 35 deletions(-) create mode 100644 changelog.d/2633.changed.md diff --git a/changelog.d/2633.changed.md b/changelog.d/2633.changed.md new file mode 100644 index 0000000000..8abad51e1f --- /dev/null +++ b/changelog.d/2633.changed.md @@ -0,0 +1 @@ +Re-derived the survey VAT consumption coverage (`gov.simulation.vat.survey_consumption_coverage`) on the certified Microcosm UK 2024-25 national cut `a6299504` and moved it from 0.67 to 0.69: on that cut, 2024-25 total `vat` comes to £176.06bn at 0.67, 3.0% above OBR receipts of £170.994bn, and £171.02bn at 0.69, 0.02% above. Moved the deprecated combined factor `gov.simulation.microdata_vat_coverage` to the new product, 0.483. diff --git a/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml b/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml index 54e8d87667..1ad053af43 100644 --- a/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml @@ -1,4 +1,4 @@ -description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.67 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead. +description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.69 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead. metadata: unit: /1 period: year @@ -8,4 +8,4 @@ metadata: - title: docs/engineering/vat-coverage.md (derivation of the two factors this combines) href: https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/engineering/vat-coverage.md values: - 2010-01-01: 0.469 + 2010-01-01: 0.483 diff --git a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml index 121d970d74..46396cc79f 100644 --- a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml @@ -4,22 +4,23 @@ metadata: period: year economy: false derivation: >- - Calibrated to the certified Microcosm UK 2024-25 national release only - (policyengine/populace-uk-private, tag - microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68, revision - f9d1922c). VAT on household energy bills (domestic_energy_vat) is not - grossed up, so the coverage applies only to VAT on other recorded - consumption. On that release, pre-scaling VAT on that consumption is 0.20 x - £396.15bn = £79.23bn in 2024 (the generic reduced-rate base is empty once - domestic fuel and power is taken out), and VAT on energy bills is £1.99bn. - OBR VAT receipts for 2024-25 (outturn, March 2026 EFO table 3.8, cash - basis) are £170.994bn. Coverage = 79.23 / (0.7 x (170.994 - 1.99)) = 0.670. - Before energy bills were split out, the same calculation gave 0.672, so - the rounded value is unchanged. Across 2024-25 to 2026-27 the coverage is - 0.650 to 0.670, and the HMRC ready reckoner (£8.8bn for one point on the - standard rate in 2026-27) implies 0.688. On Enhanced FRS 2024-25 the same - calculation gives 0.927 (policyengine-uk-data 1.56.16) or 0.967 (1.58.0), - so this value puts total VAT on that dataset 38% or 44% above 2024-25 + This value matches 2024-25 VAT receipts on one dataset only: the certified + Microcosm UK 2024-25 national release (policyengine/populace-uk-private, + tag microcosm-uk-2024-25-national-20261025T091414Z-a6299504, revision + 994cac89), whose fit excludes the obr.vat receipts target. The vat formula + adds VAT on household energy bills (domestic_energy_vat) without grossing + it up, so the coverage applies only to VAT on other recorded consumption. + On that release, pre-scaling VAT on that consumption comes to 0.20 x + £407.40bn = £81.48bn in 2024 (reduced_rate_vat_consumption takes domestic + fuel and power's share of spending out of the reduced-rate share, which + leaves the generic reduced-rate base empty), and VAT on energy bills comes + to £2.33bn. OBR VAT receipts for 2024-25 (outturn, March 2026 EFO table + 3.8, cash basis) come to £170.994bn. Coverage = 81.48 / (0.7 x (170.994 - + 2.33)) = 0.690. Across 2024-25 to 2026-27 the coverage runs from 0.666 to + 0.690, and the HMRC ready reckoner (£8.8bn for one point on the standard + rate in 2026-27) implies 0.704. On Enhanced FRS 2024-25 + (policyengine-uk-data 1.58.1) the same calculation gives 0.954 for + 2024-25, so this value puts total VAT on that file 38% above 2024-25 receipts. Re-derive when the dataset it should match changes, with docs/engineering/vat-coverage.md, which lists the inputs, file hashes and per-dataset results. These receipts already include private school VAT. @@ -33,7 +34,9 @@ metadata: reference: - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8 href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts (efo-march-2026-detailed-forecast-tables-receipts.xlsx), sheet 3.8, row 13 (Value added tax, cash basis), cell D13 for 2024-25 outturn (£bn) + href: https://obr.uk/docs/d055fbf02d5b3g6jq8l2/efo-march-2026-detailed-forecast-tables-receipts.xlsx - title: HMRC, Direct effects of illustrative tax changes bulletin (June 2025) href: https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes/direct-effects-of-illustrative-tax-changes-bulletin-january-2025 values: - 2010-01-01: 0.67 + 2010-01-01: 0.69 diff --git a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml index e7e4f54af0..2fc324c608 100644 --- a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml @@ -5,11 +5,11 @@ full_rate_vat_consumption: 10_000 reduced_rate_vat_consumption: 1_000 output: - # (10,000 x 20% + 1,000 x 5%) / (0.67 x 0.7) = 2,050 / 0.469 = 4,371.00, + # (10,000 x 20% + 1,000 x 5%) / (0.69 x 0.7) = 2,050 / 0.483 = 4,244.31, # less private school fees' official share of 2025-26 VAT receipts # (0.008353), which private_school_vat counts separately. - vat: 4_334.49 - baseline_vat: 4_334.49 + vat: 4_208.85 + baseline_vat: 4_208.85 - name: VAT in 2024 is grossed up by the same product as later years period: 2024 @@ -19,9 +19,9 @@ reduced_rate_vat_consumption: 1_000 output: # The deprecated combined factor has one undated value, so 2024 takes the - # two factors' product too: 2,050 / 0.469 - vat: 4_371.00 - baseline_vat: 4_371.00 + # two factors' product too: 2,050 / 0.483 + vat: 4_244.31 + baseline_vat: 4_244.31 - name: With both factors at one, VAT is the statutory liability on recorded consumption period: 2025 diff --git a/policyengine_uk/tests/test_private_school_vat_receipts.py b/policyengine_uk/tests/test_private_school_vat_receipts.py index c29f5b3d24..b6116da959 100644 --- a/policyengine_uk/tests/test_private_school_vat_receipts.py +++ b/policyengine_uk/tests/test_private_school_vat_receipts.py @@ -37,7 +37,7 @@ LEGACY = "gov.simulation.microdata_vat_coverage" FEES = "gov.simulation.private_school_vat.private_school_fees" BASIS = "gov.simulation.private_school_vat.private_school_vat_basis" -FACTOR = 0.67 * 0.7 +FACTOR = 0.69 * 0.7 # HMRC's OBR-certified costing of VAT on private school fees (Autumn Budget # 2024 Table 5.1, £m) and OBR cash-basis VAT receipts (March 2026 EFO detailed # forecast tables, receipts, sheet 3.8, row 13, cells E13:I13, £m), by the @@ -190,13 +190,13 @@ def test_vat_reads_neither_attendance_nor_fees(): def test_published_microcosm_receipts_check_is_unchanged(): - # Rounded aggregate inputs published in docs/engineering/vat-coverage.md, - # with energy bills split out: the standard-rated base and the energy-bill - # VAT, which is added after grossing. Model year 2024 has no school charge - # and no share to reallocate. - sim = simulation(year=2024, full=396.15e9, reduced=0, energy_vat=1.99e9) + # Rounded aggregate inputs that docs/engineering/vat-coverage.md publishes + # for Microcosm a6299504, with energy bills split out: the standard-rated + # base, and the energy-bill VAT, which vat adds after grossing. Model year + # 2024 has no school charge and no share to reallocate. + sim = simulation(year=2024, full=407.40e9, reduced=0, energy_vat=2.33e9) combined = amount(sim, "vat", 2024) + amount(sim, "private_school_vat", 2024) - expected = 0.2 * 396.15e9 / FACTOR + 1.99e9 + expected = 0.2 * 407.40e9 / FACTOR + 2.33e9 assert combined == pytest.approx(expected) assert combined / 170.994e9 == pytest.approx(1, rel=0.001) diff --git a/policyengine_uk/tests/test_scenario_baseline_parameters.py b/policyengine_uk/tests/test_scenario_baseline_parameters.py index 1e272b0810..b637a3d285 100644 --- a/policyengine_uk/tests/test_scenario_baseline_parameters.py +++ b/policyengine_uk/tests/test_scenario_baseline_parameters.py @@ -51,8 +51,8 @@ def _vat_change(sim): def test_every_reform_route_scores_vat_against_the_unreformed_baseline(kwargs): sim = Simulation(situation=SITUATION, **kwargs) # All spending here is standard-rated, so raising the rate from 20% to - # 22% raises VAT by a tenth, whatever the grossing factor (526.32 at the - # current factor). + # 22% raises VAT by a tenth, whatever the grossing factor (about 411 at + # the shipped factors). unreformed_vat = float(Simulation(situation=SITUATION).calculate("vat", 2026)[0]) assert _vat_change(sim) == pytest.approx(0.1 * unreformed_vat, abs=0.01) assert _vat_change(sim) > 0 diff --git a/policyengine_uk/tests/test_vat_grossing_legacy.py b/policyengine_uk/tests/test_vat_grossing_legacy.py index 1863aa2f5c..728f16d345 100644 --- a/policyengine_uk/tests/test_vat_grossing_legacy.py +++ b/policyengine_uk/tests/test_vat_grossing_legacy.py @@ -31,7 +31,7 @@ def test_legacy_default_is_the_product_of_the_two_factors(): year ) * p.vat.household_share_of_receipts(year) assert p.microdata_vat_coverage(year) == pytest.approx(product) - assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.469) + assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.483) def test_reform_dict_to_legacy_path_changes_vat(): From c2575c905f5faa50a86817da91e6e3b5ca5b14a0 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 09:51:37 +0000 Subject: [PATCH 07/11] Derive the VAT coverage on a6299504 in the doc and refresh its provenance Add a derivation section for the certified Microcosm cut a6299504: the cut's provenance and certification, the obr.vat exclusion, the OBR receipts workbook (URL, SHA-256, archived copy, sheet, row and cells), the chunked method, the results at 0.67 and 0.69 on the cut and on Enhanced FRS 1.58.1, the bound and a reproducing snippet at revision 994cac89. Name policyengine.py's certified cuts correctly: a6299504 today, 6cba5d71 on 20 October, and a133f37a only proposed in policyengine.py#715. Retitle the October derivation as the earlier one on f9d1922c and date its statements of the value, and point the switch plan at the new cut. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WR8LHapmX36bpbh3SMixXg --- docs/book/assumptions/mp30k-uk-switch-plan.md | 13 +- docs/engineering/vat-coverage.md | 341 +++++++++++++----- 2 files changed, 263 insertions(+), 91 deletions(-) diff --git a/docs/book/assumptions/mp30k-uk-switch-plan.md b/docs/book/assumptions/mp30k-uk-switch-plan.md index 374e9209f3..90a16f7cb7 100644 --- a/docs/book/assumptions/mp30k-uk-switch-plan.md +++ b/docs/book/assumptions/mp30k-uk-switch-plan.md @@ -115,14 +115,13 @@ A documented gate must pass before any environment's default changes: 1. Dataset choice stays a runtime setting: no code in this repo branches on which dataset is loaded. One parameter is tied to the dataset, though: - `gov.simulation.vat.survey_consumption_coverage` is calibrated to one + `gov.simulation.vat.survey_consumption_coverage` matches receipts on one Microcosm UK 2024-25 national cut only - (`microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68`; policyengine.py - 6.10.0 pins the later cut - `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), puts 2024-25 - VAT 38% (policyengine-uk-data 1.56.16) or 44% (1.58.0) above receipts on - the Enhanced FRS 2024-25, and has to be re-derived when the dataset it - should match changes (`docs/engineering/vat-coverage.md`). + (`microcosm-uk-2024-25-national-20261025T091414Z-a6299504`, which + policyengine.py 6.11.1 and 6.12.0 certify), puts 2024-25 VAT 38% above + receipts on the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1), and has + to be re-derived when the dataset it should match changes + (`docs/engineering/vat-coverage.md`). 2. While both are in use, CI runs the dataset-dependent tests on both. 3. The default is set per environment: for direct use of this package through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index eb09a738c6..3781d6cdfc 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -15,7 +15,7 @@ comparison uses `vat + private_school_vat`: theoretical liability; the rest falls on government, charities, VAT-exempt businesses and new housing. This is a published quantity and does not depend on the dataset. -- `gov.simulation.vat.survey_consumption_coverage` (0.67): the share of +- `gov.simulation.vat.survey_consumption_coverage` (0.69): the share of household VAT liability that the dataset's consumption captures. This depends on the dataset's consumption imputation and must be re-derived when the default dataset changes. @@ -32,28 +32,209 @@ Enhanced FRS 2024-25 (policyengine-uk-data 1.56.16). `gov.simulation.microdata_vat_coverage` is kept as a deprecated parameter so that existing reforms to it still work. Its value is now the product of the two -factors (0.469). If a reform sets it to any other value, `vat` and +factors (0.483). If a reform sets it to any other value, `vat` and `baseline_vat` use that value in place of the two factors (`policyengine_uk/utils/vat.py`); otherwise they use the two factors. New reforms should change the two factors instead. ## Which datasets the factor applies to -The survey coverage of 0.67 is calibrated to one dataset: the certified -Microcosm UK 2024-25 national release. It does not carry over to other -datasets, because their consumption imputations capture a different share of -household VAT. On Enhanced FRS 2024-25 it puts VAT 35% to 44% above receipts, -depending on the release and year (see the tables below). Re-derive the coverage, with -the calculation below, whenever the dataset it should match changes. +The survey coverage of 0.69 matches receipts on one dataset: the certified +Microcosm UK 2024-25 national release `a6299504`. It does not carry over to +other datasets, because their consumption imputations capture a different +share of household VAT. On Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1) +it puts total `vat` 32% to 38% above receipts from 2024-25 to 2026-27 (see +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026)). Re-derive +the coverage, with the calculation below, whenever the dataset it should match +changes. + +As of November 2026, policyengine.py certifies `a6299504`: its commit +`324ecad65` certified the cut, release 6.11.1 shipped it, and release 6.12.0 +still certifies it. The coverage comes from that cut, in +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026). +policyengine.py certified `6cba5d71` before it, on 20 October 2026 (commit +`a66e883c4`). policyengine.py#715 proposed `a133f37a` and closed unmerged, so +no policyengine.py release certified that cut. Before the energy split, 0.67 +put VAT 7.5% to 10.3% above receipts on `a133f37a` (see +[Later Microcosm cuts](#later-microcosm-cuts-24-october-2026)). + +## Derivation on `a6299504` (November 2026) + +The parameter takes this derivation's value. The model figures in this section +come from runs on 3 November 2026 with policyengine-uk 2.157.1 (commit +`36bbbf42f`), policyengine-core 3.33.2, NumPy 2.5.4, pandas 3.1.0 and Python +3.14, with `HUGGING_FACE_TOKEN` and `POLICYENGINE_UK_DEFAULT_DATASET` unset. +That commit ships a coverage of 0.67; the runs at 0.69 set +`gov.simulation.vat.survey_consumption_coverage` to 0.69 in every year by +reform. -The Microcosm releases certified since `f9d1922c` record about 10% more -standard-rated consumption. Before the energy split, 0.67 put VAT 7.5% to -10.3% above receipts on them (see -[Later Microcosm cuts](#later-microcosm-cuts-24-october-2026)). The coverage -has not yet been re-derived on them with energy bills split out; that is -PolicyEngine/policyengine-uk#2633. +### Inputs + +| Input | Value | Source | +|---|---|---| +| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face dataset repository `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261025T091414Z-a6299504` (revision `994cac89d902af9a985af7c1182d74717a582ce0`), SHA-256 `b96a8cd625b645a409c063b1de5158532708a2f0d3c3897a794a8fb0bbd5064e`, 342,348,841 bytes; 78,234 households and 181,842 persons, 28.87 million weighted households in 2024-25 | Private repository | +| Build | `uk-frs-calibration-attempt-20261025T091414Z-a6299504`, at Microcosm code pin `67ee2ddf3`, with policyengine-uk 2.151.0 | PolicyEngine/microcosm | +| Certification | policyengine.py commit `324ecad65` ("Certify the Microcosm UK national a6299504 with policyengine-uk 2.151.0", 26 October 2026, through policyengine.py#720); release 6.11.1 (27 October 2026) shipped it, and release 6.12.0 (2 November 2026) still certifies it | PolicyEngine/policyengine.py | +| VAT receipts | £170.994bn (2024-25 outturn), £180.169bn (2025-26), £187.704bn (2026-27) | OBR, Economic and fiscal outlook March 2026, detailed forecast tables: receipts, sheet "3.8 Current receipts (on a cash basis)", row 13 "Value added tax", cells D13, E13 and F13 (see below) | +| Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face model repository `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.58.1 (revision `2526edbbb687a3b54dbb11272e5389e4b62d2ac2`, SHA-256 `056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784`, 128,409,762 bytes), the file `policyengine_uk/tests/certified_dataset.yaml` names | Private repository | +| Household share of VAT liabilities | 0.7 | `household_share_of_receipts` | +| Ready reckoner | £8.8bn for one point on the standard rate, 2026-27 | HMRC, Direct effects of illustrative tax changes (June 2025) | +| Statutory rates | 20% standard, 5% reduced | `gov.hmrc.vat` | + +Microcosm's fit for this cut holds `obr.vat` out under a measure exclusion +(PolicyEngine/microcosm#1278) that runs from 24 October 2026 to 24 January +2027, so the fitted weights don't target the receipts that this derivation +compares VAT with. The cut's calibration diagnostics carry one warn-only VAT +warning, `uk_consumption_drift`: the total `full_rate_vat_base` moves by more +than 2% of its design value between the design and the final weights. + +OBR publishes the receipts workbook at +. +On 3 November 2026 that URL served 187,899 bytes with SHA-256 +`f65b6cb7f96931d4abd85940794eb60f6f65afca8bbf16c74a4548bf92766a28`, last +modified on 3 March 2026, and the +[Internet Archive's copy](https://web.archive.org/web/20261007143332/https://obr.uk/docs/d055fbf02d5b3g6jq8l2/efo-march-2026-detailed-forecast-tables-receipts.xlsx) +holds the same bytes. In sheet "3.8 Current receipts (on a cash basis)", row 13 +"Value added tax" gives, in £ billion, D13 = 170.994 (2024-25 outturn), +E13 = 180.16893868541794 (2025-26) and F13 = 187.70436725810845 (2026-27). The +sheet's note in row 66 reads "All tax lines prior to accruals adjustments on +tax line are on a cash basis". + +### Method + +The coverage follows the energy-split method (see +[Energy bills are not grossed up](#energy-bills-are-not-grossed-up)): + +```text +pre-scaling non-energy VAT = 0.20 x sum(full_rate_vat_consumption) + + 0.05 x sum(reduced_rate_vat_consumption) +energy-bill VAT = sum(domestic_energy_vat) +survey coverage = pre-scaling non-energy VAT + / (0.7 x (OBR VAT receipts - energy-bill VAT)) +``` + +Every sum weights households by `household_weight`. + +A run of the whole cut in one process needed more than the 8 GiB of memory +available, so the runs split it into ten household chunks. Chunk k holds every +tenth household from the k-th, by sorted `household_id`, with its persons and +benefit units. Each chunk runs `Microsimulation` on +`extend_single_year_dataset(chunk, parameters, end_year=2026)`. Each +household's VAT depends only on its own inputs: `full_rate_vat_consumption` +multiplies the household's `consumption` by its own +`full_rate_vat_expenditure_rate`, the uprating multiplies each household's +inputs by growth factors, and `vat` divides each household's statutory +liability by one scalar factor. So weighted VAT totals over the ten chunks sum +exactly to the whole cut's. As a check, for each VAT input and `vat` in +2024-25 to 2026-27, chunk 0 matched a separate run of the same households to +the last digit, and chunk 0's two halves summed to it within 2.7e-16 of the +total. + +### Results + +At the coverage of 0.67 that `36bbbf42f` ships: + +| Fiscal year | Standard-rated base | Reduced-rated base | Pre-scaling non-energy VAT | Energy-bill VAT | OBR VAT receipts | Coverage | Total `vat` at 0.67 | Against receipts | +|---|---|---|---|---|---|---|---|---| +| 2024-25 | £407.40bn | £0.00bn | £81.48bn | £2.33bn | £170.994bn (outturn) | 0.690 | £176.06bn | +3.0% | +| 2025-26 | £422.51bn | £0.00bn | £84.50bn | £2.42bn | £180.169bn | 0.679 | £181.09bn | +0.5% | +| 2026-27 | £433.75bn | £0.00bn | £86.75bn | £1.75bn | £187.704bn | 0.666 | £185.18bn | -1.3% | + +Unrounded (£): standard-rated base 407,399,379,048, 422,514,711,572 and +433,745,364,638; energy-bill VAT 2,328,998,935, 2,415,410,320 and +1,749,888,519; pre-scaling non-energy VAT 81,479,875,810, 84,502,942,314 and +86,749,072,928; total `vat` at 0.67 176,060,075,948, 181,087,241,846 and +185,178,679,581. Coverage: 0.690124, 0.679134 and 0.666436. The run read +private school fees' share of receipts as 0, 0.008353 and 0.008311. + +### Value + +The parameter takes 0.69 (product 0.483), the 2024-25 value: as in the +[`f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026), the +latest outturn year sets it. Receipts for later years come from OBR forecasts, +so the factor stays undated. The coverage falls 3.4% from 2024-25 to 2026-27. + +### Totals at 0.67 and 0.69 + +| Dataset | Fiscal year | Total `vat` at 0.67 | Against receipts | Total `vat` at 0.69 | Against receipts | +|---|---|---|---|---|---| +| Microcosm UK 2024-25 (`a6299504`) | 2024-25 | £176.06bn | +3.0% | £171.02bn | +0.02% | +| Microcosm UK 2024-25 (`a6299504`) | 2025-26 | £181.09bn | +0.5% | £175.91bn | -2.4% | +| Microcosm UK 2024-25 (`a6299504`) | 2026-27 | £185.18bn | -1.3% | £179.86bn | -4.2% | +| Enhanced FRS 2024-25, uk-data 1.58.1 | 2024-25 | £242.54bn | +41.8% | £235.58bn | +37.8% | +| Enhanced FRS 2024-25, uk-data 1.58.1 | 2025-26 | £249.37bn | +38.4% | £242.22bn | +34.4% | +| Enhanced FRS 2024-25, uk-data 1.58.1 | 2026-27 | £255.28bn | +36.0% | £247.93bn | +32.1% | + +Unrounded Microcosm totals at 0.69 (£): 171,024,393,280, 175,908,348,963 and +179,861,903,668. The Enhanced FRS run used the whole file in one process. On +that file the same calculation gives a coverage of 0.954, 0.939 and 0.921, from +a standard-rated base of £563.06bn, £583.95bn and £599.47bn, a reduced-rated +base of £0.00bn, and energy-bill VAT of £2.43bn, £2.43bn and £1.76bn. + +Total `vat` leaves out `private_school_vat`, which the receipts include: about +£1.5bn a year from 2025-26 on HMRC's costing (see +[Private school VAT](#private-school-vat-splitting-the-existing-receipts-total)). +Model year 2024 has no school charge. + +### Bound + +On `a6299504` two routes bound the coverage, at a household share of 0.7: + +- receipts across 2024-25 to 2026-27 give 0.666 to 0.690; +- the ready reckoner, which does not use the level of receipts: one point on + the standard rate raises £4.34bn before scaling in 2026-27 (1% of + £433.75bn), so matching HMRC's £8.8bn needs a product of 0.493 (coverage + 0.704). + +So the coverage lies between 0.666 and 0.704, and 0.69 sits inside that range. +At 0.69, a run that sets the standard rate to 21% for 2026-27 alone raises +total `vat` in that year from £179.862bn to £188.767bn, by £8.906bn, exceeding +the ready reckoner by 1.2%. That rise matches 1% of the 2026-27 standard-rated +base, times one minus the school share, over the product: +£4,337,453,646 x (1 - 0.008311) / 0.483 = £8,905,599,522, within £7,934 of the +run's £8,905,591,588. + +### Reproducing + +The datasets sit in private Hugging Face repositories. Anyone with +PolicyEngine access can rerun the calculation; anyone else can check the +arithmetic from the published aggregates above. + +```python +from huggingface_hub import hf_hub_download +from policyengine_uk import Microsimulation + +path = hf_hub_download( + "policyengine/populace-uk-private", + "microcosm_uk_2024_25.h5", + repo_type="dataset", + revision="994cac89d902af9a985af7c1182d74717a582ce0", +) +sim = Microsimulation(dataset=path) +# OBR March 2026 EFO, table 3.8, cash-basis VAT receipts +receipts = {2024: 170.994e9, 2025: 180.169e9, 2026: 187.704e9} +for year, obr in receipts.items(): + p = sim.tax_benefit_system.parameters(f"{year}-01-01").gov.hmrc.vat + full = sim.calculate("full_rate_vat_consumption", year).sum() + reduced = sim.calculate("reduced_rate_vat_consumption", year).sum() + energy = sim.calculate("domestic_energy_vat", year).sum() + raw = full * p.standard_rate + reduced * p.reduced_rate + print(year, raw / 1e9, energy / 1e9, raw / (0.7 * (obr - energy))) +``` + +A run of the whole cut in one process needed more than 8 GiB of memory, so the +runs above summed ten household chunks (see [Method](#method)). + +For Enhanced FRS, download `enhanced_frs_2024_25.h5` from the model repository +`policyengine/policyengine-uk-data-private` at policyengine-uk-data 1.58.1 +(`revision="2526edbbb687a3b54dbb11272e5389e4b62d2ac2"`). For an earlier cut, +set `revision` to the one its inputs table gives. + +## Earlier derivation on `f9d1922c` (October 2026) -## Derivation (October 2026) +This section records the October 2026 derivation on `f9d1922c`, which set the +coverage at 0.67 before the energy split. The +[`a6299504` derivation](#derivation-on-a6299504-november-2026) replaces it. ### Inputs @@ -87,9 +268,11 @@ equal to it: combined receipts are `(1 - s) * sum(w * G) + sum(w * E) + sum(w * P0)` (see the receipt-partition derivation below), so they differ from `G + E` by `sum(w * P0) - s * sum(w * G)`. On the certified Enhanced FRS -(policyengine-uk-data 1.58.1) that difference is -£5.5m to -£142.5m a year from -2025-26 to 2031-32, at most 0.06% of VAT (Aggregate impact run -[40116746457](https://github.com/PolicyEngine/policyengine-uk/actions/runs/40116746457)). +(policyengine-uk-data 1.58.1) at 0.67 that difference came to -£5.5m to +-£142.5m a year from 2025-26 to 2031-32, at most 0.06% of VAT (Aggregate impact +run +[40116746457](https://github.com/PolicyEngine/policyengine-uk/actions/runs/40116746457), +24 October 2026). ### Results @@ -105,14 +288,14 @@ derivation below), so they differ from `G + E` by | Enhanced FRS 2024-25, uk-data 1.58.0 | 2025-26 | £593.92bn | £34.54bn | £120.51bn | £180.2bn | 0.669 | 0.956 | | Enhanced FRS 2024-25, uk-data 1.58.0 | 2026-27 | £609.89bn | £35.47bn | £123.75bn | £187.7bn | 0.659 | 0.942 | -The parameter takes the Microcosm 2024-25 value, 0.67 (product 0.469), the +The parameter took the Microcosm 2024-25 value, 0.67 (product 0.469), the latest outturn year. With that product, the gross amount against receipts was as below, before energy bills were split out. On the certified Enhanced FRS -(policyengine-uk-data 1.58.1), current-law `vat + private_school_vat` is -within 0.06% of the receipt-calibrated `G + E` (see above); the other rows -weren't checked against that bound: +(policyengine-uk-data 1.58.1) at 0.67, current-law `vat + private_school_vat` +came within 0.06% of the receipt-calibrated `G + E` (see above); no run +checked the other rows against that bound: | Dataset | 2024-25 | 2025-26 | 2026-27 | |---|---|---|---| @@ -124,51 +307,30 @@ weren't checked against that bound: Total VAT depends only on the product, so the household share being a rounded "around 70%" changes how the product is split between the two factors, -not the totals. On Microcosm the product is bounded by: +not the totals. On `f9d1922c` two routes bounded the product: - the receipts route across 2024-25 to 2026-27: 0.459 to 0.471 (coverage 0.655 to 0.672); - the ready reckoner, which does not use the level of receipts: one point on - the standard rate raises £4.24bn before scaling in 2026-27 (1% of - £423.66bn), so matching HMRC's £8.8bn needs a product of 0.481 (coverage + the standard rate raised £4.24bn before scaling in 2026-27 (1% of + £423.66bn), so matching HMRC's £8.8bn needed a product of 0.481 (coverage 0.688). -So on Microcosm the product lies between 0.459 and 0.481, and the coverage -between 0.655 and 0.688 at a household share of 0.7. The value used, 0.469 -(0.67), sits inside that range, and the totals it gives are within 2.2% of -receipts from 2024-25 to 2026-27. The implied product moves by less than 3% -over those years, so the factors are not dated by year. +So on `f9d1922c` the product lay between 0.459 and 0.481, and the coverage +between 0.655 and 0.688 at a household share of 0.7. The value then used, +0.469 (0.67), sat inside that range, and the totals it gave came within 2.2% of +receipts from 2024-25 to 2026-27. The implied product moved by less than 3% +over those years. ### Reproducing -The two datasets are in private Hugging Face repositories. Anyone with -PolicyEngine access can rerun the calculation; anyone else can check the -arithmetic from the published aggregates in the results table. - -```python -from huggingface_hub import hf_hub_download -from policyengine_uk import Microsimulation - -path = hf_hub_download( - "policyengine/populace-uk-private", - "microcosm_uk_2024_25.h5", - repo_type="dataset", - revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88", -) -sim = Microsimulation(dataset=path) -# OBR March 2026 EFO, table 3.8, cash-basis VAT receipts -receipts = {2024: 170.994e9, 2025: 180.169e9, 2026: 187.704e9} -for year, obr in receipts.items(): - p = sim.tax_benefit_system.parameters(f"{year}-01-01").gov.hmrc.vat - full = sim.calculate("full_rate_vat_consumption", year).sum() - reduced = sim.calculate("reduced_rate_vat_consumption", year).sum() - raw = full * p.standard_rate + reduced * p.reduced_rate - print(year, raw / 1e9, raw / obr, raw / obr / 0.7) -``` - -For Enhanced FRS, download `enhanced_frs_2024_25.h5` from the model repository -`policyengine/policyengine-uk-data-private` at the release tag (for example -`revision="1.58.0"`). +Run the snippet in +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026) with +`revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88"`. The pre-split rows +above came from policyengine-uk as of #2187, which divided pre-scaling VAT, +energy bills included, by the receipts themselves (`raw / obr / 0.7`). For +Enhanced FRS 1.56.16 or 1.58.0, download `enhanced_frs_2024_25.h5` at that +release tag (for example `revision="1.58.0"`). ## Energy bills are not grossed up @@ -201,7 +363,8 @@ actually pay on it. With energy split out, the coverage is derived against receipts net of energy VAT: coverage = pre-scaling non-energy VAT / (0.7 x (receipts - energy VAT)). -On the same inputs as the derivation above: +On the same inputs as the +[`f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026): | Dataset | Year | Pre-scaling non-energy VAT | Energy-bill VAT | OBR VAT receipts | Survey coverage | |---|---|---|---|---|---| @@ -213,15 +376,19 @@ On the same inputs as the derivation above: The pre-scaling non-energy VAT is the standard-rated part alone, since the generic reduced-rate base is empty once domestic fuel and power is taken out. -The Microcosm 2024-25 value rounds to the same 0.67 as the derivation above, -so the parameter is unchanged, and the bound above still holds (0.650 to 0.670 -by receipts, 0.688 by the ready reckoner). With it, total `vat` on Microcosm -is within 0.1% of 2024-25 receipts and 3.0% below in 2026-27; on Enhanced FRS -it is 38% (1.56.16) or 44% (1.58.0) above 2024-25 receipts. - -To reproduce, run the snippet above on this version and subtract -`sim.calculate("domestic_energy_vat", year).sum()` from the receipts before -dividing. +On `f9d1922c` the 2024-25 value rounded to the same 0.67 as the pre-split +derivation, so the energy split left the parameter at 0.67, and the `f9d1922c` +bound still held (0.650 to 0.670 by receipts, 0.688 by the ready reckoner). +With 0.67, total `vat` on `f9d1922c` came within 0.1% of 2024-25 receipts and +fell 3.0% below them in 2026-27; on Enhanced FRS it exceeded 2024-25 receipts +by 38% (1.56.16) or 44% (1.58.0). The +[`a6299504` derivation](#derivation-on-a6299504-november-2026) sets the +current value. + +To reproduce these rows, run the snippet in +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026), which +subtracts `sim.calculate("domestic_energy_vat", year).sum()` from the receipts +before dividing, with each dataset's revision. ## Private school VAT: splitting the existing receipts total @@ -231,8 +398,8 @@ The OBR VAT total already includes VAT on private school fees. HMRC's £1,505m in 2025-26, £1,560m in 2026-27, £1,610m in 2027-28, £1,665m in 2028-29 and £1,725m in 2029-30. Adding a separate school VAT head to the gross amount above would count that receipt twice. The model instead takes -the fees' share of receipts out of the gross amount, keeping the survey -coverage at 0.67. +the fees' share of receipts out of the gross amount, which leaves the survey +coverage unchanged. For each household, let `G` be the grossed-up VAT on recorded consumption other than energy bills (the statutory liability divided by the two factors), @@ -312,24 +479,29 @@ that coincidence likelier. A household situation would also measure its own school charge against its own VAT. The official share keeps each household's VAT a function of its own consumption. -Measured on the certified Enhanced FRS (Aggregate impact run -[40116746457](https://github.com/PolicyEngine/policyengine-uk/actions/runs/40116746457)), -the official share changes current-law combined receipts by -£0.14bn in -2025-26, -£0.13bn in 2026-27, -£0.10bn in 2027-28, -£0.08bn in 2028-29 and --£0.01bn in 2029-30, at most 0.06% of that dataset's VAT. Two overstatements +Measured on the certified Enhanced FRS at a coverage of 0.67 (Aggregate impact +run +[40116746457](https://github.com/PolicyEngine/policyengine-uk/actions/runs/40116746457), +24 October 2026), the official share changes current-law combined receipts by +-£0.14bn in 2025-26, -£0.13bn in 2026-27, -£0.10bn in 2027-28, -£0.08bn in +2028-29 and -£0.01bn in 2029-30, at most 0.06% of that dataset's VAT. Two overstatements nearly offset there: VAT is 40% above OBR receipts (£252.5bn against £180.2bn in 2025-26), and the modelled school charge is £1.95bn, against HMRC's £1.505bn, because attendance is imputed for 751,000 weighted pupils (policyengine-uk-data issue 568). The difference belongs to those two calibrations, not to the split. -The receipts check for the coverage factor is unchanged. Model year 2024 has -no school charge and a zero share, so, using the rounded 2024-25 Microcosm -inputs in the energy table above, `79.23 / (0.67 * 0.7) + 1.99 = £170.924bn`, -compared with OBR's £170.994bn: about 0.04% below receipts. The existing -differences from receipts, including Enhanced FRS's larger overstatement, still require -dataset-specific calibration. Aggregate impact CI compares the combined VAT -report group against the PR's base. +The split leaves the receipts check for the coverage factor unchanged. Model +year 2024 has no school charge and a zero share, so, using the rounded 2024-25 +inputs for `a6299504`, `81.48 / (0.69 * 0.7) + 2.33 = £171.026bn`, against +OBR's £170.994bn: 0.02% above receipts. The run at 0.69 gives £171.02bn on the +unrounded inputs (see +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026)). With 0.67, +the rounded `f9d1922c` inputs gave `79.23 / (0.67 * 0.7) + 1.99 = £170.924bn`, +about 0.04% below receipts. The existing differences from receipts, including +Enhanced FRS's larger overstatement, still require dataset-specific +calibration. Aggregate impact CI compares the combined VAT report group +against the PR's base. Model year 2024 represents 2024-25 but reads annual policy before the January 2025 commencement, so `P0` and the share are both zero for that @@ -339,8 +511,9 @@ VAT; this change does not model its separate timing. ## Later Microcosm cuts (24 October 2026) These runs predate the energy split above, so they use the formula in -[Derivation](#derivation-october-2026). They record why the coverage has to be -re-derived (PolicyEngine/policyengine-uk#2633). They do not set the parameter. +[the `f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026). +They record why the coverage has to be re-derived +(PolicyEngine/policyengine-uk#2633). They do not set the parameter. | Input | Value | |---|---| From 31306e57c94ff7c3c0800ef8ea2d6772fcfa8544 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 09:51:37 +0000 Subject: [PATCH 08/11] Put the VAT property docstring and added fragment in the active voice Use the proposed wording for the property module's docstring, take "times one minus" for the school share in the added fragment, and drop the fragment's claims that the cuts were certified since f9d1922c and that the value waits for re-derivation. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WR8LHapmX36bpbh3SMixXg --- changelog.d/2633.added.md | 2 +- policyengine_uk/tests/test_vat_grossing_properties.py | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/changelog.d/2633.added.md b/changelog.d/2633.added.md index 1ada80c7d3..abeb087d0d 100644 --- a/changelog.d/2633.added.md +++ b/changelog.d/2633.added.md @@ -1 +1 @@ -Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, less private school fees' share of VAT receipts, on a vector of households), a test that 2024 is grossed by the same product as later years, and a record of how the shipped coverage performs on the Microcosm UK cuts certified since `f9d1922c`. The coverage value is unchanged pending re-derivation on the next certified cut (#2633). +Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, times one minus private school fees' share of VAT receipts, on a vector of households), a test that the model grosses 2024 by the same product as later years, and a record of how the coverage of 0.67 performed on the later Microcosm UK cuts `6cba5d71` and `a133f37a` (#2633). diff --git a/policyengine_uk/tests/test_vat_grossing_properties.py b/policyengine_uk/tests/test_vat_grossing_properties.py index 654d5bdfcb..4a9dfa5f13 100644 --- a/policyengine_uk/tests/test_vat_grossing_properties.py +++ b/policyengine_uk/tests/test_vat_grossing_properties.py @@ -2,9 +2,10 @@ `vat` is the statutory liability on recorded consumption divided by one grossing factor, times one minus private school fees' share of VAT receipts, -plus VAT on energy bills, which is not grossed up. The factor is the survey -coverage times the household share of receipts, unless a reform moves the -deprecated combined parameter, in which case that value is used. +plus VAT on energy bills, which the formula adds without grossing up. The +factor is the survey coverage times the household share of receipts, unless a +reform moves the deprecated combined parameter, in which case the formula uses +that value. """ from types import SimpleNamespace From 303e6585224cd6aa79e65e83059223681c1ec7e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 10:37:06 +0000 Subject: [PATCH 09/11] Correct the VAT doc's arithmetic check and tighten its wording The one-point check 4,337,453,646 x (1 - 0.008311) / 0.483 comes to 8,905,600,556, within 8,968 of the run, and the six-decimal 2026-27 coverage did not follow from the published inputs, so the doc keeps the three-decimal values. Date the 0.38 factor's and run 40116746457's figures, qualify the ready reckoner's 0.704 as leaving out the school share, say that baseline_vat keeps the two factors under a reform to the deprecated one, and attribute the pre-split division by receipts to the calculation rather than the model. Say that 0.69 matches 2024-25 receipts, show the 2025 YAML case's full computation, and name the cuts the added fragment's record covers. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WR8LHapmX36bpbh3SMixXg --- changelog.d/2633.added.md | 2 +- docs/book/assumptions/mp30k-uk-switch-plan.md | 12 +-- docs/engineering/vat-coverage.md | 87 ++++++++++--------- .../vat/survey_consumption_coverage.yaml | 16 ++-- .../baseline/gov/hmrc/vat/vat_grossing.yaml | 5 +- 5 files changed, 64 insertions(+), 58 deletions(-) diff --git a/changelog.d/2633.added.md b/changelog.d/2633.added.md index abeb087d0d..0f32cdb04c 100644 --- a/changelog.d/2633.added.md +++ b/changelog.d/2633.added.md @@ -1 +1 @@ -Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, times one minus private school fees' share of VAT receipts, on a vector of households), a test that the model grosses 2024 by the same product as later years, and a record of how the coverage of 0.67 performed on the later Microcosm UK cuts `6cba5d71` and `a133f37a` (#2633). +Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, times one minus private school fees' share of VAT receipts, on a vector of households), a test that the model grosses 2024 by the same product as later years, and a record of how the coverage of 0.67 performed on the Microcosm UK cuts `6cba5d71` and `a133f37a`, which followed `f9d1922c` (#2633). diff --git a/docs/book/assumptions/mp30k-uk-switch-plan.md b/docs/book/assumptions/mp30k-uk-switch-plan.md index 90a16f7cb7..bec4a26a15 100644 --- a/docs/book/assumptions/mp30k-uk-switch-plan.md +++ b/docs/book/assumptions/mp30k-uk-switch-plan.md @@ -115,13 +115,13 @@ A documented gate must pass before any environment's default changes: 1. Dataset choice stays a runtime setting: no code in this repo branches on which dataset is loaded. One parameter is tied to the dataset, though: - `gov.simulation.vat.survey_consumption_coverage` matches receipts on one - Microcosm UK 2024-25 national cut only + `gov.simulation.vat.survey_consumption_coverage` matches 2024-25 receipts + on one Microcosm UK 2024-25 national cut only (`microcosm-uk-2024-25-national-20261025T091414Z-a6299504`, which - policyengine.py 6.11.1 and 6.12.0 certify), puts 2024-25 VAT 38% above - receipts on the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1), and has - to be re-derived when the dataset it should match changes - (`docs/engineering/vat-coverage.md`). + policyengine.py 6.11.1 and 6.12.0 certify), and puts 2024-25 VAT 38% above + receipts on the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1). + Re-derive it, with `docs/engineering/vat-coverage.md`, whenever the dataset + it should match changes. 2. While both are in use, CI runs the dataset-dependent tests on both. 3. The default is set per environment: for direct use of this package through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index 3781d6cdfc..366c2542ad 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -26,24 +26,26 @@ to households in proportion to their own VAT-liable spending. These two factors replace `gov.simulation.microdata_vat_coverage`, a single undated 0.38 added in January 2023 against an older LCFS-based consumption -imputation, with no derivation recorded. Its product with today's datasets put -2024-25 VAT 24% above OBR receipts on Microcosm UK 2024-25 and 71% above on -Enhanced FRS 2024-25 (policyengine-uk-data 1.56.16). +imputation, with no derivation recorded. Its product with the October 2026 +datasets put 2024-25 VAT 24% above OBR receipts on Microcosm UK 2024-25 +(`f9d1922c`) and 71% above on Enhanced FRS 2024-25 (policyengine-uk-data +1.56.16). `gov.simulation.microdata_vat_coverage` is kept as a deprecated parameter so that existing reforms to it still work. Its value is now the product of the two -factors (0.483). If a reform sets it to any other value, `vat` and -`baseline_vat` use that value in place of the two factors -(`policyengine_uk/utils/vat.py`); otherwise they use the two factors. New -reforms should change the two factors instead. +factors (0.483). If a reform sets it to any other value, `vat` uses that value +in place of the two factors (`policyengine_uk/utils/vat.py`), and +`baseline_vat`, which reads the unreformed parameters, keeps the two factors. +New reforms should change the two factors instead. ## Which datasets the factor applies to -The survey coverage of 0.69 matches receipts on one dataset: the certified -Microcosm UK 2024-25 national release `a6299504`. It does not carry over to -other datasets, because their consumption imputations capture a different -share of household VAT. On Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1) -it puts total `vat` 32% to 38% above receipts from 2024-25 to 2026-27 (see +The survey coverage of 0.69 matches 2024-25 receipts on one dataset: the +certified Microcosm UK 2024-25 national release `a6299504`. It does not carry +over to other datasets, because their consumption imputations capture a +different share of household VAT. On Enhanced FRS 2024-25 +(policyengine-uk-data 1.58.1) it puts total `vat` 38% above receipts in +2024-25, falling to 32% in 2026-27 (see [Derivation on `a6299504`](#derivation-on-a6299504-november-2026)). Re-derive the coverage, with the calculation below, whenever the dataset it should match changes. @@ -76,17 +78,18 @@ reform. | Build | `uk-frs-calibration-attempt-20261025T091414Z-a6299504`, at Microcosm code pin `67ee2ddf3`, with policyengine-uk 2.151.0 | PolicyEngine/microcosm | | Certification | policyengine.py commit `324ecad65` ("Certify the Microcosm UK national a6299504 with policyengine-uk 2.151.0", 26 October 2026, through policyengine.py#720); release 6.11.1 (27 October 2026) shipped it, and release 6.12.0 (2 November 2026) still certifies it | PolicyEngine/policyengine.py | | VAT receipts | £170.994bn (2024-25 outturn), £180.169bn (2025-26), £187.704bn (2026-27) | OBR, Economic and fiscal outlook March 2026, detailed forecast tables: receipts, sheet "3.8 Current receipts (on a cash basis)", row 13 "Value added tax", cells D13, E13 and F13 (see below) | -| Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face model repository `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.58.1 (revision `2526edbbb687a3b54dbb11272e5389e4b62d2ac2`, SHA-256 `056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784`, 128,409,762 bytes), the file `policyengine_uk/tests/certified_dataset.yaml` names | Private repository | +| Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face model repository `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.58.1 (revision `2526edbbb687a3b54dbb11272e5389e4b62d2ac2`, SHA-256 `056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784`, 128,409,762 bytes), the release that `policyengine_uk/tests/certified_dataset.yaml` names | Private repository | | Household share of VAT liabilities | 0.7 | `household_share_of_receipts` | | Ready reckoner | £8.8bn for one point on the standard rate, 2026-27 | HMRC, Direct effects of illustrative tax changes (June 2025) | | Statutory rates | 20% standard, 5% reduced | `gov.hmrc.vat` | Microcosm's fit for this cut holds `obr.vat` out under a measure exclusion -(PolicyEngine/microcosm#1278) that runs from 24 October 2026 to 24 January -2027, so the fitted weights don't target the receipts that this derivation -compares VAT with. The cut's calibration diagnostics carry one warn-only VAT -warning, `uk_consumption_drift`: the total `full_rate_vat_base` moves by more -than 2% of its design value between the design and the final weights. +(PolicyEngine/microcosm#1278) that a Microcosm maintainer approved on +24 October 2026 and that expires on 24 January 2027, so the fitted weights +don't target the receipts that this derivation compares VAT with. The cut's +calibration diagnostics carry one warn-only VAT warning, +`uk_consumption_drift`: the total `full_rate_vat_base` moves by more than 2% +of its design value between the design and the final weights. OBR publishes the receipts workbook at . @@ -102,8 +105,8 @@ tax line are on a cash basis". ### Method -The coverage follows the energy-split method (see -[Energy bills are not grossed up](#energy-bills-are-not-grossed-up)): +The coverage follows the energy-split method in +[the energy section](#energy-bills-are-not-grossed-up): ```text pre-scaling non-energy VAT = 0.20 x sum(full_rate_vat_consumption) @@ -144,8 +147,8 @@ Unrounded (£): standard-rated base 407,399,379,048, 422,514,711,572 and 433,745,364,638; energy-bill VAT 2,328,998,935, 2,415,410,320 and 1,749,888,519; pre-scaling non-energy VAT 81,479,875,810, 84,502,942,314 and 86,749,072,928; total `vat` at 0.67 176,060,075,948, 181,087,241,846 and -185,178,679,581. Coverage: 0.690124, 0.679134 and 0.666436. The run read -private school fees' share of receipts as 0, 0.008353 and 0.008311. +185,178,679,581. The run read private school fees' share of receipts as 0, +0.008353 and 0.008311. ### Value @@ -183,15 +186,15 @@ On `a6299504` two routes bound the coverage, at a household share of 0.7: - receipts across 2024-25 to 2026-27 give 0.666 to 0.690; - the ready reckoner, which does not use the level of receipts: one point on the standard rate raises £4.34bn before scaling in 2026-27 (1% of - £433.75bn), so matching HMRC's £8.8bn needs a product of 0.493 (coverage - 0.704). + £433.75bn), so, leaving out the school share, matching HMRC's £8.8bn needs + a product of 0.493 (coverage 0.704). So the coverage lies between 0.666 and 0.704, and 0.69 sits inside that range. At 0.69, a run that sets the standard rate to 21% for 2026-27 alone raises total `vat` in that year from £179.862bn to £188.767bn, by £8.906bn, exceeding the ready reckoner by 1.2%. That rise matches 1% of the 2026-27 standard-rated base, times one minus the school share, over the product: -£4,337,453,646 x (1 - 0.008311) / 0.483 = £8,905,599,522, within £7,934 of the +£4,337,453,646 x (1 - 0.008311) / 0.483 = £8,905,600,556, within £8,968 of the run's £8,905,591,588. ### Reproducing @@ -228,7 +231,7 @@ runs above summed ten household chunks (see [Method](#method)). For Enhanced FRS, download `enhanced_frs_2024_25.h5` from the model repository `policyengine/policyengine-uk-data-private` at policyengine-uk-data 1.58.1 (`revision="2526edbbb687a3b54dbb11272e5389e4b62d2ac2"`). For an earlier cut, -set `revision` to the one its inputs table gives. +set `revision` to the tag or full revision that its inputs table gives. ## Earlier derivation on `f9d1922c` (October 2026) @@ -326,11 +329,12 @@ over those years. Run the snippet in [Derivation on `a6299504`](#derivation-on-a6299504-november-2026) with -`revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88"`. The pre-split rows -above came from policyengine-uk as of #2187, which divided pre-scaling VAT, -energy bills included, by the receipts themselves (`raw / obr / 0.7`). For -Enhanced FRS 1.56.16 or 1.58.0, download `enhanced_frs_2024_25.h5` at that -release tag (for example `revision="1.58.0"`). +`revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88"`. For the pre-split rows +above, run it on policyengine-uk as of #2187, before energy bills left +`reduced_rate_vat_consumption`, and divide pre-scaling VAT by the receipts +themselves (`raw / obr / 0.7`). For Enhanced FRS 1.56.16 or 1.58.0, download +`enhanced_frs_2024_25.h5` at that release tag (for example +`revision="1.58.0"`). ## Energy bills are not grossed up @@ -386,9 +390,11 @@ by 38% (1.56.16) or 44% (1.58.0). The current value. To reproduce these rows, run the snippet in -[Derivation on `a6299504`](#derivation-on-a6299504-november-2026), which -subtracts `sim.calculate("domestic_energy_vat", year).sum()` from the receipts -before dividing, with each dataset's revision. +[Derivation on `a6299504`](#derivation-on-a6299504-november-2026) with each +dataset's revision, and for Enhanced FRS with the repository and file that +section gives. The snippet subtracts +`sim.calculate("domestic_energy_vat", year).sum()` from the receipts before +dividing. ## Private school VAT: splitting the existing receipts total @@ -484,10 +490,10 @@ run [40116746457](https://github.com/PolicyEngine/policyengine-uk/actions/runs/40116746457), 24 October 2026), the official share changes current-law combined receipts by -£0.14bn in 2025-26, -£0.13bn in 2026-27, -£0.10bn in 2027-28, -£0.08bn in -2028-29 and -£0.01bn in 2029-30, at most 0.06% of that dataset's VAT. Two overstatements -nearly offset there: VAT is 40% above OBR receipts (£252.5bn against -£180.2bn in 2025-26), and the modelled school charge is £1.95bn, against -HMRC's £1.505bn, because +2028-29 and -£0.01bn in 2029-30, at most 0.06% of that dataset's VAT. Two +overstatements nearly offset in that run: VAT exceeded OBR receipts by 40% +(£252.5bn against £180.2bn in 2025-26), and the modelled school charge came to +£1.95bn, against HMRC's £1.505bn, because attendance is imputed for 751,000 weighted pupils (policyengine-uk-data issue 568). The difference belongs to those two calibrations, not to the split. @@ -511,9 +517,8 @@ VAT; this change does not model its separate timing. ## Later Microcosm cuts (24 October 2026) These runs predate the energy split above, so they use the formula in -[the `f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026). -They record why the coverage has to be re-derived -(PolicyEngine/policyengine-uk#2633). They do not set the parameter. +[the `f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026). They record why the coverage has to be +re-derived (PolicyEngine/policyengine-uk#2633). They do not set the parameter. | Input | Value | |---|---| diff --git a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml index 46396cc79f..6eb57aaf5d 100644 --- a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml @@ -16,16 +16,16 @@ metadata: leaves the generic reduced-rate base empty), and VAT on energy bills comes to £2.33bn. OBR VAT receipts for 2024-25 (outturn, March 2026 EFO table 3.8, cash basis) come to £170.994bn. Coverage = 81.48 / (0.7 x (170.994 - - 2.33)) = 0.690. Across 2024-25 to 2026-27 the coverage runs from 0.666 to - 0.690, and the HMRC ready reckoner (£8.8bn for one point on the standard - rate in 2026-27) implies 0.704. On Enhanced FRS 2024-25 - (policyengine-uk-data 1.58.1) the same calculation gives 0.954 for - 2024-25, so this value puts total VAT on that file 38% above 2024-25 + 2.33)) = 0.690. From 2024-25 to 2026-27 the coverage falls from 0.690 to + 0.666, and the HMRC ready reckoner (£8.8bn for one point on the standard + rate in 2026-27) implies 0.704, leaving out the school share. On Enhanced + FRS 2024-25 (policyengine-uk-data 1.58.1) the same calculation gives 0.954 + for 2024-25, so this value puts total VAT on that file 38% above 2024-25 receipts. Re-derive when the dataset it should match changes, with docs/engineering/vat-coverage.md, which lists the inputs, file hashes and - per-dataset results. These receipts already include private school VAT. - The vat formula therefore removes school-fee VAT's share of receipts from - the grossed-up amount (not from energy-bill VAT) before gov_tax adds + per-dataset results. These receipts already include private school VAT. The + vat formula therefore removes school-fee VAT's share of receipts from the + grossed-up amount (not from energy-bill VAT) before gov_tax adds private_school_vat. At current law on Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1), the two heads sum to the receipt-calibrated VAT to within the share's accuracy (at most 0.06% below it), so no extra diff --git a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml index 2fc324c608..208fed08a1 100644 --- a/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml +++ b/policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml @@ -6,8 +6,9 @@ reduced_rate_vat_consumption: 1_000 output: # (10,000 x 20% + 1,000 x 5%) / (0.69 x 0.7) = 2,050 / 0.483 = 4,244.31, - # less private school fees' official share of 2025-26 VAT receipts - # (0.008353), which private_school_vat counts separately. + # times one minus private school fees' official share of 2025-26 VAT + # receipts (0.008353), which private_school_vat counts separately: + # 2,050 / 0.483 x (1 - 0.008353) = 4,208.85. vat: 4_208.85 baseline_vat: 4_208.85 From af0b602ca3170417bf306e162adcd130f94f60f6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 3 Nov 2026 07:05:54 -0500 Subject: [PATCH 10/11] Say why the cut's own year sets the VAT coverage, and date later rows main's October 2026 EFO gives 2025-26 outturn, so "the latest outturn year" no longer names 2024-25. The doc now gives the reason that holds under both EFOs, and says that the 2025-26 and 2026-27 rows come from policyengine-uk 2.157.1 on the March 2026 EFO. Co-Authored-By: Claude Opus 5.5 --- docs/engineering/vat-coverage.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/engineering/vat-coverage.md b/docs/engineering/vat-coverage.md index 8a63f02c15..b64edecc23 100644 --- a/docs/engineering/vat-coverage.md +++ b/docs/engineering/vat-coverage.md @@ -152,10 +152,18 @@ Unrounded (£): standard-rated base 407,399,379,048, 422,514,711,572 and ### Value -The parameter takes 0.69 (product 0.483), the 2024-25 value: as in the -[`f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026), the -latest outturn year sets it. Receipts for later years come from OBR forecasts, -so the factor stays undated. The coverage falls 3.4% from 2024-25 to 2026-27. +The parameter takes 0.69 (product 0.483). The 2024-25 value sets it, as in +the [`f9d1922c` derivation](#earlier-derivation-on-f9d1922c-october-2026): +2024-25 is the cut's own year, which `apply_uprating` leaves as the file gives +it, and OBR's figure for it is outturn. Later years add the model's uprating +and, beyond outturn, OBR's forecasts, so the factor stays undated. On the +March 2026 EFO's receipts the coverage falls 3.4% from 2024-25 to 2026-27. + +The runs here predate the October 2026 EFO that `main` now uses. That EFO +gives 2025-26 outturn of £179.861bn and forecasts £186.154bn for 2026-27, and +`main` uprates consumption with its October forecasts. The 2025-26 and 2026-27 +rows here record policyengine-uk 2.157.1, which used the March 2026 EFO for +both. ### Totals at 0.67 and 0.69 From ffe20fb9165ddcda48ac57531f87847ecc317b9e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 4 Nov 2026 19:53:14 -0500 Subject: [PATCH 11/11] Date the 0.06% school VAT bound and move the 2pp VAT fixture to 27.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The coverage description's 0.06% bound comes from the run at a coverage of 0.67 (Aggregate impact run 40116746457, 24 October 2026), so it now says so, as docs/engineering/vat-coverage.md does. The Aggregate impact report for this branch at af0b602ca, merged into main at 1d4d020aa (run 41633358040, policyengine-uk 2.172.1), costs the 2pp VAT rise at £27.07bn in 2029-30 on Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1), the change in gov_balance that test_reform_impacts.py measures. The fixture now expects 27.1 and keeps that costing in its receipts. Co-Authored-By: Claude Opus 5.5 --- .../simulation/vat/survey_consumption_coverage.yaml | 11 ++++++----- .../tests/microsimulation/reforms_config.yaml | 5 ++++- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml index 6eb57aaf5d..4ac312c888 100644 --- a/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml +++ b/policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml @@ -26,11 +26,12 @@ metadata: per-dataset results. These receipts already include private school VAT. The vat formula therefore removes school-fee VAT's share of receipts from the grossed-up amount (not from energy-bill VAT) before gov_tax adds - private_school_vat. At current law on Enhanced FRS 2024-25 - (policyengine-uk-data 1.58.1), the two heads sum to the receipt-calibrated - VAT to within the share's accuracy (at most 0.06% below it), so no extra - school receipts are added to this denominator. See the receipt-partition - derivation in that document. + private_school_vat. At a coverage of 0.67 on Enhanced FRS 2024-25 + (policyengine-uk-data 1.58.1), the two heads summed to the + receipt-calibrated VAT at current law to within the share's accuracy, at + most 0.06% below it (Aggregate impact run 40116746457, 24 October 2026), + so this denominator needs no extra school receipts. See the + receipt-partition derivation in that document. reference: - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8 href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ diff --git a/policyengine_uk/tests/microsimulation/reforms_config.yaml b/policyengine_uk/tests/microsimulation/reforms_config.yaml index 0d43f6b6fa..022450ec2b 100644 --- a/policyengine_uk/tests/microsimulation/reforms_config.yaml +++ b/policyengine_uk/tests/microsimulation/reforms_config.yaml @@ -159,7 +159,7 @@ reforms: data: 1.58.1 enhanced_frs_2024_25 model: 2.135.7 (merged 7717438bf, main 7ad002f54, 2026-10-18) - name: Raise VAT standard rate by 2pp - expected_impact: 27.6 + expected_impact: 27.1 tolerance: 1.5 parameters: gov.hmrc.vat.standard_rate: 0.22 @@ -194,6 +194,9 @@ reforms: - impact: 27.581 data: 1.58.1 enhanced_frs_2024_25 model: 2.135.7 (merged 7717438bf, main 7ad002f54, 2026-10-18) + - impact: 27.071 + data: 1.58.1 enhanced_frs_2024_25 + model: 2.172.1 (merged af0b602ca, main 1d4d020aa, 2026-11-03) - name: Raise additional rate by 3pp expected_impact: 3.7 parameters: