Move the VAT survey coverage to 0.69 on Microcosm a6299504

changelog.d/2633.added.md

@@ -0,0 +1 @@
+Added property tests for VAT grossing (the routing between the survey coverage and the deprecated combined factor, including the closeness band, and VAT as the statutory liability over the shipped product, times one minus private school fees' share of VAT receipts, on a vector of households), a test that the model grosses 2024 by the same product as later years, and a record of how the coverage of 0.67 performed on the Microcosm UK cuts `6cba5d71` and `a133f37a`, which followed `f9d1922c` (#2633).

changelog.d/2633.changed.md

@@ -0,0 +1 @@
+Re-derived the survey VAT consumption coverage (`gov.simulation.vat.survey_consumption_coverage`) on the certified Microcosm UK 2024-25 national cut `a6299504` and moved it from 0.67 to 0.69: on that cut, 2024-25 total `vat` comes to £176.06bn at 0.67, 3.0% above OBR receipts of £170.994bn, and £171.02bn at 0.69, 0.02% above. Moved the deprecated combined factor `gov.simulation.microdata_vat_coverage` to the new product, 0.483.

docs/book/assumptions/mp30k-uk-switch-plan.md

@@ -115,14 +115,13 @@ A documented gate must pass before any environment's default changes:
 
 1. Dataset choice stays a runtime setting: no code in this repo branches on
    which dataset is loaded. One parameter is tied to the dataset, though:
-   `gov.simulation.vat.survey_consumption_coverage` is calibrated to one
-   Microcosm UK 2024-25 national cut only
-   (`microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68`; policyengine.py
-   6.10.0 pins the later cut
-   `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), puts 2024-25
-   VAT 38% (policyengine-uk-data 1.56.16) or 44% (1.58.0) above receipts on
-   the Enhanced FRS 2024-25, and has to be re-derived when the dataset it
-   should match changes (`docs/engineering/vat-coverage.md`).
+   `gov.simulation.vat.survey_consumption_coverage` matches 2024-25 receipts
+   on one Microcosm UK 2024-25 national cut only
+   (`microcosm-uk-2024-25-national-20261025T091414Z-a6299504`, which
+   policyengine.py 6.11.1 and 6.12.0 certify), and puts 2024-25 VAT 38% above
+   receipts on the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1).
+   Re-derive it, with `docs/engineering/vat-coverage.md`, whenever the dataset
+   it should match changes.
 2. While both are in use, CI runs the dataset-dependent tests on both.
 3. The default is set per environment: for direct use of this package
    through `POLICYENGINE_UK_DEFAULT_DATASET`; for this repository's tests

policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml

@@ -1,4 +1,4 @@
-description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.67 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead.
+description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.69 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead.
 metadata:
   unit: /1
   period: year
@@ -8,4 +8,4 @@ metadata:
     - title: docs/engineering/vat-coverage.md (derivation of the two factors this combines)
       href: https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/engineering/vat-coverage.md
 values:
-  2010-01-01: 0.469
+  2010-01-01: 0.483

policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml

@@ -4,27 +4,28 @@ metadata:
   period: year
   economy: false
   derivation: >-
-    Calibrated to the certified Microcosm UK 2024-25 national release only
-    (policyengine/populace-uk-private, tag
-    microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68, revision
-    f9d1922c). VAT on household energy bills (domestic_energy_vat) is not
-    grossed up, so the coverage applies only to VAT on other recorded
-    consumption. On that release, pre-scaling VAT on that consumption is 0.20 x
-    £396.15bn = £79.23bn in 2024 (the generic reduced-rate base is empty once
-    domestic fuel and power is taken out), and VAT on energy bills is £1.99bn.
-    OBR VAT receipts for 2024-25 (outturn, March 2026 EFO table 3.8, cash
-    basis) are £170.994bn. Coverage = 79.23 / (0.7 x (170.994 - 1.99)) = 0.670.
-    Before energy bills were split out, the same calculation gave 0.672, so
-    the rounded value is unchanged. Across 2024-25 to 2026-27 the coverage is
-    0.650 to 0.670, and the HMRC ready reckoner (£8.8bn for one point on the
-    standard rate in 2026-27) implies 0.688. On Enhanced FRS 2024-25 the same
-    calculation gives 0.927 (policyengine-uk-data 1.56.16) or 0.967 (1.58.0),
-    so this value puts total VAT on that dataset 38% or 44% above 2024-25
+    This value matches 2024-25 VAT receipts on one dataset only: the certified
+    Microcosm UK 2024-25 national release (policyengine/populace-uk-private,
+    tag microcosm-uk-2024-25-national-20261025T091414Z-a6299504, revision
+    994cac89), whose fit excludes the obr.vat receipts target. The vat formula
+    adds VAT on household energy bills (domestic_energy_vat) without grossing
+    it up, so the coverage applies only to VAT on other recorded consumption.
+    On that release, pre-scaling VAT on that consumption comes to 0.20 x
+    £407.40bn = £81.48bn in 2024 (reduced_rate_vat_consumption takes domestic
+    fuel and power's share of spending out of the reduced-rate share, which
+    leaves the generic reduced-rate base empty), and VAT on energy bills comes
+    to £2.33bn. OBR VAT receipts for 2024-25 (outturn, March 2026 EFO table
+    3.8, cash basis) come to £170.994bn. Coverage = 81.48 / (0.7 x (170.994 -
+    2.33)) = 0.690. From 2024-25 to 2026-27 the coverage falls from 0.690 to
+    0.666, and the HMRC ready reckoner (£8.8bn for one point on the standard
+    rate in 2026-27) implies 0.704, leaving out the school share. On Enhanced
+    FRS 2024-25 (policyengine-uk-data 1.58.1) the same calculation gives 0.954
+    for 2024-25, so this value puts total VAT on that file 38% above 2024-25
     receipts. Re-derive when the dataset it should match changes, with
     docs/engineering/vat-coverage.md, which lists the inputs, file hashes and
-    per-dataset results. These receipts already include private school VAT.
-    The vat formula therefore removes school-fee VAT's share of receipts from
-    the grossed-up amount (not from energy-bill VAT) before gov_tax adds
+    per-dataset results. These receipts already include private school VAT. The
+    vat formula therefore removes school-fee VAT's share of receipts from the
+    grossed-up amount (not from energy-bill VAT) before gov_tax adds
     private_school_vat. At current law on Enhanced FRS 2024-25
     (policyengine-uk-data 1.58.1), the two heads sum to the receipt-calibrated
     VAT to within the share's accuracy (at most 0.06% below it), so no extra
@@ -33,7 +34,9 @@ metadata:
   reference:
     - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8
       href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/
+    - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts (efo-march-2026-detailed-forecast-tables-receipts.xlsx), sheet 3.8, row 13 (Value added tax, cash basis), cell D13 for 2024-25 outturn (£bn)
+      href: https://obr.uk/docs/d055fbf02d5b3g6jq8l2/efo-march-2026-detailed-forecast-tables-receipts.xlsx
     - title: HMRC, Direct effects of illustrative tax changes bulletin (June 2025)
       href: https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes/direct-effects-of-illustrative-tax-changes-bulletin-january-2025
 values:
-  2010-01-01: 0.67
+  2010-01-01: 0.69

policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml

@@ -5,11 +5,24 @@
     full_rate_vat_consumption: 10_000
     reduced_rate_vat_consumption: 1_000
   output:
-    # (10,000 x 20% + 1,000 x 5%) / (0.67 x 0.7) = 2,050 / 0.469 = 4,371.00,
-    # less private school fees' official share of 2025-26 VAT receipts
-    # (0.008368), which private_school_vat counts separately.
-    vat: 4_334.43
-    baseline_vat: 4_334.43
+    # (10,000 x 20% + 1,000 x 5%) / (0.69 x 0.7) = 2,050 / 0.483 = 4,244.31,
+    # times one minus private school fees' official share of 2025-26 VAT
+    # receipts (0.008368), which private_school_vat counts separately:
+    # 2,050 / 0.483 x (1 - 0.008368) = 4,208.79.
+    vat: 4_208.79
+    baseline_vat: 4_208.79
+
+- name: VAT in 2024 is grossed up by the same product as later years
+  period: 2024
+  absolute_error_margin: 0.01
+  input:
+    full_rate_vat_consumption: 10_000
+    reduced_rate_vat_consumption: 1_000
+  output:
+    # The deprecated combined factor has one undated value, so 2024 takes the
+    # two factors' product too: 2,050 / 0.483
+    vat: 4_244.31
+    baseline_vat: 4_244.31
 
 - name: With both factors at one, VAT is the statutory liability on recorded consumption
   period: 2025

policyengine_uk/tests/test_private_school_vat_receipts.py

@@ -37,7 +37,7 @@
 LEGACY = "gov.simulation.microdata_vat_coverage"
 FEES = "gov.simulation.private_school_vat.private_school_fees"
 BASIS = "gov.simulation.private_school_vat.private_school_vat_basis"
-FACTOR = 0.67 * 0.7
+FACTOR = 0.69 * 0.7
 # HMRC's OBR-certified costing of VAT on private school fees (Autumn Budget
 # 2024 Table 5.1, £m) and OBR cash-basis VAT receipts (October 2026 EFO
 # detailed forecast tables, receipts, sheet 4.8, row 13, cells D13:H13, £m),
@@ -190,13 +190,13 @@ def test_vat_reads_neither_attendance_nor_fees():
 
 
 def test_published_microcosm_receipts_check_is_unchanged():
-    # Rounded aggregate inputs published in docs/engineering/vat-coverage.md,
-    # with energy bills split out: the standard-rated base and the energy-bill
-    # VAT, which is added after grossing. Model year 2024 has no school charge
-    # and no share to reallocate.
-    sim = simulation(year=2024, full=396.15e9, reduced=0, energy_vat=1.99e9)
+    # Rounded aggregate inputs that docs/engineering/vat-coverage.md publishes
+    # for Microcosm a6299504, with energy bills split out: the standard-rated
+    # base, and the energy-bill VAT, which vat adds after grossing. Model year
+    # 2024 has no school charge and no share to reallocate.
+    sim = simulation(year=2024, full=407.40e9, reduced=0, energy_vat=2.33e9)
     combined = amount(sim, "vat", 2024) + amount(sim, "private_school_vat", 2024)
-    expected = 0.2 * 396.15e9 / FACTOR + 1.99e9
+    expected = 0.2 * 407.40e9 / FACTOR + 2.33e9
     assert combined == pytest.approx(expected)
     assert combined / 170.994e9 == pytest.approx(1, rel=0.001)
 

policyengine_uk/tests/test_scenario_baseline_parameters.py

@@ -51,8 +51,8 @@ def _vat_change(sim):
 def test_every_reform_route_scores_vat_against_the_unreformed_baseline(kwargs):
     sim = Simulation(situation=SITUATION, **kwargs)
     # All spending here is standard-rated, so raising the rate from 20% to
-    # 22% raises VAT by a tenth, whatever the grossing factor (526.32 at the
-    # current factor).
+    # 22% raises VAT by a tenth, whatever the grossing factor (about 411 at
+    # the shipped factors).
     unreformed_vat = float(Simulation(situation=SITUATION).calculate("vat", 2026)[0])
     assert _vat_change(sim) == pytest.approx(0.1 * unreformed_vat, abs=0.01)
     assert _vat_change(sim) > 0

policyengine_uk/tests/test_vat_grossing_legacy.py

@@ -31,7 +31,7 @@ def test_legacy_default_is_the_product_of_the_two_factors():
             year
         ) * p.vat.household_share_of_receipts(year)
         assert p.microdata_vat_coverage(year) == pytest.approx(product)
-    assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.469)
+    assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.483)
 
 
 def test_reform_dict_to_legacy_path_changes_vat():

policyengine_uk/tests/test_vat_grossing_properties.py

@@ -0,0 +1,118 @@
+"""Properties of VAT grossing that hold for every input (#2633).
+
+`vat` is the statutory liability on recorded consumption divided by one
+grossing factor, times one minus private school fees' share of VAT receipts,
+plus VAT on energy bills, which the formula adds without grossing up. The
+factor is the survey coverage times the household share of receipts, unless a
+reform moves the deprecated combined parameter, in which case the formula uses
+that value.
+"""
+
+from types import SimpleNamespace
+
+import pytest
+from hypothesis import assume, example, given, settings
+from hypothesis import strategies as st
+
+from policyengine_uk import Simulation
+from policyengine_uk.system import system
+from policyengine_uk.utils.vat import LEGACY_COVERAGE_DEFAULT, vat_grossing_factor
+
+SHARE = st.floats(min_value=0.01, max_value=1, allow_nan=False)
+SPEND = st.floats(min_value=0, max_value=1e6, allow_nan=False)
+# A legacy value counts as untouched when it is within NumPy's default
+# closeness of the declared default: 1e-8 plus 1e-5 times the default.
+BAND = 1e-8 + 1e-5 * abs(LEGACY_COVERAGE_DEFAULT)
+
+
+def params(legacy, coverage, share):
+    return SimpleNamespace(
+        microdata_vat_coverage=legacy,
+        vat=SimpleNamespace(
+            survey_consumption_coverage=coverage,
+            household_share_of_receipts=share,
+        ),
+    )
+
+
+# The helper properties check algebra and routing, not speed, so they have no
+# deadline: a slow first example on a loaded host is not a failure.
+@settings(deadline=None)
+@given(coverage=SHARE, share=SHARE)
+def test_untouched_legacy_factor_uses_the_product_of_the_two(coverage, share):
+    factor = vat_grossing_factor(params(LEGACY_COVERAGE_DEFAULT, coverage, share))
+    assert factor == pytest.approx(coverage * share)
+
+
+@settings(deadline=None)
+@given(
+    offset=st.floats(min_value=-0.99, max_value=0.99),
+    coverage=SHARE,
+    share=SHARE,
+)
+def test_legacy_factor_within_the_band_counts_as_untouched(offset, coverage, share):
+    legacy = LEGACY_COVERAGE_DEFAULT + offset * BAND
+    factor = vat_grossing_factor(params(legacy, coverage, share))
+    assert factor == pytest.approx(coverage * share)
+
+
+@settings(deadline=None)
+@given(legacy=SHARE, coverage=SHARE, share=SHARE)
+def test_moved_legacy_factor_overrides_the_two(legacy, coverage, share):
+    assume(abs(legacy - LEGACY_COVERAGE_DEFAULT) > 1.01 * BAND)
+    factor = vat_grossing_factor(params(legacy, coverage, share))
+    assert factor == legacy
+
+
+@pytest.mark.parametrize(
+    "multiple, moved", [(-1.01, True), (-0.99, False), (0.99, False), (1.01, True)]
+)
+def test_legacy_factor_either_side_of_the_band(multiple, moved):
+    legacy = LEGACY_COVERAGE_DEFAULT + multiple * BAND
+    factor = vat_grossing_factor(params(legacy, 0.5, 0.7))
+    assert factor == (legacy if moved else pytest.approx(0.35))
+
+
+def household_vat(spend, year):
+    """`vat` for one household per (full-rate, reduced-rate) spend pair."""
+    situation = {
+        "people": {f"p{i}": {"age": {year: 40}} for i in range(len(spend))},
+        "benunits": {f"b{i}": {"members": [f"p{i}"]} for i in range(len(spend))},
+        "households": {
+            f"h{i}": {
+                "members": [f"p{i}"],
+                "full_rate_vat_consumption": {year: full},
+                "reduced_rate_vat_consumption": {year: reduced},
+            }
+            for i, (full, reduced) in enumerate(spend)
+        },
+    }
+    sim = Simulation(situation=situation)
+    # The households record no energy bills, so no energy-bill VAT is added.
+    assert (sim.calculate("domestic_energy_vat", year) == 0).all()
+    return sim.calculate("vat", year)
+
+
+def expected_vat(full, reduced, year):
+    """Statutory liability over the shipped product, times (1 - school share)."""
+    p = system.parameters(year).gov
+    product = (
+        p.simulation.vat.survey_consumption_coverage
+        * p.simulation.vat.household_share_of_receipts
+    )
+    liability = full * p.hmrc.vat.standard_rate + reduced * p.hmrc.vat.reduced_rate
+    return liability / product * (1 - p.simulation.vat.private_school_share_of_receipts)
+
+
+# Each spend pair is (full_rate_vat_consumption, reduced_rate_vat_consumption).
+@settings(max_examples=5, deadline=None)
+@given(spend=st.lists(st.tuples(SPEND, SPEND), min_size=1, max_size=20))
+@example(spend=[(0.0, 15_438.0), (10_000.0, 0.0)])
+def test_vat_is_statutory_liability_over_the_shipped_product(spend):
+    # One simulation of many households, so the property is checked on a
+    # vector: VAT is linear in each household's recorded consumption, with
+    # no cross-household term.
+    vat = household_vat(spend, 2025)
+    for value, (full, reduced) in zip(vat, spend):
+        expected = expected_vat(full, reduced, 2025)
+        assert value == pytest.approx(expected, rel=1e-5, abs=1e-2)