Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
5f26856
Add the Autumn Budget 2026 measure registry and stage HM Treasury's T…
MaxGhenis Oct 29, 2026
9a0189c
Check the AB2026 harvest contract with one function a lane can run on…
MaxGhenis Oct 29, 2026
d37c86f
State which harvest numbers are estimates and how a measure's parts a…
MaxGhenis Oct 29, 2026
7a76bb3
Test the harvest contract on broken rows, and list every proposed key…
MaxGhenis Oct 29, 2026
9dbb52d
Correct rule 3's line 2 example and name proposed keys by scope
MaxGhenis Oct 29, 2026
2ef7870
Land the reworked HM Treasury red book family (432 rows)
claude Nov 3, 2026
7693110
Land the IFS family with the addendum's rule 8 and 9 checks (106 to 8…
claude Nov 3, 2026
836ea50
Land the devolved governments, misc and Trussell families with the th…
claude Nov 3, 2026
d5fccb4
Land the TPA, CenTax and IPPR families with the thinktanks-a review a…
claude Nov 3, 2026
30bc714
Land the HM Treasury policy costings family with its review applied
claude Nov 3, 2026
2bbdd3f
Land the Resolution Foundation family with its review applied (767 to…
claude Nov 3, 2026
0d09200
Land the OBR family reworked under its review (2,891 to 2,797 rows)
claude Nov 3, 2026
5bd845b
Give each proposed scope one key across families, and record the fami…
claude Nov 3, 2026
d5e323d
Land the IFS addendum's sections 1 and 2 (87 to 144 rows)
MaxGhenis Nov 3, 2026
f4c7c94
Answer the IFS review of d5e323dc: findings 1, 2 and 4
MaxGhenis Nov 3, 2026
8b7b23d
Answer finding 3 of the IFS review of d5e323dc: README rule 10
MaxGhenis Nov 3, 2026
dfebe9c
Answer the delta review of 8b7b23db: rule 10 by PSNB scope
MaxGhenis Nov 4, 2026
4f424b9
Correct who keys the OBR's Table 1.1 lines in a test's docstring
MaxGhenis Nov 4, 2026
6a1c06f
uk_hmt_tables: apply review A's T-1 to T-9
claude Nov 4, 2026
98e4840
registry: apply review A's R1 to R9 and regenerate
claude Nov 4, 2026
cb3d41a
uk_hmt_redbook: apply review A's RB-1 to RB-21 and K-7(b)
claude Nov 4, 2026
f0230bf
uk_rf_ab2026: apply review B's RF findings, K-4, K-6 and K-7
claude Nov 4, 2026
ab2cf63
uk_tpa: apply review B's TPA 1 to 6 and K-5
claude Nov 4, 2026
1e16211
uk_hmt_costings: apply review A's C-1 to C-7 and T-3's NOTES
claude Nov 4, 2026
0ff32ab
uk_centax: apply review B's CenTax 1 to 6
claude Nov 4, 2026
5c3fc6d
uk_ippr: apply review B's IPPR 1 and 2 and K-7(f)
claude Nov 4, 2026
c6c19c4
uk_devolved_govts: apply review B's devolved 1 and 2
claude Nov 4, 2026
a7fe8fb
uk_thinktank_misc, uk_trussell_wpi: apply review B's misc 1-3, Trusse…
claude Nov 4, 2026
451c303
uk_ifs_ab2026: apply review A's K-7(a), K-7(e) and H-6(a)
claude Nov 4, 2026
80b23f4
harness: apply review A's H-1 to H-4, H-6 and H-7
claude Nov 4, 2026
6d1b12a
uk_obr_tables: apply review B's OBR 1 to 16 and review A's K-1, K-2, K-5
claude Nov 4, 2026
3f6b078
tests: pin the harvest's reviewed sign readings (review A H-5)
claude Nov 4, 2026
1ecc1e5
harvest: regenerate PROPOSED_KEYS.md and COUNTS.json; README rules an…
claude Nov 4, 2026
bbf4f4b
signs.py: read a conflict only where one stretch of text reads both ways
claude Nov 4, 2026
15c0773
tests: reach the contract's manifest branches (review A H-7)
claude Nov 4, 2026
93d2407
uk_hmt_redbook: finish RB-18 and RB-19 from the pinned primaries
MaxGhenis Nov 4, 2026
3b1abda
uk_hmt_tables NOTES: quote Table 6.4's footnote 2 (review A T-6)
MaxGhenis Nov 4, 2026
6e06907
tests: pin each receipt's policyengine version (review A)
MaxGhenis Nov 4, 2026
ac4d14a
Apply delta reviews D2 and D3: costings row 584 and red book row refe…
MaxGhenis Nov 4, 2026
f27b927
harness: apply delta review D1's checker and test findings
MaxGhenis Nov 4, 2026
fe58796
Answer the final review of f27b927c: ratios, tolerance, malformed inp…
MaxGhenis Nov 5, 2026
1bf4507
Close rule 4 and manifest gaps found before the delta review
MaxGhenis Nov 5, 2026
72cdfdc
Split the README's 44-word rule 4 sentence
MaxGhenis Nov 5, 2026
615d6a8
Read more ratio forms, worded fractions and number words
MaxGhenis Nov 5, 2026
92f364c
Read each ratio number whole and check every field's type
MaxGhenis Nov 5, 2026
a9f81e8
Read value_raw whole or report it, in answer to the 92f364c7 reviews
MaxGhenis Nov 5, 2026
9fd73b1
Tighten the checker's scales, ranges, years and rule 4, in answer to …
MaxGhenis Nov 5, 2026
3c65fd7
Cut one example from the README's ratio form
MaxGhenis Nov 5, 2026
8206882
Read each factor whole, close the nil note and widen rule 4
MaxGhenis Nov 5, 2026
4707e91
Report unread operations and look past punctuation in rule 4
MaxGhenis Nov 5, 2026
2f60cdb
Report more unread operations and read nested wrappers in rule 4
MaxGhenis Nov 5, 2026
05265c2
Report fractions as operands and more spellings in factors and rule 4
MaxGhenis Nov 5, 2026
4dd5281
Merge main into uk/ab2026-harvest
MaxGhenis Nov 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,305 changes: 2,305 additions & 0 deletions data/uk/ab2026_measures.json

Large diffs are not rendered by default.

1,062 changes: 1,062 additions & 0 deletions pipeline/ab2026_harvest_contract.py

Large diffs are not rendered by default.

206 changes: 206 additions & 0 deletions pipeline/ab2026_measure_keys.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
"""Autumn Budget 2026 measure keys, one per HM Treasury Table 6.1 line.

The key space of the AB2026 harvest (#230): every staged claim's measure_key is
a key from data/uk/ab2026_measures.json, null with
conditions.proposed_measure_key (README rule 2), or for Table 6.4 recostings
the earlier event's key named in conditions.recosting_of.
Keys are written out here rather than derived from titles, so a reworded
title in a later edition of the table cannot move a key. Both the Table 6.1
stager (pipeline/stage_ab2026_hmt_tables.py) and the registry builder
(pipeline/build_ab2026_registry.py) import this module, so a staged row and
its registry entry cannot disagree about a key.
"""

PREFIX = "ab2026__"

# Table 6.1 "Budget 2026 policy decisions (£ million)", lines 1-98.
TABLE_61 = {
1: "household_energy_payment_100_bands_a_to_d",
2: "electricity_vat_zero_rate_to_july_2027",
3: "fuel_duty_5p_cut_extended_and_rpi_rise_deferred",
4: "bus_fare_cap_2_england_outside_london_2027",
5: "mileage_allowance_payments_55p",
6: "parental_allowance_fathers_partners_adopters",
7: "carers_allowance_earnings_taper",
8: "uc_surplus_earnings_threshold_2500_extended",
9: "early_years_books_access",
10: "food_and_heating_oil_tariff_suspensions",
11: "cash_isa_limit_held_at_20000",
12: "business_rates_pubs_relief_35pc",
13: "business_rates_sbrr_rhl_thresholds_raised",
14: "business_rates_rhl_transitional_support",
15: "trading_allowance_3000",
16: "business_tariff_suspensions",
17: "climate_change_levy_rates_2028",
18: "red_diesel_duty_cut",
19: "alcohol_duty_small_producer_relief",
20: "good_growth_funds",
21: "business_rates_retention_mayoral_authorities",
22: "local_apprenticeship_services",
23: "regional_growth_investments",
24: "cambridge_development_corporation",
25: "strategic_authorities_capacity_funding",
26: "founders_relief_cgt",
27: "rd_relief_claim_notification_form_removed",
28: "emi2_option_limit_120000",
29: "british_business_bank_backing_innovation",
30: "national_wealth_fund_capital",
31: "artificial_intelligence_package",
32: "ev_charging_public_chargers",
33: "rough_sleeping_investment",
34: "rough_sleeping_reprioritised_mhclg_budgets",
35: "social_and_affordable_homes_programme",
36: "vat_zero_rate_land_for_social_housing",
37: "your_first_home_equity_loan",
38: "first_time_buyer_isa",
39: "lifetime_isa_withdrawal_charge_20pc_above_cap",
40: "defence_investment_plan_further_savings",
41: "defence_investment_plan_capital_reallocation",
42: "defence_admin_savings_reinvested",
43: "ukraine_winter_air_defence",
44: "uk_export_finance_defence_capacity",
45: "hmrc_tax_gap_package",
46: "online_marketplace_liability_uk_businesses",
47: "dwp_fraud_and_error_prevention",
48: "benefits_removed_hospitalised_serious_offenders",
49: "distributions_framework_modernised",
50: "advance_corporation_tax_abolished",
51: "vat_reverse_charge_overseas_services_restricted",
52: "pension_tax_allowance",
53: "pension_death_benefits_two_year_rule_removed",
54: "foreign_sovereign_investments_taxation",
55: "bank_surcharge_6pc_temporary",
56: "machine_games_duty_higher_stake_25pc",
57: "business_rates_vape_shops_and_agcs_reliefs_removed",
58: "writing_down_allowance_12pc_and_fya_50pc",
59: "private_hire_operators_vat_full_fare",
60: "badr_lifetime_limit_500000",
61: "property_income_rates_plus_2pp",
62: "savings_income_rates_plus_2pp",
63: "civil_service_admin_budgets_cut_20pc",
64: "civil_service_ai_fund",
65: "hgv_ved_1_for_12_months",
66: "vat_5pc_family_days_out_summer_2026",
67: "free_bus_travel_children_august_2026",
68: "foreign_permanent_establishment_exemption_mandatory",
69: "electricity_generator_levy_55pc",
70: "british_industrial_competitiveness_scheme",
71: "carbon_price_support_removed",
72: "youth_guarantee_and_apprenticeships_spring_2026",
73: "union_learning_fund",
74: "investment_for_growth",
75: "growth_guarantee_scheme_expansion",
76: "leeds_city_fund",
77: "uk_india_ceta_and_double_contributions_convention",
78: "steel_tariff_and_quotas",
79: "securities_transfer_tax",
80: "business_rates_ev_charging_relief",
81: "misogyny_prevention_programmes",
82: "infected_blood_compensation_additional_report",
83: "student_loan_interest_cap_6pc_plan_2_and_3",
84: "waste_crime_landfill_tax_clearances",
85: "further_priorities_nhs_and_vawg",
86: "company_car_tax_rates_2030_31",
87: "supported_housing_licensing",
88: "vat_zero_rate_donated_medicines",
89: "vat_medical_deputies_staffing",
90: "landfill_communities_fund_credit_cap",
91: "franchised_higher_education_registration",
92: "high_needs_local_authority_spending",
93: "hvcts_deferrals_discounts_exemptions",
94: "benefits_in_kind_payrolling_delay",
95: "low_value_imports_customs_relief_removed",
96: "ets_energy_from_waste_delay",
97: "gas_distribution_revenue_recognition_spread",
98: "us_llc_double_taxation_relief",
}

# Table 6.2 "Budget 2026 Financial Transactions (cash impact, £ million)".
# Each line is the financial-transactions leg of a Table 6.1 measure, except
# line 4 (Sovereign AI), which has no Table 6.1 line and so its own key.
TABLE_62 = {
1: TABLE_61[20],
2: TABLE_61[30],
3: TABLE_61[29],
4: "sovereign_ai_financial_transactions",
5: TABLE_61[37],
6: TABLE_61[37],
7: TABLE_61[74],
8: TABLE_61[24],
9: TABLE_61[47],
}

# Table 6.4 "Measures announced at Spring Statement 2026 or earlier that will
# take effect from 29 October 2026 or later": recostings of earlier events'
# measures, keyed to those events' registries (full keys, not slugs).
TABLE_64 = {
1: ("autumn_budget_2025", "ab2025__cbam_indirect_emissions_removed"),
2: ("autumn_budget_2025", "ab2025__student_finance_lle_launch_and_loan_outlay"),
3: ("autumn_budget_2025", "ab2025__ppf_and_fas_pre_1997_inflation_protection"),
4: (
"autumn_budget_2025",
"ab2025__fya_zero_emission_cars_and_charge_points_extended",
),
5: ("autumn_budget_2025", "ab2025__student_loans_plan2_threshold_freeze"),
6: ("autumn_budget_2025", "ab2025__property_income_separate_rates"),
7: ("autumn_budget_2025", "ab2025__apd_higher_rate_extended_to_private_jets"),
8: ("autumn_budget_2025", "ab2025__db_pension_scheme_surplus_extraction"),
9: ("autumn_budget_2025", "ab2025__ets_expansion_to_international_maritime"),
10: ("autumn_budget_2025", "ab2025__sdil_threshold_cut_and_milk_based_drinks"),
11: ("autumn_budget_2025", "ab2025__high_value_council_tax_surcharge"),
12: ("autumn_budget_2025", "ab2025__eved_mileage_supplement_electric_and_phev"),
13: ("autumn_budget_2025", "ab2025__low_value_imports_customs_reform"),
14: ("autumn_budget_2025", "ab2025__salary_sacrifice_pension_nics_cap_2000"),
15: (
"autumn_budget_2024",
"autumn_budget_2024__company_car_tax_set_appropriate_percentages_for_electric_vehicles_hybrids_and_internal_combustion_engine_vehic_f0345c3d",
),
16: (
"autumn_budget_2024",
"autumn_budget_2024__inheritance_tax_include_unused_pension_funds_and_death_benefits_payable_from_a_pension_in_the_value_of_estates_215d16bf",
),
17: (
"spring_budget_2024",
"spring_budget_2024__carbon_border_adjustment_mechanism_cbam_introduce_from_1_january_2027_46125f17",
),
}

# Packages: compositions of Table 6.1 lines that HM Treasury totals or that
# outside producers score as one thing. A producer's package row joins here
# when its scope is exactly these lines; any other scope is a proposal.
PACKAGES = {
"package_total_policy_decisions": {
"title": "Table 6.1 total policy decisions",
"lines": sorted(TABLE_61),
"table_61_row": "Total policy decisions",
},
"package_total_tax_policy_decisions": {
"title": "Table 6.1 total tax policy decisions",
"lines": None, # the table's own Tax/Spend head split
"table_61_row": "Total tax policy decisions",
},
"package_total_spending_policy_decisions": {
"title": "Table 6.1 total spending policy decisions",
"lines": None,
"table_61_row": "Total spending policy decisions",
},
"package_household_energy_vat_and_fuel_duty": {
"title": "The Household Energy Payment, zero VAT on domestic electricity and the fuel duty cut, together",
"lines": [1, 2, 3],
"table_61_row": None,
},
"package_policyengine_seven": {
"title": "The seven household measures PolicyEngine modelled on Budget day, together",
"lines": [1, 2, 3, 7, 15, 52, 61],
"table_61_row": None,
},
}


def key(line):
"""The full measure key of a Table 6.1 line."""
return PREFIX + TABLE_61[line]


def package_key(slug):
return PREFIX + slug
98 changes: 98 additions & 0 deletions pipeline/ab2026_proposed_keys.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
"""Write or check the AB2026 harvest's PROPOSED_KEYS.md.

A row whose scope matches no registered measure has `measure_key: null` and
names a proposed key in `conditions.proposed_measure_key` (README rule 2).
Tranche 2 of #230 registers each proposed key, or folds it into a registered
one, before the ingest runs. This lists every proposed key once, with the
families and producers that use it and its row count, so that work starts
from a complete list. Each family's NOTES.md gives the scope of every key its
rows use; the contract checks that.

python pipeline/ab2026_proposed_keys.py sources/harvest-uk-ab2026-2026-10-29
python pipeline/ab2026_proposed_keys.py sources/harvest-uk-ab2026-2026-10-29 --check
"""

import argparse
import sys
from collections import defaultdict
from pathlib import Path

from pipeline.ab2026_harvest_contract import read_rows

HEADER = """\
# Proposed measure keys in the AB2026 harvest

Generated by `pipeline/ab2026_proposed_keys.py` from the staged families; don't edit by hand.

A row whose scope matches no registered measure has `measure_key: null` and names one of these keys in `conditions.proposed_measure_key` (README rule 2). A key is a measure with no Table 6.1 line (`ab2026__<slug>`), a package (`ab2026__package_<slug>`) or an option (`ab2026_option__<slug>`). Each family's NOTES.md gives the key's scope as Table 6.1 lines or the producer's own words. Tranche 2 registers each key in `data/uk/ab2026_measures.json`, or folds it into a registered key, before the ingest.
"""


def kind(key):
"""README rule 2's three kinds of proposed key."""
if key.startswith("ab2026_option__"):
return "option"
if key.startswith("ab2026__package_"):
return "package"
return "measure"


def inventory(harvest):
"""{proposed key: {"rows": n, "families": set, "sources": set}}."""
out = defaultdict(lambda: {"rows": 0, "families": set(), "sources": set()})
for fam in sorted(Path(harvest).iterdir()):
if not (fam / "claims_staged.jsonl.gz").exists():
continue
for r in read_rows(fam):
pk = r["conditions"].get("proposed_measure_key")
if pk:
out[pk]["rows"] += 1
out[pk]["families"].add(fam.name)
out[pk]["sources"].add(r["source"])
return dict(out)


def render(harvest):
inv = inventory(harvest)
kinds = [kind(k) for k in inv]
rows = sum(v["rows"] for v in inv.values())
counted = ", ".join(
f"{kinds.count(c)} {c}s" for c in ("measure", "package", "option")
)
lines = [HEADER, f"{len(inv)} proposed keys ({counted}) on {rows} rows.", ""]
if inv:
lines += [
"| proposed key | kind | rows | families | producers |",
"|---|---|--:|---|---|",
]
for k in sorted(inv):
v = inv[k]
lines.append(
f"| `{k}` | {kind(k)} | {v['rows']} "
f"| {', '.join(sorted(v['families']))} "
f"| {', '.join(sorted(v['sources']))} |"
)
lines.append("")
return "\n".join(lines)


def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("harvest")
ap.add_argument("--check", action="store_true")
args = ap.parse_args(argv)
path = Path(args.harvest) / "PROPOSED_KEYS.md"
text = render(args.harvest)
if args.check:
if not path.exists() or path.read_text() != text:
print("PROPOSED_KEYS.md is stale")
return 1
print("PROPOSED_KEYS.md matches the families")
return 0
path.write_text(text)
print(f"wrote {path}")
return 0


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Loading
Loading