Skip to content

Merge figures a producer prints more than once in the AB2026 ingest - #256

Merged
MaxGhenis merged 11 commits into
mainfrom
uk/ab2026-repeated-printings
Nov 13, 2026
Merged

MaxGhenis merged 11 commits into
mainfrom
uk/ab2026-repeated-printings

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

For #230, the AB2026 ingest now merges 176 rows into another printing of the same figure, so we score each of those figures once. We also fix the two low findings from GPT-6.1 Sol's second review of #255. At 849d6a93 Sol found 49 more repeats and six smaller faults in this PR, and we fix each of them in fc94bcbc. In c3511c7d we test each field of the check and tighten two of its comparisons. At c3511c7d Sol found that the check would accept two pairs of different TPA figures as repeats, and eight smaller faults, and we fix each of them in 30967b6c. In 7e15402a the ingest reads a part of a measure from a printing's component as well as its row type, and a new test covers both. In 71bffe49 we name a part on both printings where the compared-condition test varies the component. At 71bffe49 Sol found that the scan read a part of a measure from its row type alone, and that we misstate a source in two comments. We fix them in 39b17c50, and merge main in f3c07220. At f3c07220 Sol approved, and asked for a test of each marker of a part alone and for no review labels on test docstrings. We add both in 950683de.

Changes

Figures a producer prints more than once (1c181d8e)

In 53 staged rows, a producer prints a figure that it prints elsewhere in its family: in prose and in a table or chart, in a chart and its workbook, or in two of its documents. The Resolution Foundation (RF) prints 21 of these repeats, HM Treasury 11, Tax Policy Associates (TPA) 9, the OBR 6, CenTax 5 and IFS 1. On main the ingest makes a claim from each printing, so we would score each such figure more than once.

We list each row in the ingest's REPEATED_PRINTINGS, with the row whose claim we keep and our reading of the pair, in which we name where the producer prints the figure. The ingest drops the repeat as figure_printed_again and lists it under the kept claim's publication.also_printed_in, with its family, table, locator, column and printed value. We keep the most precise printing, and a point value over a bound. For example, the OBR gives Founders' Relief's 2029-30 effect as a £0.3 billion cut in receipts in paragraph 3.18 and as −261 (£ million) in Table 1.1, so we keep the table's claim.

Before it merges a pair, the ingest checks that the two claims share a source, reform key, metric, unit, year, time basis and window. It raises if they differ in any of these, or if the producer costs part of a measure in one printing and the whole measure in the other. To compare the values, the ingest finds the place of each printing's last digit in the producer's text, such as £100 million in "£2.6 billion". Where the producer states each printing's sign convention, it compares the values in PolicyEngine's orientation, and otherwise it compares their magnitudes. It raises if they differ by more than half the coarser printing's last digit, if we keep the coarser printing, or if we keep a bound over a point value. In 32 of the 53 pairs, the producer prints the same value twice. In the tightest pair, HM Treasury rounds Table 6.3's +2,250m to £2.3 billion in footnote 73, a gap of exactly half a last digit, which the ingest accepts.

The ingest also raises if it keeps a claim that shares its id with a repeat it drops, since we would then score the figure twice. For example, RF prints one figure, 2.6 percentage points, in its key findings and again on page 45, and gives it in full in the workbook for Figure 23. On main the ingest merges the first two printings (rows 60 and 205) into one claim as twins. If we merged row 205 alone into the workbook's claim (row 765), we would keep row 60's claim beside it, so we merge both.

So at this commit the ingest reads 5,959 rows and writes 4,636 claims and 1,323 drops: 52 fewer claims and 52 more drops than on main. It drops 53 rows as figure_printed_again, and one fewer as republished_twin_merged, since it no longer merges rows 60 and 205 as twins. In each family's notes we list the rows we merge, and in REPEATED_PRINTINGS we name the row we keep for each.

Tests for three surviving mutants (849d6a93)

We ran 15 mutants of the ingest's printing code at 1c181d8e, and three of them passed every test. When we removed the check that we keep the more precise printing, the property test still raised on the tolerance alone. So we add two examples to it: one value at two precisions, and a pair exactly half a last digit apart. A mutant that compared magnitudes where both sign conventions classify also passed, so a new test flips the sign of the OBR's Table 1.1 printing of Founders' Relief and requires the ingest to raise.

The third mutant ignored a workbook exponent and changed no result, because the ingest folds the exponent into the power of ten between the printed number and the value. So we drop the exponent from _printed_place, and the ingest writes the same claims.

Findings from Sol's first review of this PR (fc94bcbc)

Sol listed 44 repeats that we had missed (M1): 38 OBR rows, five RF rows and one TPA row. For example, the OBR prints the council tax rebate's 2026-27 cost as 2.6 (£ billion) of extra borrowing in Table 3.1 and as −2,627 (£ million) in Table 1.1. When we widened our own scan to every two claims that share an identity, zeros included, we found 74 more:

  • 60 rows of HM Treasury's policy costings, where it costs the ten measures in the government's package to close the tax gap for six years each. It prints the same figures again in red book Table 6.3, where we keep them, so in REPEATED_PRINTINGS we now give a kept row in another family as (family, row);
  • red book row 189, where HM Treasury prints its count of 20 million households a third time;
  • the council tax rebate from 2027-28 to 2031-32 and the unfunded pensions valuation effect in 2026-27 (rows 2565–2569 and 2576), which the OBR prints as zero in both Table 3.1 and Table 1.1;
  • the OBR's roundings, in paragraph 3.19, of the electricity VAT cut's cost in 2026-27 and 2027-28 (rows 2680 and 2681);
  • and the online marketplace measure's yield from 2027-28 to 2031-32, which the OBR prints in both Table 3.3 and Table 1.1 (rows 2712–2716).

Sol advised keeping those online marketplace rows apart from the measure's total, since it read them as the measure's VAT component. The OBR costs that measure in VAT receipts alone, so it prints one figure in both tables, with opposite signs, and we stage the Table 3.3 rows as the whole measure. So we merge them, and we ask Sol to check this call again.

In three of the repeats Sol listed, the producer gives a bound in one printing and a point value in the other (M2). CenTax says the bottom 50% would pay less than 0.5% of the revenue from NICs on partnerships, and prints their share as 0.40%. RF prints "just over half" and 52 per cent for one share, and "over a million" and 1,572,152 for one count of families. Our check read each bound as a rounded point value, so it raised when we listed these pairs. Now we keep the point value, and we merge a bound into it only where the point lies at most half its own last digit past the bound, and the two printings share a sign convention. We never merge a point value into a bound. As Sol advised, we also merge RF's second "over a million" (row 137) into the workbook's 1,572,152, and both of RF's printings of 91% (rows 59 and 138) into the unrounded share in its workbook.

HM Treasury prints two figures that we had staged on different time bases (M3). It expects the government to take around £50 off the annual Ofgem price cap in April 2027, and counts up to 20 million households that the government pays the Household Energy Payment from January 2027. In ROW_TIME_BASES we now stage row 5 as annual, as we stage rows 188, 190 and 197, and rows 180 and 189 at a point in time, as we stage row 7. Then we merge row 5 into row 188, and rows 180 and 189 into row 7. The ingest raises if a restaged row's printed value, or the time basis on which we stage it in the harvest, changes.

Our check also let through two printings that differ in scope (L1). For example, if RF gave its workbook figure for the fuel duty freeze for Scotland alone, the check would still merge the chart's label into it. Now the ingest raises if a repeat and the printing we keep differ in any of eleven scope keys, such as geography, income group, component and price basis, unless we list the two wordings in _SAME_SCOPE. There we list 33 pairs of wordings that we read as one scope, such as each tax-gap measure's full name in HM Treasury's costings and its short name in red book Table 6.3. We read a claim without an income group as covering everyone, and one without a component as covering the whole measure.

In binary floating point, 0.58 − 0.575 = 0.0050000000000000044, so our check would have rejected 58% against 57.5%, exactly half a last digit apart (L2). Now we allow a few units in the last place of each value. The ingest also raised when a repeat shared its claim id with the claim we merge it into, though only that claim remained (L3). Now it tracks the claims we merge each repeat into, and raises only when we keep another claim with the repeat's id, such as RF row 60's if we merged row 205 alone into row 765.

Our test of the rule, test_no_claim_repeats_another_printing_of_its_figure, missed these repeats because it skipped zeros and any two claims whose conditions differed outside a short list of keys. Now it groups claims by identity and part flag alone, zeros included, and skips two lines of one table and two claims whose scopes differ. It finds 24 pairs outside REPEATED_PRINTINGS, and we list each in NOT_REPEATS with our reason for reading them as two figures. For example, the OBR costs the changes to HMRC's compliance workforce at £0.4 billion in 2031-32, in paragraph 3.13, and prints the whole tax-gap package's onshore corporation tax that year as 445 (£ million) in Table 1.1.

In this description we now say which rows we list in each family's notes (L4), and we split the staging paragraph (L6). In a test docstring we name ourselves, not RF, as printing the example (L5), and in each family's notes we list the new merges.

So the ingest now reads 5,959 rows and writes 4,514 claims and 1,445 drops: 122 fewer claims than at 849d6a93, and 174 fewer than on main. It drops 176 rows as figure_printed_again: 74 of HM Treasury's (14 in the red book and 60 in its costings), 57 of the OBR's, 28 of RF's, 10 of TPA's, 6 of CenTax's and 1 of IFS's. It drops two fewer rows than main as republished_twin_merged, since we now merge red book rows 180 and 189, and RF rows 60 and 205, as repeats rather than as twins.

Tests for more surviving mutants, and two tighter comparisons (c3511c7d)

At fc94bcbc we ran 31 more mutants of the printing check, on what the earlier rounds left out, and 14 of them passed every test. In the identity, our tests changed only the year, metric and part flag, and they changed only five of the eleven scope keys. So we could have dropped the source, reform key, unit, time basis or window from the identity, or six scope keys from _SCOPE, and no test would have failed. Now test_a_printing_of_another_figure_raises changes each field of the identity in turn, and test_a_printing_in_another_scope_raises gives each scope key a wording we don't list.

New tests also cover two bounds of opposite directions and printings where the producer gives the direction in a label, and check that we list in _SAME_SCOPE only the wordings that repeats use. We add two examples to the property test, a gap just over half a last digit and a pair of opposite signs, since its random search doesn't reliably reach them.

We also tighten two comparisons, and neither changes a claim. Past a bound, the ingest compared values without the allowance for units in the last place that we added for L2. In binary floating point, 0.03 + 0.005 falls short of 0.035, so it would have rejected 3 per cent against a bound of over 3.5 per cent, and now _within_bound calls _within_half for that comparison. The ingest also compared two printings' estimate kinds only where the producer gave a bound in one of them. So it would have merged a cumulative figure into a single year's figure of the same size, and now two point values must share a kind.

In the docstring of _within_bound, in a test docstring and in this description, we reword "with half its last digit to spare". A reader could take it to mean that a point must clear the bound by half a digit, and the ingest lets a point lie up to half a digit past the bound.

Findings from Sol's second review of this PR (30967b6c)

Sol found that the printing check would accept two pairs of different TPA figures as repeats (L7). TPA puts a notional investment bank's tax in Singapore at 14.9% of its profit, in total on row 274 and from corporate income taxes alone on row 284. On rows 287 and 288 it puts the bank's UK bank levies and deposit insurance at 2.3% of profit, before and after the Budget. The check compared only the keys in _SCOPE, and we had left household type, tax component, country and scenario out of that list. It also accepted each pair of wordings in _SAME_SCOPE for any repeat. We listed an unstated subgroup and first-time buyers as one scope for red book row 256, so the check would have merged RF row 147 into row 433 had we staged row 433 for first-time buyers.

Now the check compares every condition outside _PER_PRINTING, the 14 keys we read as belonging to each printing rather than to the figure, such as its table, heading and printed value. It still reads a claim without an income group as covering everyone, and one without a component as covering the whole measure. Where we stage a condition with one wording on a repeat and another on the claim we keep, we list both wordings in _READINGS with the repeat's row. The check raises on any other repeat that differs in that condition.

In 154 of the 176 repeats, the two printings differ in at least one compared condition, and we list 577 pairs of wordings, one for each repeat and condition. For example, we label the basis of HM Treasury's costings a forecast and that of red book Table 6.3 post-behavioural, though HM Treasury prints each costing after any behavioural effect in both. We list that pair of wordings for the 60 costings rows alone. New tests require the check to raise on each of the pairs above, and when we give any compared condition a wording that we don't list.

Sol also found that the scan for unlisted repeats folded a third printing of CenTax row 90 into the pair we keep apart in NOT_REPEATS (L8). It keyed each claim by its family, locator, year and printed value, so it couldn't tell the third printing from row 90. Now it keys each pair by its two claim ids, and we key NOT_REPEATS by the two rows of each pair. A new test adds a third printing of row 90 and requires the scan to pair it with rows 90 and 92 and no other claim.

In Sol's run, a mutant that dropped the staged time basis from _check_row_calls passed every test, because our test moved the basis and the printed value together (L9). Two new cases move each field of red book row 5 alone, and two more do the same for the metric and printed value of IFS row 46.

We wrote in this description that the tests reject a value off its printing by a power of ten, but _printed_place accepts 50 or 500 for a printed £5 (F1). Now we write that the tests require the ingest to raise on a value other than its printed number times a power of ten, and we rename the test and a mutant to match. We also cut a clause from the _printed_place docstring, since the ingest merges RF's "over a million" with 1,572,152 through the bound check (L10).

In the scan's docstring we now say that we staged each condition for both printings (L11). In the guard comment we say that the producer sums several years in one printing and gives one year in the other (L13). We drop the label from the costings notes' paragraph on red book Table 6.3 (L12), and give the five families' differences their own sentence in the staging paragraph below (L6).

The ingest writes the same 4,514 claims and 1,445 drops at 30967b6c as at c3511c7d, and the tranche 3 stager reads the same 352 claims.

The part flag in the identity (7e15402a and 71bffe49)

Where the producer costs part of a measure, we stage its printing with a component other than the measure's total, or with the row type of a measure component. The ingest set the identity's part flag from the row type alone. So had we listed a component we name and the measure's total as two wordings for one repeat, the ingest could have merged a part of the measure into the whole. When we reran the third round of mutants at 30967b6c, the mutant that drops the part flag from the identity passed every test. Our test of the flag marked a part by its row type, and at 30967b6c the check also compared the row type as a condition, so it raised before it reached the identity.

In 7e15402a, _costs_part reads a part of a measure from either the component or the row type, and the ingest sets the identity's part flag from it. test_a_listed_wording_never_merges_a_part_into_the_whole marks the claim we keep as a part, once by its component and once by its row type, lists both wordings for the repeat, and requires the ingest to raise. On all 176 repeats the old and new flags agree, so the ingest writes the same 4,514 claims at 7e15402a as at c3511c7d.

At 7e15402a the compared-condition test named a component on one printing alone, so the check raised on the identity before it compared the component. A mutant that reads the component as belonging to each printing then failed only the test that pins _PER_PRINTING. In 71bffe49 the test names a part on the other printing too, so the check must catch the different part, and accept it once we list both wordings.

Findings from Sol's third review of this PR (39b17c50)

Sol found that the scan for unlisted repeats read a part of a measure from its row type alone (L14). Had we marked a printing as a part by its component alone, the scan would have grouped it apart from the printings we mark by row type, and missed a repeat between them. Now the scan reads a part from either the component or the row type, in its own code rather than through _costs_part. test_the_scan_reads_a_part_from_its_component_alone adds a third printing of CenTax row 90, stages it with row 90's component and the row type of a whole measure, and requires the scan to pair it with rows 90 and 92. We stage no claim that way, so the scan still finds the 24 pairs in NOT_REPEATS and no others.

Sol also found that we misstate a source in two comments. CenTax prints 2030 with both printings of the £640 million, and we record the year in the kept printing's year note alone (F2). HM Treasury prints each tax-gap measure's full name in red book Table 6.3, and we record a short name on those rows (F3). We now say so in each comment, and in the comment on the compliance workforce, HMRC expands its workforce under the measure (L15). Sol flagged two sentences in the commit message of 30967b6c where rows act (L16). We leave that pushed message as it stands, and in this description's section on 30967b6c, TPA puts each figure on its row.

In 39b17c50 we change only comments in the ingest, so it writes the same claims as at 71bffe49. When we merge main at be27e8ac, we take in changes to the replay sandbox's scripts, CI and replay tests, none of them in the ingest or its tests.

Findings from Sol's fourth review of this PR (f3c07220)

Sol approved at f3c07220 and raised two low findings. Two mutants of the scan's part reading passed every scan test, and one of them read a part from its named component alone (L17). test_the_scan_reads_a_part_from_either_marker adds a third printing of CenTax row 90 in three cases: we drop its component, name the measure's total in its place, or drop its row type. In each case the scan must pair the printing with rows 90 and 92 and no other claim. Sol also asked us to drop the review label at the start of the new test's docstring (L18), and we drop it from all nine test docstrings to which this PR adds one.

Tranche 3 (1c181d8e and fc94bcbc)

At 1c181d8e the tranche 3 stager reads 363 claims, 5 fewer than on main, and pairs the same 35 with our estimates. It lists the other 328 under the same 16 reasons, so we re-pin its expected tally and regenerate STAGING_TALLY.json and STAGING_UNANSWERED.jsonl. It no longer reads five claims: two on the electricity VAT cut's typical-bill saving, and one each on fuel duty's count of motorists, the energy payment's share of households and the Pension Tax Allowance's worked example.

At fc94bcbc it reads 352 claims, 11 fewer, pairs the same 35 and lists the other 317 under the same 16 reasons, so we re-pin and regenerate the same three files. It no longer reads HM Treasury's "around £50" off the price cap or its count of 20 million households, or RF's "around one-in-eight" on page 20 of its report. Nor does it read eight OBR claims: two roundings of the electricity VAT cut in paragraph 3.19, and six Table 3.1 lines on the council tax rebate.

Wording fixes from #255's review (950a1830)

GPT-6.1 Sol approved #255 with two low findings on wording, which we fix here. In scorecard_db/baselines.py we rewrite the latest-policy baseline's description to name who acts: the OBR assumes that the government implements the Budget's measures, and the government freezes fuel duty until July 2027 and raises it from then. In the OBR notes we name the government as the body that would borrow more and collect less if it kept the freeze.

Staging (4c363635, d63699ce and 62dcd82c on uk/ab2026-harvest-staging)

On the staging branch we fix the OBR notes as on main, and list the rows we merge in the OBR and IFS notes. In 62dcd82c we list the 57 OBR rows we now merge, so we keep one text of each on both branches. Our notes for HM Treasury's costings and red book, RF, TPA and CenTax differ between the two branches in 346, 294, 344, 205 and 161 lines. We change those five notes on this branch alone, and leave their parity to a separate change.

Invariants

  1. In each pair in REPEATED_PRINTINGS, the two claims share a source, reform key, metric, unit, year, time basis and window, and the producer costs either the whole measure in both or part of it in both. The two claims also agree on every condition outside _PER_PRINTING, or differ only by wordings that we list for that repeat in _SAME_READING. We read a claim without an income group as covering everyone, and one without a component as covering the whole measure.
  2. In each pair of point values, the two printings share an estimate kind, and the two values differ by at most half the coarser printing's last digit. We keep the printing whose last digit has the finer place, or either one when the places match. Where the producer gives a bound in one printing, we keep the other, a point value under the same sign convention, which lies at most half its own last digit past the bound. Where it gives a bound in both, it bounds the figure in the same direction in each.
  3. The ingest reads 5,959 rows and writes 4,514 claims and 1,445 drops, and in each family the claims and drops add up to the rows it reads.
  4. We keep no claim that shares its id with a repeat we drop, other than the claim we merge that repeat into.
  5. Against main's ingest, we remove 174 claims and add none. Apart from also_printed_in, the ingest writes the other 4,514 with the same id, value, conditions and publication, and on 166 of them it lists the 176 repeat rows under also_printed_in, each row once.
  6. Among the claims we keep, no two share the identity in invariant 1, cover the same place, housing cost basis, income group and averaging window, and agree to half the coarser printing's last digit. We leave out two lines of one table, and the 24 pairs in NOT_REPEATS, where we give our reason for reading each pair as two figures.
  7. The tranche 3 stager reads 352 claims, pairs the same 35 and lists the other 317 under 16 reasons.

Tests

  • test_a_repeated_printing_merges_into_the_printing_we_keep runs once for each of the 176 pairs. It stages the two rows alone, in their own families, and requires one claim, with the kept row's printed value and locator, and the repeat under also_printed_in.
  • test_each_repeated_printing_pairs_the_rows_we_read pins both printed values of each pair, as the producer prints them, against the staged rows.
  • test_each_repeated_printing_leaves_one_claim checks the full ingest's drops and listings in each family, and across families for HM Treasury's costings.
  • test_no_claim_repeats_another_printing_of_its_figure checks invariant 6. It pairs claims by a rule we wrote apart from REPEATED_PRINTINGS, with its own reading of the last digit and the sign, and finds no pair outside the 24 in NOT_REPEATS. It keys each pair by its two claim ids, and test_the_scan_pairs_a_third_printing_with_both_listed_claims requires it to pair a third printing of CenTax row 90 with rows 90 and 92 and no other claim.
  • test_the_scan_reads_a_part_from_its_component_alone requires the scan to read a part of a measure from its component when we stage the printing under the row type of a whole measure.
  • test_the_scan_reads_a_part_from_either_marker requires the scan to read a part of a measure from either marker alone: the row type measure_component, with no component or the measure's total in its place, or a named component with no row type.
  • Hypothesis tests check the half-digit tolerance and the place of the last digit across commas, decimals, units and exponents, and that the ingest raises on a value other than its printed number times a power of ten. The existing test_any_sample_of_the_harvest_closes_its_accounting still passes, and the ingest skips a pair when a sample lacks one of its rows.
  • Example tests change each field of the identity in invariant 1 and give each condition outside _PER_PRINTING a wording we don't list, and the ingest must raise on each. They also require it to raise on a kept bound, on two bounds of opposite directions, on two point values of different estimate kinds, and on a point value outside a bound or under the opposite sign convention.
  • test_a_printing_for_another_part_scenario_or_group_raises requires the ingest to raise on TPA rows 274 and 284, and on rows 287 and 288. When we stage red book row 400 for one adult, or RF row 433 for first-time buyers, it also requires the ingest to raise on that row and the repeat we merge into it.
  • test_we_list_each_wording_for_the_repeat_that_uses_it requires us to list in _SAME_READING each wording on which a repeat and the claim we keep differ, under that repeat's row, and no others. test_the_printing_check_leaves_out_the_conditions_we_name pins the 14 keys of _PER_PRINTING.
  • test_a_listed_wording_never_merges_a_part_into_the_whole marks the claim we keep as part of the measure, by its component or by its row type, and requires the ingest to raise even when we list both wordings for the repeat.
  • Other examples require the ingest to raise on a printing whose sign we flip and on a missing row in a full ingest. On a restaged row, it must raise when either its printed value or its time basis or metric in the harvest changes.
  • We also require the ingest to merge a bound into a point value that its rounding alone brings within the bound, up to exactly half its last digit past the bound, and to accept 58% against 57.5%. It must compare magnitudes where a producer gives a printing's direction in its label, and merge a repeat that shares its kept claim's id. It must also stage each restaged row on the time basis we give it and find no last digit in a printing without digits. We check the twin guard on RF's rows 60, 205 and 765.

Checks

We ran these at 71bffe49, in a clone with origin/main at 1a93ec23:

PYTHONPATH=. .venv-tests/bin/python -m pytest --noconftest -n 4 -rs \
  tests/test_ab2026_harvest_contract.py tests/test_ab2026_key_decisions.py \
  tests/test_ab2026_registry.py tests/test_uk_ab2026_harvest.py \
  tests/test_uk_ab2026_ingest.py tests/test_uk_ab2026_counterparts.py tests/test_signs.py
  • All 2,696 tests passed, and pytest skipped none. We linked the policyengine-uk 2.151.0 venv into the clone, so test_the_registry_is_regenerated_byte_for_byte ran. We pass --noconftest to skip the repository database build, which needs files outside our sparse clone.
  • python -B pipeline/stage_uk_ab2026.py stage rewrote nothing.
  • apply_ab2026_key_decisions.py --check, build_ab2026_registry.py --check, harvest_counts.py --check and our write_expected.py --check passed, and so did ruff format --check on scorecard_db, tests and pipeline.
  • We ran 1a93ec23's ingest and this branch's over the same harvest and compared their claims to confirm invariant 5.
  • Among the 167 pairs of point values, we find the widest gap, relative to the tolerance, where HM Treasury rounds +2,250m to £2.3 billion: exactly half a last digit. In 103 of the pairs the two values match once we orient their signs.

We applied each mutant to the ingest on its own, ran tests/test_uk_ab2026_ingest.py, and restored the ingest after each. Where the test module fails to load, the full ingest raises when the module runs it.

Mutant, at 1c181d8e Result
We skip the merge loop 61 failed
We accept a gap of a whole last digit 1 failed
We reject a gap of exactly half a last digit Fails to load, on HM Treasury's £2.3 billion
We drop the check that we keep the more precise printing Passed; fails at 849d6a93
We drop the twin guard 1 failed
We ignore a part of a measure in the identity 1 failed
We ignore the year in the identity 1 failed
We drop the check that we keep no bound over a point value 1 failed
We ignore a workbook exponent Passed; we drop the exponent at 849d6a93
We accept a value other than its printed number times a power of ten 1 failed
We compare magnitudes where both sign conventions classify Passed; fails at 849d6a93
We list no other printing on the kept claim 54 failed
We skip a pair with a missing row in a full ingest 1 failed
We keep RF row 764 in place of row 765 Fails to load
We drop TPA row 8 from the list 7 failed
Mutant, at 849d6a93 Result
We drop the check that we keep the more precise printing 1 failed
We compare magnitudes where both sign conventions classify 1 failed
We drop the decimal places from the place of the last digit 3 failed
We read only the whole number before the point Fails to load
Mutant, at fc94bcbc Result
We skip the merge loop 186 failed
We accept a gap of a whole last digit 1 failed
We reject a gap of exactly half a last digit Fails to load
We allow no units in the last place 1 failed
We may keep the coarser printing 1 failed
We may keep a bound over a point value 1 failed
We merge a bound into any point value 8 failed
We allow a point value nothing past a bound 2 failed
We ignore a bound's sign convention 1 failed
We compare magnitudes where both sign conventions classify 1 failed
We drop the decimal places from the place of the last digit Fails to load
We read only the whole number before the point Fails to load
We accept a value that doesn't match its printing 1 failed
We drop the twin guard 1 failed
The twin guard ignores the claim we merge into 1 failed
We ignore a part of a measure in the identity 1 failed
We ignore the year in the identity 1 failed
We drop the scope check 5 failed
We read no two wordings as the same scope Fails to load
We read a claim without an income group as its own scope Fails to load
We read a claim without a component as its own scope Fails to load
We drop geography from the scope keys 1 failed
We drop the red book's restaged time bases Fails to load
We restage red book row 5 to the time basis it already has Fails to load
We skip the check that a restaged row still reads as we read it 1 failed
We look up a kept row in the repeat's family Fails to load
We pair each tax-gap costing with the red book's next year Fails to load
We list no other printing on the kept claim 177 failed
We skip a pair with a missing row in a full ingest 1 failed
We keep RF row 764 in place of row 765 Fails to load
We drop TPA row 8 from the list 7 failed
Mutant, at fc94bcbc, fourth round Result
We ignore the source in the identity Passed; 1 failed at c3511c7d
We ignore the reform key in the identity Passed; 1 failed at c3511c7d
We ignore the metric in the identity 1 failed
We ignore the unit in the identity Passed; 1 failed at c3511c7d
We ignore the time basis in the identity Passed; 1 failed at c3511c7d
We ignore the window's start in the identity Passed; 1 failed at c3511c7d
We ignore the window's end in the identity Passed; 1 failed at c3511c7d
We drop income_group from the scope keys 1 failed
We drop unit_population from the scope keys Passed; 2 failed at c3511c7d
We drop subgroup from the scope keys Passed; 2 failed at c3511c7d
We drop denominator from the scope keys Passed; 2 failed at c3511c7d
We drop statistic from the scope keys Passed; 2 failed at c3511c7d
We drop housing_costs from the scope keys 1 failed
We drop component from the scope keys 1 failed
We drop aggregation from the scope keys 1 failed
We drop horizon from the scope keys Passed; 2 failed at c3511c7d
We drop price_basis from the scope keys Passed; 2 failed at c3511c7d
We read a lower bound as an upper bound Fails to load
We take a bound's allowance from the bound's last digit 2 failed
We read only a lower bound as a bound Fails to load
We read only an upper bound as a bound Fails to load
We merge two bounds of opposite directions Passed; 1 failed at c3511c7d
We compare raw values where a sign convention doesn't classify Passed; 3 failed at c3511c7d
We orient only the repeat's value Fails to load
We list a printing without its locator 176 failed
We list a printing without its printed value 177 failed
We list a printing without its family 177 failed
We list a printing without its table 176 failed
We list a printing without its column 176 failed
We tally a repeat as a twin 181 failed
We keep the family's claim count when we merge Fails to load

At c3511c7d we reran the third and fourth rounds, 62 mutants in all, on a clean clone, and each of them fails a test or stops the module loading. We add a fifth round for the edits we make in c3511c7d:

Mutant, at c3511c7d, fifth round Result
We list a wording in _SAME_SCOPE that no repeat uses 1 failed
We leave the listing of other printings unsorted 1 failed
We orient both values when either printing's sign convention classifies 3 failed
We allow no units in the last place past a bound 2 failed
We merge two point values of different estimate kinds 1 failed
We drop the check that a point value lies within a lower bound 4 failed
We drop the check that a point value lies within an upper bound 3 failed

At 71bffe49 we rerun the third to fifth rounds, 69 mutants in all, on a clean clone, and each of them fails a test or stops the module loading. Sixteen of them changed _SCOPE or _SAME_SCOPE, and one changed the identity's part flag, so we rewrite those 17 for the new code:

Rewritten mutant, at 71bffe49 Result
We ignore a part of a measure in the identity 2 failed
We drop the check of the other conditions 6 failed
We accept no listed wording Fails to load
We stop reading a claim without an income group as covering everyone Fails to load
We stop reading a claim without a component as covering the whole measure Fails to load
We read geography as belonging to each printing 3 failed
We read income_group as belonging to each printing 3 failed
We read unit_population as belonging to each printing 3 failed
We read subgroup as belonging to each printing 4 failed
We read denominator as belonging to each printing 3 failed
We read statistic as belonging to each printing 3 failed
We read housing_costs as belonging to each printing 3 failed
We read component as belonging to each printing 3 failed
We read aggregation as belonging to each printing 3 failed
We read horizon as belonging to each printing 3 failed
We read price_basis as belonging to each printing 3 failed
We list a wording for a repeat that doesn't use it 1 failed

We add a sixth round for the edits we make in 30967b6c and 7e15402a:

Mutant, at 71bffe49, sixth round Result
We read tax_component as belonging to each printing 4 failed
We read scenario as belonging to each printing 3 failed
We read household_type as belonging to each printing 4 failed
We read country as belonging to each printing 3 failed
We accept a listed wording for any repeat 1 failed
We compare only the conditions that both claims state 4 failed
We drop the defaults for a missing income group and component Fails to load
We drop the last row of each range we list Fails to load
We move the compliance workforce measure to the next measure's rows Fails to load
We check a restaged time basis by its printed value alone 1 failed
We check a restaged time basis by its staged basis alone 1 failed
We check a restaged metric by its metric alone 2 failed
We check a restaged metric by its printed value alone 1 failed
We read a part of a measure from its row type alone 1 failed
We read a part of a measure from a named component alone 1 failed

At 39b17c50 the 348 tests in tests/test_uk_ab2026_ingest.py pass, and we add a seventh round for the scan:

Mutant, at 39b17c50, seventh round Result
We read a part in the scan from its row type alone 1 failed
We drop the part flag from the scan 3 failed

At 950683de the 351 tests in tests/test_uk_ab2026_ingest.py pass, and we add an eighth round for the scan's part reading:

Mutant, at 950683de, eighth round Result
We read a part from its row type alone 2 failed
We read a part from its component alone 2 failed
We read the measure's total as a named component 6 failed
We require both markers 4 failed
We read any row type but a whole measure's as a part 6 failed
We drop the part flag from the scan 6 failed

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits November 5, 2026 22:12
We answer the two low findings from GPT-6.1 Sol's second review of #255. In
the latest-policy baseline's description (R1), we name the OBR as the body
that assumes the Budget's measures and costs a continued freeze, and the
government as the one that freezes fuel duty and raises it. In the OBR notes
(R2), we name the government as the body that would borrow more and collect
less if it kept the freeze.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In 53 staged rows, a producer prints a figure that it prints elsewhere in its
family: in prose and in a table or chart, in a chart and its workbook, or in
two of its documents. We list each row in REPEATED_PRINTINGS with the row
whose claim we keep and our reading of the pair. The ingest drops the repeat
as figure_printed_again and lists it under the kept claim's also_printed_in,
so it now writes 4,636 claims and 1,323 drops from 5,959 rows.

The ingest raises if a pair differs in key, metric, unit, year or time basis,
or if the producer costs part of a measure in one and the whole measure in
the other. It also raises if the values differ by more than half the coarser
printing's last digit, or if we keep the coarser printing or a bound over a
point value. If we keep a claim that shares its id with a repeat we dropped,
the ingest raises as well. New tests pin each pair's printed values, check
each guard, and scan the claims for any pair we haven't listed.

The tranche 3 stager now reads 363 claims, 5 fewer, and still pairs 35, so we
re-pin its tally and regenerate STAGING_TALLY.json and
STAGING_UNANSWERED.jsonl. In each family's notes we list the rows we merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three of the fifteen mutants we ran on 1c181d8 passed every test. When we
removed the check that we keep the more precise printing, the property test
still raised on the tolerance alone. So we add two examples to it: one value
at two precisions, and a pair exactly half a last digit apart. A mutant that
compared magnitudes where both sign conventions classify also passed, so a
new test flips the sign of the OBR's Table 1.1 printing of Founders' Relief
and requires the ingest to raise.

The third, which ignored a workbook exponent, changed no result, because we
fold the exponent into the power of ten between the printed number and the
value. So we drop the exponent from _printed_place and say why in its
docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policyengine-scorecard Ready Ready Preview Nov 13, 2026 9:35pm UTC

Request Review

MaxGhenis added a commit that referenced this pull request Nov 6, 2026
In #256 we now merge 57 OBR rows into another printing of the same figure
at ingest. We list them in these notes too, so we keep one text of the OBR
notes on both branches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 849d6a9 GPT-6.1 Sol found 49 more rows on which a producer prints a
figure again (M1 to M3) and three faults in our checks (L1 to L3). With
a wider scan of our own we found 74 more, among them 60 of HM Treasury's
costings that it prints again in red book Table 6.3, so we now merge 176
rows. In REPEATED_PRINTINGS we give a kept row in another family as
(family, row).

We merge a bound only into a point value that lies within it, with half
the point's last digit to spare and under the same sign convention, and
we never merge a point value into a bound. In ROW_TIME_BASES we restage
red book rows 5, 180 and 189 on the time basis of the printing we keep,
and the ingest raises if a restaged row's printing, or the time basis on
which we stage it in the harvest, changes.

The ingest now raises if a repeat and the printing we keep differ in a
scope key, unless we list the two wordings in _SAME_SCOPE (L1). At half
a last digit it allows a few units in the last place (L2), and its twin
guard lets through a kept claim that shares its id with its own repeat
(L3). The repeat scan now groups claims, zeros included, by identity and
part flag alone, and in NOT_REPEATS we give our reason for reading each
of the 24 pairs it finds as two figures.

So the ingest writes 4,514 claims and 1,445 drops from 5,959 rows, and
the tranche 3 stager reads 352 claims, pairs the same 35 and lists 317
under 16 reasons. We re-pin both, regenerate the stager's outputs and
list the new merges in each family's notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mutants at fc94bcb passed every test when they dropped the source, reform
key, unit, time basis or window from a printing's identity, or six scope keys
from _SCOPE. So test_a_printing_of_another_figure_raises now changes each field
of the identity in turn, and test_a_printing_in_another_scope_raises gives each
scope key a wording we don't list. New tests also cover two bounds of opposite
directions, printings that give their direction in their labels, and _SAME_SCOPE
itself, which must list only the wordings that repeats use. Two more examples
on the property test catch a gap just over half a last digit and a pair of
opposite signs.

Past a bound, the ingest now calls _within_half, so it allows a few units in
the last place there too: 0.03 + 0.005 falls short of 0.035 in binary floating
point. Two point values must now share an estimate kind, where before the
ingest compared kinds only when one printing gave a bound. Neither change moves
a claim.

We also reword "with half its last digit to spare". A reader could take it to
mean that a point must clear the bound by half a digit, and the ingest lets a
point lie up to half a digit past the bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing c3511c7, GPT-6.1 Sol found that the printing check would accept
two TPA pairs that we don't list. Rows 274 and 284 put a notional bank's tax in
Singapore at 14.9% of profit, in total and from corporate income taxes alone.
Rows 287 and 288 put its UK bank levies at 2.3% before and after the Budget.
The check compared only the conditions in _SCOPE, and it accepted each pair of
wordings in _SAME_SCOPE for any repeat.

It now compares every condition outside _PER_PRINTING, the 14 keys we read as
belonging to each printing, such as its table, heading and printed value. It
reads a claim without an income group as covering everyone, and one without a
component as covering the whole measure. Where we stage a condition with one
wording on a repeat and another on the claim we keep, we list both wordings in
_READINGS with the rows of the repeats that use them. The check raises on any
other repeat that differs in that condition. In 154 of the 176 repeats, the
two printings differ in at least one such condition, and we list 577 pairs of
wordings, one for each repeat and condition. A test fails if we list a pair
that its repeat doesn't use, or leave out one that a repeat uses.

A new test changes each compared condition in turn, on RF row 147 and then on
row 433, which we merge it into, and the check must raise on each change.
Others require it to raise on the two TPA pairs, and on red book rows 362 and
400 once we stage row 400 for one adult. When we stage RF row 433 for
first-time buyers, it raises on rows 147 and 433 too.

Sol also found that the scan for unlisted repeats folded a third printing of
CenTax row 90 into the pair we keep apart in NOT_REPEATS. It keyed each claim
by its family, locator, year and printed value. It now keys each pair by
its two claim ids, and NOT_REPEATS names the two rows of each pair. A new test
adds a third printing of row 90 and requires the scan to pair it with rows 90
and 92 and no other.

In Sol's run, a mutant that dropped the staged time basis from _check_row_calls
passed every test, because the old test moved the basis and the printed value
together. Two new cases move each field of red book row 5 alone, and two more
do the same for the metric and printed value of IFS row 46.

We also cut a clause from the _printed_place docstring, since the ingest merges
RF's "over a million" with 1,572,152 through the bound check. We name who acts
in the guard comment and the scan's docstring, drop a label from the costings
notes, and rename the power-of-ten test after what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng identity

When we reran the third mutant round at 30967b6, no test failed after we
dropped the part flag from the printing identity. The one test that marked
a part did so by its row type, and the ingest compares the row type as a
condition too, so it raised on the conditions before it reached the
identity.

We read a part of a measure from a component other than the measure's total
as well as from its row type (`_costs_part`), and put that flag in the
identity. A new test lists both wordings for a repeat that differs only in
marking a part, by component or by row type, and requires the ingest to
raise. The compared-condition test expects the ingest to refuse a listed
component wording, since the ingest reads one printing with that wording as
part of the measure.

On all 176 repeats the old and new flags agree, so the ingest merges the
same pairs, and we get the same claims as at fc94bcb, byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the component

At 7e15402 the ingest reads a printing with a component other than the
measure's total as part of the measure. The compared-condition test named a
component on one printing alone, so the check raised on the identity before
it compared the component. A mutant that reads the component as belonging to
each printing then failed only the test that pins `_PER_PRINTING`.

For the component, the test now names a part on the other printing too, so
the ingest reads both as parts of the measure and must catch the different
part, then accept it once we list both wordings. The mutant fails both cases
of the test again, and the test again expects the ingest to accept every
listed wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 13, 2026 14:05
Reviewing 71bffe4, GPT-6.1 Sol found that we misstate the sources in two
comments. CenTax prints 2030 with both printings of the £640 million, and we
record the year in the kept printing's note alone. HM Treasury prints each
tax-gap measure's full name in red book Table 6.3, where we record a short
name.

Sol also found that the scan read a part of a measure from its row type
alone, so it would miss a repeat that we mark as a part by its component.
The scan now reads a part from either, apart from the ingest's _costs_part,
and test_the_scan_reads_a_part_from_its_component_alone stages a third
printing of CenTax row 90 that we mark by its component alone. We mark no
staged claim that way, so the scan finds the same 24 pairs.

In the comment on the compliance workforce, HMRC now expands its workforce
under the measure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…from test docstrings

Sol's review of f3c0722 found two mutants of the scan's part reading that
pass every scan test, one of which reads a part from its named component
alone (L17). We add a test with three cases, each a third printing of CenTax
row 90's 4%: we drop its component, name the measure's total in its place, or
drop its row type. In each case the scan must pair the printing with rows 90
and 92 and no other claim.

Sol also asked us to drop the review label at the start of the new test's
docstring (L18). We drop it there, and from the eight other test docstrings
to which this PR adds one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 13, 2026 23:59
@MaxGhenis
MaxGhenis merged commit bc64975 into main Nov 13, 2026
15 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

We merged this at 950683de. GPT-6.1 Sol approved f3c07220 in its fourth review (Subfleet job 20261113-143026-pr256-review-f3c07220) and raised two low findings. Sol graded them before we moved to P levels: we read L17, a missing test, as P2, and L18, a label on a docstring, as P3. We fix both in 950683de, so under our merge rules we treat the review as finished and need no new round.

At 950683de all 15 checks pass, and the 351 tests in tests/test_uk_ab2026_ingest.py pass locally. Each of six mutants of the scan's part reading fails at least two of them. scorecard_db/build_db.py doesn't import this ingest yet, so nothing in the live database changes until we wire it in, which we'll take up separately.

🤖 Generated with Claude Code

@MaxGhenis
MaxGhenis deleted the uk/ab2026-repeated-printings branch November 13, 2026 23:59

This branch was successfully deployed

1 active deployment
Preview — 950683de Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant