Repository navigation
Move the UK spine to geography first (#931) - #1538
Draft
juaristi22 wants to merge 29 commits into
Draft
juaristi22 wants to merge 29 commits into
juaristi22 wants to merge 29 commits into
Conversation
…ine-block machinery (#931) Per-clone engine blocks in the dense measure step, and the per-clone spine batches that follow, run policyengine-uk over part of the pool, so the formulas that read beyond their own households see the part. This adds the pieces the next commits wire in. - microcosm.frame.adapters.engine_population is country-agnostic: numpy and duck-typed simulations, no engine import. It holds the pool-share correction c_b = f_b * S_b / S with its zero rules, checks (a) and (b), and injection into a simulation, its baseline and every sibling, refused once a branch exists or a recorder is armed. A per-instance _run_formula recorder, which core's clone rebinds into later branches, and a known-periods walk feed a judge over declarations, parameter predicates and pass policies. A held value is judged unless the simulation holding it ran it after arming, and a run reaches the position window by the last year its own period covers. The verdict carries names, periods and booleans only. Household batches, the nesting check, the scatter back to one engine's order, a float32 share bound that follows numpy's pairwise-sum leaves and counts the adds forming each key, and a per-row bound for values that add share terms to others complete it. - microcosm.frame.adapters.policyengine_uk_population classifies pe-uk 2.174.1's 19 WHOLE_POPULATION_VARIABLES: three corrected shares, two stored inputs, eight zero at policy, attendance inert through private_school_vat_change, six refused ranks. It adds the window from REBALANCING_YEARS (2026 on), the constructor's structural reform factories, sha256 pins of the sources the classes rest on, the spine batch column, and uk_engine_population(), which imports pe-uk and validates only when called, failing closed. Nothing imports either module yet, so no kernel, stage or target hash moves and no output changes. Engine-free tests drive the machinery on fake core-style simulations, covering each guard condition a small change to the module could drop. Engine tests hold the declaration to the installed engine: list, adds, pins and what they cover, predicates at 2024 and 2025 and moved off current law, reform factories, window, the readers of attendance and private-school VAT, and drift lists of axis-free reductions and of whole-array branches in formulas and the engine functions they reach. They correct two synthetic clones that differ to one engine over the pool within the float32 bounds, household_tax per row, with household-local outputs bitwise equal. A build test holds the batch column and the share inputs to the build's own constants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The annual-income simulation carries each input's other spells from the dataset's year by the factor the engine projects that input by. The factor was the median of the engine's own year-on-year ratios over the rows the simulation holds. That is a statistic of whichever block ran, so per-clone engine blocks that differ would carry the same spell by different factors, and no block would match one engine over the pool. Following María's ruling of 6 Nov, the factor is now the product of one plus the engine's own rate, for each year after the dataset's, in the growth series policyengine-uk's dataset projection uses for the input: average earnings for pay, mixed income per head for profit. The rates are read from the simulation's parameters. The engine's rows must still agree with the factor within 1e-5, or the resolver refuses, as before. This moves data on today's national and dense lines. The median sat within float32 rounding of the parameter, so an annual input with other spells can move by one float32 step. The *_annual matrix entries, calibrated weights and H5 bytes can move, and other_spells_engine_factors changes in every resolver receipt that carries it. A K=1 parity check should expect *_annual inputs within one float32 step, not an identical H5. The full target and problem kernels and the national target kernels re-key with measure_simulation; no spine stage does. An engine-free test holds the factor to the parameters and to the rows' tolerance. An engine test pins the declared series to policyengine-uk's own uprating map and checks the factor on a synthetic frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he whole-population formulas it runs (#931) After the move the dense measure step's per-clone engine blocks are no longer identical copies, so policyengine-uk's share formulas (shareholding, corporate_land_value, consumption_shareholding) would read a block's denominator instead of the pool's. - The shared engine_population module gains EngineBlock, the life of one block on its own engine. It validates the declaration, refuses the position window and periods before the data, and runs checks (a) and (b). It sets each share formula to its own value times c_b = f_b * S_b / S in the simulation and its baseline, and arms the formula-run recorder. Every simulation built later gets the same arrays and recorder. At the end it checks that each still holds the block's keys and weights, then judges them all. A record pass records refusals and a raise pass raises; work after the verdict is refused. The country passes its declaration and a pass. validate_pool_shares and PopulationDeclaration.correction_reads come with it. - UKMeasureResolver takes engine_pool_shares (scratch mode only) and opens an EngineBlock with the UK declaration and the measure pass, which records and never raises. A sibling that would set a key, its inputs or the weights is refused. close_engine_block judges the block. - full_measure builds each block's pool shares from the totals the representation already records and passes them to the block's resolver. After the blocks run, exact holds only if every block validated the declaration, passed both checks, applied the correction in every simulation it built, kept its keys and weights to the end, agreed with the others on its growth and annual-income factors, and got a clean verdict. exact_basis names this basis. exact_scope says float32-level, not bitwise, and names the Universal Credit rebalancing draw the window refuses. The identical-copies checks stay as diagnostics. block_sensitivity matches the land rows' engine variables, so present_in_this_run is no longer always empty. - The two frame modules join UK_TARGET_COMPILE_MODULE_NAMES. The fake engine the shared tests use moves to test_support so the build tests can use it too. No data changes on today's lines. Identical copies take a correction of 1 and give the same matrix and local metrics, bitwise, as without it, and a single-block resolver records none of the new keys, so national receipts keep their bytes. Today's K=15 dense receipts change by design: - the representation gains exact_basis, share_correction, population_reach and annual_income_projection_factors; - exact_scope is rewritten, and the release manifest's engine_measurement limitation copies it; - block_sensitivity is rewritten; - each block's resolver receipt gains engine_pool_shares and engine_population_reach; - exact now also needs a clean guard verdict, a stricter release guard on that line. The full and national target kernels re-key; no spine stage does. On a synthetic pool of three clones that differ, three blocks match one block over the pool: household-local and annual rows and every local metric bitwise, and the land rows and each block's share formulas within the float32 bound. Uncorrected land rows miss by more than 1%, and a decile read, a 2026 period and a counterfactual on corporate wealth are each refused. Engine-free tests run EngineBlock on fake engines. They drive checks (a) and (b), a sibling that moves its keys or weights, the window and a raise pass, and hold its receipts to the representation's finalisation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…931) One simulation over the move's K=15 spine pool does not fit in memory, so the UK adapter runs a call over more than one clone clone by clone. It decides from the frame, never from a constructor flag: a household table carrying household_clone_index with more than one clone batches, and a frame without the column, or with one clone, takes the single simulation it always has. National never batches. - The shared engine_population module gains materialize_in_batches and frame_key_sums. Before any engine is built, the driver judges the call's requests: a period in the rebalancing window, a refused rank, a stored formula the frame does not supply, and attendance or its levels and carriers while it is unstored. It forms each batch's share from the frame, c_b = S_b / S at unscaled weights. Each batch, in ascending clone order, then runs an EngineBlock under the spine's raising pass: checks (a) and (b), the pool's shares set in the simulation and its baseline, the recorder armed, the requested values, the verdict. The engine is deleted and collected before the next is built. Every batch must grow the keys and the weights by one factor. People come back in the frame's order and groups by id, as one engine returns them. The driver refuses a recording pass, since its arrays carry no verdict. - PolicyEngineUKEngine passes the UK declaration and the spine pass. It gains simulation_batches(frame), parameters(period) for the stage-52 restatement, and batch_record(). Its optional clones, the spine's K once G1 adds it, refuses a frame carrying any other number of clones, so no call runs per clone or over a sub-pool. microcosm.frame.rules and the RulesEngine.materialize signature are unchanged. - The spine sidecar's rules_engine gains engine_batches only once a call batched. The engine identity is still hashed from the provenance alone, and no batch cap is added. - graph_kernels binds engine_population and policyengine_uk_population beside the adapter in all 16 engine-calling stages. A test derives those stages from their engine calls. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call takes the single simulation, unchanged. Identities move. Every spine stage kernel hashes graph_kernels.py, so every spine node re-keys, and the 16 engine stages also bind the two modules. engine_population.py is a target compile module, so the full and national target kernels re-key again. The spine graph declaration (the H2 fixture uk_spine.json), source_stages.json and the coverage manifest are unchanged. An engine test runs a synthetic three-clone frame whose clones differ, interleave by id and list people out of order, at the dataset year and projected a year on. Against one simulation over the same frame, the share formulas match within the float32 bound, household_tax within its per-row bound, and household-local values bitwise, with people and groups in one engine's order. A one-clone frame is bitwise the single simulation. A batched attendance request, a decile, a 2026 period, unstored months and a frame with fewer clones than K are each refused before any simulation is built. Engine-free tests run the driver on fake engines, one engine alive at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine values (#931) policyengine-uk's attends_private_school ranks household net income over the people a simulation holds, so once a spine's engine calls run clone by clone, one clone's simulation would rank that clone alone. Stage 52 now restates the formula's last step itself, over every person in the frame: the members' income tax and National Insurance summed per household as policyengine-core sums them, net income from the household's market income and benefits, the household's weight once per member, microdf's MicroSeries.percentile_rank (the rank the formula calls) over the people of positive weight, the attendance rates and population factor from the engine's own parameters (parameters(period), at the two paths PRIVATE_SCHOOL_ATTENDANCE_PARAMETERS names), the one-million floor, and the stored draw for under-18s. Every input is the engine's own value at the calibration year, decided by a household's own records. - Where the engine computes the frame in one simulation (national, and every spine today), the stage asks for the engine's attendance beside those inputs, stores the engine's flags, and raises if the restatement differs from them bit for bit, so each such build re-checks it on its own data. Where the adapter computes the frame clone by clone (simulation_batches above one), it asks for the inputs only and stores the restatement. An engine without simulation_batches runs one. - The evidence gains rank_route (engine_with_parity or pooled_restatement) and engine_parity_checked, a boolean: whether the engine's flags were checked. A failed check raises, so neither records a mismatch. weighted_pupils stays. The parity error names the formula, never a count. - The stage's notes in sources.yaml describe both routes. The notes in source_stages.json are re-derived from it, and the release input coverage manifest is regenerated (its source_manifest_sha256 fields). - Left for G1, which makes the clone-by-clone route live and regenerates the spine fixtures: the stage's source line ("evaluated by the pinned engine"), the notes' sentence that the stage materializes the engine's attendance, and the H2 fixture's sources/fixture.json, which keeps the stage's earlier notes until tools/graph_uk_spine_fixture.py runs. All three hold while every call takes one simulation. The store_rules_engine_input operation describes both routes: each stores attendance as an engine input from the stored draw, at the calibration year and the spine's design weights. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call takes one simulation and stores the engine's flags, as before. Identities move. Stage 52's kernel hashes private_school_attendance.py, so stage 52 and every node after it re-key. The packaged resources enter the full and national target kernels' country-spec fingerprint, so those re-key again. National's stage-52 evidence gains the two fields. The spine graph declaration (uk_spine.json) is unchanged, since stage contracts do not cover notes. An engine test runs a synthetic pool of three clones of twelve households: some alike in every clone and some not, most with several children, weights spread about one to ten, ties, a household of no weight, and each clone under the floor while the pool is over it. Household ids interleave the clones and the people are listed out of order. Each child's draw sits 3% under or over the rate the formula gives it over the pool, from the formula's own percentiles and the engine's parameters, so a rank, rate, interpolation or factor a few percent off would store other flags. A spy on microdf's percentile_rank shows the restatement hands it the formula's net incomes and weights and gets its percentiles back, bit for bit, in one simulation and clone by clone. The stage run clone by clone stores what one simulation stores, where a clone's own simulation imputes nobody; a population factor of zero, or 6% lower, breaks the parity check; below the floor nobody attends on either route, while design weights just under it impute on both, since the formula reads the engine's weights at the calibration year; and a frame of one clone stores the engine's attendance as the stage asked for it before. Engine-free tests run both routes, the parity refusal and the floor on a fake engine with a stand-in rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raph doc (#931) The dense runbook and the local pre-flight still said a per-clone engine run was exact when its blocks were identical copies scaled to the pool. Exact now means every block corrected to the pool's share denominator with a clean guard (exact_basis pool_share_injection_with_population_guard): equal to one engine over the pool to float32 rounding, not bitwise, with nothing computed in policyengine-uk's Universal Credit rebalancing years (2026 on). - docs/uk-dense-release-assembly-runbook-762.md says so where it describes --engine-blocks, the pre-flight and the engine_measurement limitation. It adds that a calibration year or dated measure's year of 2026 or later leaves every per-clone run inexact and refuses every batched spine call, and that the pre-flight does not require exact_basis, so a receipt from before the correction passes too; that is open for Max. - docs/uk-full-build-graph.md gains a section on the corrected measure blocks and their guard, on how a spine engine call over more than one clone runs clone by clone and what refuses it, and on stage 52's rank over the whole frame. - tools/preflight_uk_local_release_candidate.py: the docstring and the comment over the engine check only. The gate code is unchanged. test_uk_local_release_preflight gains two cases: a per-clone resolution whose representation has another mode, or another block count, fails however exact it says it is. - Two changelog fragments cover group (0). One describes the engine blocks, the spine batches, stage 52 and the shared modules. The other describes the annual-income factor for other spells, the one data change, which can move an annual input by one float32 step on both lines. Only comments, docs, fragments and tests change. No kernel hashes these files, so no node key, H2 fixture, source_stages.json or coverage manifest changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estatement (#931) Review follow-ups to batching spine engine calls by clone. - The spine sidecar's engine_batches is now read from the final frame, never from the calls this process ran. A resumed spine whose engine stages hit their checkpoints runs no kernel, so it under-reported batches; now it records what a fresh run does. PolicyEngineUKEngine.batch_record(frame) takes the same decision and refusals as simulation_batches. - PopulationDeclaration gains restated: stored formulas a caller may compute over the pool from household-local engine values, with the parameters each reads. Each pe-uk simulation builds and edits its own parameter tree, and parameters(period) is the adapter's fresh copy. So a new check_restated_parameters holds every batch's tree to the adapter's wherever a restatement reads it. The UK declares attends_private_school with its attendance rates and population factor, the parameters stage 52 reads. - PolicyEngineUKEngine(clones=...) accepts any whole number. A frame that drops the clone column still takes one simulation; G1's integration check rules that out after G1. - The test that finds the engine-calling stages matches a materialize call on any receiver, and any transform taking an engine. It keeps isa_subscriptions, which only checks the country, on an asserted allow-list. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call still takes one simulation and no sidecar gains engine_batches. Identities move again: every spine node re-keys through graph_kernels.py and the engine modules, and the full and national target kernels re-key through the two frame modules. The spine graph declaration (uk_spine.json), source_stages.json and the coverage manifest are unchanged. An engine test adds a frame carrying none of the keys' inputs, which gives the pool's values bitwise. A batch whose own tree moved a restated parameter is refused before it computes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The WAS donor's income, DVTotinc_bhcR8, is net regular household income before housing costs less council tax: HBAI's concept. Both WAS-trained stages matched it to the engine's household_net_income, which also takes off stamp duty, the TV licence and the business rates and corporate stamp duty that shareholding spreads over the households the engine simulates, and adds benefits in kind. At was_lisa (stage 24) those shares already read the corporate wealth was_wealth had just drawn, at the spine's weights before calibration. Both stages now read the engine's HBAI net income before the reset to zero, the concept LCFS, ETB services and NTS read (#1164), named hbai_household_net_income on donor and recipient (María's ruling, 6 Nov). - was_wealth materializes UK_HBAI_NET_INCOME_VARIABLE, and UK_WAS_ENGINE_PREDICTOR_NAMES gives it the predictor name UK_HBAI_NET_INCOME_PREDICTOR; the donor cleaning maps DVTotinc_bhcR8 to the same name. The chain's segments, their order and seeds are unchanged. - Its derive step declares the donor column in direct_maps, and the drift check refuses a declared direct map the cleaning does not read. - was_lisa takes the WAS household predictor, so the ownership logit, the balance forest and the realised profile's income tertiles read it. The profile key becomes by_hbai_household_net_income_tertile and the logit's feature log1p_hbai_household_net_income. Its declared engine reads are now what it materializes (UK_WAS_LISA_ENGINE_PREDICTORS): num_adults and num_children, from the FRS family roles since #1095, leave the list. - sources.yaml carries the declarations and both stages' notes. source_stages.json is re-derived from it, the release input coverage manifest is regenerated (its source_manifest_sha256 fields), and so is the H2 fixture: sources/fixture.json carries the two stages' declarations and notes, and stage 52's notes from 019f5d4, which waited for the tool to run; uk_spine.json carries the two stages' new contract hashes. No synthetic fact in the fixture moves. - docs/uk-lisa-1003.md and a changelog fragment describe the concept. What moves: every WAS draw can move on both lines, and with them the Lifetime ISA draws and the later stages that read wealth (means-tested capital, CGT selection, property components, high-value support). The size is unmeasured; a sampled f010 arm measures it. was_wealth, was_lisa and every later spine node re-key. Tests: the WAS recipient reads the engine's HBAI income, not a household_net_income decoy, and the donor maps DVTotinc_bhcR8 to the same name; the packaged declarations name both sides; drift to the old engine read or a wrong direct map is refused; the LISA recipient and donor carry the WAS predictor and its declared engine reads are what it materializes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 640b2cf. Its text said the WAS and Lifetime ISA stages read one income concept on both sides. The code matches the ruling, but the two sides are not the same income: policyengine-uk's HBAI net income also counts HBAI's in-kind items (free school meals, school milk and fruit, the free TV licence's value, Healthy Start vouchers, the Tax-Free Childcare top-up) and the Warm Home Discount, and the WAS user guide defines the income behind DVTotinc_bhcR8 only as total net regular household income less the council tax paid, so how it treats pension contributions, maintenance paid, student loan repayments and negative incomes is not verified. - The was_wealth and was_lisa notes, docs/uk-lisa-1003.md, the changelog fragment and the code comments call the recipient's income the engine's closest concept to the donor's and name those residuals. The was_wealth notes keep the income among the predictors that are not fully like-for-like. - household_net_income is described as adding the imputed value of NHS, education and transport spending, not "benefits in kind", which HBAI's income also counts in part. - Only the LCFS and ETB services stages name the income as a predictor; NTS reads the same engine variable to build gross income. The comments and test docstrings say so, and the two tests are renamed to match. - The changelog's What moves adds the calibrated weights on both lines. - source_stages.json is re-derived from sources.yaml (the two notes), the release input coverage manifest is regenerated (its source_manifest_sha256 fields), and the H2 fixture's sources/fixture.json carries the two notes. uk_spine.json does not move: notes are outside the stage contract hash. What moves: nothing in the data or any contract hash. Text only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#931) María's ruling of 6 Nov on the move, in line with workstream 2 of microcosm#1521 and the first must item of Vahid's review there. Two of the three ONS land rows cannot usefully bind weights: - ons.land.corporate_land_value (E43B, £2,063,534m): policyengine-uk's corporate_land_value hands out its aggregate by each household's share of the weighted corporate-sector wealth, so a released file meets that aggregate at any weights. Bound to the column measured at the starting weights, the row pinned only the weighted mass of the WAS corporate and private pension wealth, which no other row targets, not land. - ons.land.land_value (E44R, £6,872,455m) counts the land of financial corporations, government and NPISH, which the engine's land_value does not hold. With the household row (E44N, £4,485,171m) and the corporate row it asked for £323,750m that no weights could supply. Both get entries in uk/calibration_measure_exclusions.json in the obr.council_tax_england shape (tracking microcosm#1521, approved 2026-11-06, expiring 2027-02-06) and are measured on every evaluation. ons.land.household_land_value stays bound. Its binding note in uk/uk_population_targets.json said households_and_npish; E44N is the households sector alone, as the Chronicle package labels it, and the note now says so. The contract is hand-written. tools/generate_uk_target_references.py reads no binding note, so target_references.json and the membership report do not move (their regeneration test needs the pinned Chronicle feed, which is not on this machine, and skips here). What moves: the solve drops two rows on both lines, so calibrated weights move slightly; a calibrate-only rerun on the move's spine attributes it. Released corporate land value still equals the engine's aggregate. Tests: the packaged register's count and census take the two rows, and a new test pins their fields, the published values the reasons quote against target_reference_membership.json, the household row staying bound and its corrected note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…931) Follow-up to 410e822. Its test pinned the register's two entries, their fields and the published values they quote, but not what the entries do to a run. The new engine-free test compiles the three committed ONS land references with stand-in values and applies the packaged register's land entries, as both target loaders do after reconciliation. The solve surface keeps ons.land.household_land_value alone, the exclusion receipt carries the two entries' review fields, and the compiled register, which every run keeps beside the solve surface as its band-edge register (a national run writes it as national_contract_registry.json), still holds all three. On a synthetic four-household frame the national problem encoder binds the household row alone, with the receipt and the compiled register's digest in its bindings, while the compiled register materializes and scores all three rows at their weighted values. What moves: nothing. Test only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 410e822 and 161c777. The two entries hold the right rows, but some of their text contradicted itself, the record of what moves was short, and one claim in 410e822's message was wrong. - The total-economy reason, the changelog, the test docstring and the census comment now tie the £323,750m gap to the rows: E44R also counts the land of financial corporations, government and NPISH, which neither E44N nor E43B covers, and policyengine-uk's land_value is the sum of the two columns those rows read. The old text said the engine's land_value does not represent or hold financial corporations' land, while the corporate entry says policyengine-uk describes its corporate aggregate as the land of non-financial and financial corporations. That entry now says "describes as", not "labels". - The census comment no longer says the corporate row is met by construction: the column meets the engine's own aggregate at any weights, not E43B. The adjudications give Vahid's suggestion in his order (met by construction once the engine's parameter comes from the target's ONS series), and say the calibrate-only rerun is to attribute the change, since it has not run. - The two held rows' binding notes in uk/uk_population_targets.json record the hold, as the other held rows' notes do. The parity register's land concern names the two exclusions; uk_data_target_parity.json is regenerated with tools/census_uk_data_target_parity.py, and --check passes. The UK_BLOCK_SENSITIVE_MEASURE_COLUMNS comment and its test's comment no longer say the national targets bind the two variables. - 410e822's message said the target-reference regeneration test skipped because the pinned Chronicle feed is not on this machine. The feed is here. tools/generate_uk_target_references.py, run with the test's arguments on that feed (facts and manifest hashes and the row count match uk/chronicle_feed.json) and on this commit's contract, reproduces target_references.json and target_reference_membership.json byte for byte. It ran directly, through the engine slot, rather than inside pytest, whose own load of the 10 GB feed would double the memory. The local generator keeps only local-level contract rows, so the land rows' notes cannot reach the local surface. What moves, all from 410e822 (this commit is text, a regenerated parity register and tests): - Both lines split each family's equal share of the objective evenly over its bound rows (the dense line within each grain, where the land rows are amounts). ons.land.household_land_value, now the ons_land family's only bound row, carries the family's whole share: three times its loss weight before, on both lines. No other row's weight changes. The calibrate-only rerun measures this together with the two dropped rows. - The national frozen scoring register (national_target_registry.json, the approved register) loses the two rows, so a national run given a frozen register derived before 410e822 refuses, as with obr.ni_employer. The dense loader checks the reconciled register, which keeps them. - The incumbent and release-cut scores, uk_target_fit and calibration_diagnostics.json read only the rows the solve binds, so the two rows leave them. - Neither line resolves land_value or corporate_land_value any more: the dense measure receipt's national_inputs falls by two, and a per-block dense run's block_sensitivity.present_in_this_run is empty. Evaluations that score the un-excluded register (tools/evaluate_uk_incumbent_surface.py) still measure the two rows, as they do every held row; 161c777's test shows the compiled register can score them, doing the scoring itself. - The dense release lists the two exclusions among its limitations, each with the first 300 characters of its reason, which cite only published ONS figures and code facts. - The edited files are UK country-spec resources, so the national and dense target, measure and solve nodes re-key; full_measure.py feeds only those kernels. Spine nodes, source_stages.json, the release input coverage manifest and the H2 fixture do not move. The regional_home_values stage note still says the home value reaches three national rows, ons.land.land_value among them. It changes with group (c)'s rewrite of that stage, which regenerates source_stages.json and the H2 fixture anyway. Tests: test_the_corporate_and_total_land_rows_are_measured_diagnostics also pins the "4.7 percent" the total reason quotes and the two held notes. test_the_two_land_diagnostics_are_measured_but_not_fitted adds a bound row of another family (obr.tv_licence_fee) and pins the household row's family_equal weight at three times its share beside the other two land rows, with the other row's weight unchanged; its docstring says it applies the entries as the loaders do, without calling them. test_uk_measure_simulation, test_uk_full_measure, test_uk_data_target_parity, test_uk_population_targets, test_uk_incumbent_surface_evaluation, test_spec_source_citations, test_uk_target_references (its feed cases skip without CHRONICLE_UK_FACTS; the generator run above stands in for the national one) and test_uk_solve_scope pass, and ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every UK copy layer took its id offset from ten to the digit count of the largest id it saw. Once (b) clones households at the spine's start, every offset would grow with K, and the K=1 national spine would stop being the clone-0 slice of the K=15 spine. Copy ids now come from fixed decimal fields declared in uk_runtime/copy_id_table.py, so every copy id is a function of its source id and its copy path, at today's stage order. The table is UK-local (María's rulings of 7 Nov); the shared table, a declaration digest and the int32 check come with the follow-up stage. - The table, least significant first: vintage (10 ordinals at 10**8, kept for pooled FRS vintages; nothing mints it), geographic_support (100 at 10**9), spi_support_channel (3 at 10**11: the SPI copy at 1, the band donor at 2), high_value_support (6 at 10**12), cgt_support_split (10,000 at 10**13), cgt_incidence_clone (2 at 10**17) and cgt_residential_split (8 at 10**18). The largest mintable id, about 7.2e18, is checked below int64 when the module loads. Helpers: radix, require_below, mint (refuses an ordinal outside its field, a row that already carries one and an int64 overflow), ordinal and parent. - Mint sites: frs_spine refuses raw ids at or above 10**8. Stages 16 (with its _remap_ids, which had no overflow check), 17 (sources read from the source-id columns), 26, 42, 43 and 46 mint through the table. Stage 42 refuses a household needing more than 10,000 copies with a fixed message. The anchor (45) pairs each twin by the twin field's radix. The executor takes each id-offset stage's declared radix, refuses an incumbent id at or above it and recovers parents on exact int64 ids; the row walk it replaces upcast all-numeric rows to float64, inexact above 2**53. Refusals print counts, never ids. The tests-only replace path mints through the table and refuses a base whose ids reach the SPI field. - Dense: the dense expand keeps its multiplier above the spine's largest id. A table-minted spine's twins and arms leave no int64 room above them, so every dense entry point (uk.full.expand, the ladder and crosswalk clone routes, the row-wise tool's plans) refuses n_clones > 1 in one message naming #931 and the clone-first spine. Dense K>1 builds wait for (c). K=1, and a spine without copy rows, expand as before; the expand kernel no longer multiplies an empty column by a multiplier above int64. - Declarations: the id_remapping texts in sources.yaml, source_stages.json and the code dicts name the table field. stack_zero_weight_donors and stack_income_band_donor_households gain the key, required by sources.schema.json and checked against the code. copy_id_table is bound into frs_spine and every minting stage, spi_support into stages 16 and 17, rowwise_geography into 43 and 45 and cgt_support into 26. What moves: base FRS rows keep their ids. Every copy row's five id columns change: SPI copies +10**11, band donors +2*10**11, high-value copies k*10**12, CGT copies k*10**13, twins +10**17, arms j*10**18. Relative id order is unchanged at K=1, so sorts, ties, ranks and folds are unchanged. Values move on copy rows through every draw keyed on the copy's own id: stage 18's stage-2 fill, stages 19-24, 27 and 28-41, every twin's Advani-Summers prior (so the CGT gains layer re-realises), stages 48-54 and policyengine-uk's warm_home_discount_random_draw on copy households. Base rows move second-order through pooled steps: stages 26-28's selections and walks, the residential solve, the anchor, the CGT walks and calibration. Draws keyed on person_source_id do not move. National ids pass 2**53 from stage 42 on. Dense K>1 builds are refused until (c). Every node re-keys. The coverage manifest's source digests and the H2 fixture are regenerated: the fixture's declarations, and its synthetic CGT asset-type, property purchase and rental facts, which are sized to the synthetic spine's copy-row values. Left for 1b: the identity tool's decoders still read the retired rule, so 11 tests in test_uk_identity_stability_receipts fail until 1b; also size_evaluation's decoders, the remaining id-printing messages and the stochastic_assignment binding sweep. Tests: the table's fields, bounds, order and ceiling, and the helpers' round trips and refusals; each copy stage's ids against its roots, found by lineage; a K=1 chain through all six layers sorts as the retired rule did and keeps base ids; clone-first K=1 and K=3 frames nest exactly and the retired rule does not (test_support/microcosm_build/ uk_clone_first_frames.py); the anchor pairs at a one-digit CGT ordinal; the executor's refusal and parents, exact above 2**53; the dense refusal and the K=1 path; the fixed CGT message; an AST guard against the retired rule; the binding tests. H2 parity passes on the regenerated fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 22bcdbe found texts that overstate or misstate what it does, and refusals and dense paths that no test drives. This commit corrects the texts and adds the tests. The identity tool's decoders and the other items ruled into 1b stay in the next commit. - Stage 46's notes said the geography identity kernel and the identity tool read the arm rule. The kernel reads lineage columns, and the tool decodes the retired rule until 1b, so the notes now name the executor only, as the module docstring does. source_stages.json is re-derived, and the coverage manifest's source digests and the H2 fixture's notes are regenerated. - The row walk the executor replaced read a row of only integer and float columns as float64; an all-integer row stays int64. The _source_lineage docstring and the changelog now say so, and the exactness test carries a float column, on which the replaced walk fails. - The dense refusal fires wherever the dense offset would overflow int64, as a spine with residential arms at multiples of 10**18 does at K >= 2. Its docstring, its message and the changelog now say that. require_below's message no longer says that only a later stage may fill the field. - The changelog says each copy row now sits at its parent's ids plus its layer's offset, where it used to sit a power of ten above the largest id its layer saw. It also says that national ids pass 2**53 from the twin on (and at stage 42 from ordinal 901), that only the new refusals print counts, that create_uk_spi_support_tables no longer takes id_multiplier, and that copy rows join across versions by lineage, not by build ids. - The CGT split's bound check moves into a helper so its boundary can be tested. The lineage test helper reports counts, never ids or keys. What moves: no value or id. Every node re-keys again, through the text in graph_kernels.py, copy_id_table.py and rowwise_dataset.py and the helper in cgt_support.py. Tests: - Stage 16 on clone-first K=1 and K=3 frames nests exactly. - The executor takes the declared radix where the frame's maximum would give another, names each stage's own field, and refuses parentless rows by count. - The id_remapping texts quote the table's radices. - Stages 26 and 42 mint from the immediate parent on frames with SPI copies in FRS id spaces. - frs_spine (end to end), stages 42, 43 and 46 and the replace path refuse incoming ids by count. - The CGT bound holds 10,000 rows and refuses 10,001. - A drifted SPI id_remapping is refused. - The copy-id binding set is derived from imports. - The dense refusal and the K=1 path run through the pool expand, the crosswalk route, both dry-run plans and the graph's uk.full.expand, on ids above 2**53. - The twin's negative-prior count is pinned. - H2 parity passes on the regenerated fixture. The 11 tests in test_uk_identity_stability_receipts still fail until 1b, as at 22bcdbe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22bcdbe and its follow-up 2f0f8ce moved every mint site, the executor and the anchor onto the copy-id table's fixed radices. Two decoders outside the stages still used the retired rule. The identity tool read the CGT support copies and the residential arms at ten to the digit count of the largest pre-split id. The dataset-size evaluation multiplied by a dense multiplier of 10**19, which a one-clone expansion of a table-minted spine records, and overflowed int64. This commit moves the identity tool onto the table and both decoders onto exact integers. It also makes the copy path's refusals count-only, adds static guards against the retired rule, and binds the shared keyed-draw helpers into every stage that draws through them. - Identity tool: _support_copy_lineage and _residential_arm_lineage take each copy's ordinal and parent at its field's declared radix (copy_id_table.ordinal and parent). They read the household table only, as exact int64; a non-integer id column is refused. Each receipt's id_multiplier now records the declared radix (10**13 for the support split, 10**18 for the arms). Before any receipt, the tool refuses a spine whose copy ids disagree with their lineage columns under the table, as every spine built before the table does. The refusal is one count-only message saying to run the tool at the commit that built the spine. The refusal and the decode are two small functions, so a retired-rule path could replace them and nothing else. - Size evaluation: dense ids decode back to spine ids exactly, at the multiplier the expansion recorded (the new id_multiplier argument of weight_tables), else the spine's, else the clone-0 ids'. 10**19 is included. A clone offset that int64 cannot hold, a non-integer id column and a multiplier that does not fit the dataset are refused by count. A dense file that keeps its true sources computes its spine stretch again; it used to fail on int(None). - Count-only refusals: - rowwise_geography: _remap_ids' overflow (it printed the multiplier and the largest id), clone_entity_frame's and assign_household_geography's duplicate checks (up to five ids), and the crosswalk route's check for duplicate household ids before cloning; - rowwise_dataset: its uniqueness and person-link checks; - spi_support: its uniqueness and person-link checks, the donor's duplicate and missing-source checks, and the selected-household check; - spi_spine: the stage-14 retained-leaf check; - cgt_structure: the CGT carrier check; - cgt_imputation: the redraw's missing-weight check. - Guards: only the dense line's own modules name id_multiplier_for_values until (c), each with its reason: rowwise_geography, rowwise_dataset, the package re-export, size_evaluation and the row-wise tool. Every module that mints, bounds or decodes copy ids reads the table, the identity tool among them, and names no retired rule. - Bindings: every spine stage whose module or bound helpers import microcosm.build.stochastic_assignment now binds it. A derived test holds the set, and a helper-only edit to it re-keys all 36 such stages and no control. No dense node calls it. - Texts: the stage-46 notes and the residential split's docstring say the identity tool reads the arm rule too. The changelog's binding sentence names only the helpers that mint or weigh copies (spi_support, cgt_support, rowwise_geography). What moves: no data value or id. Stage keys re-key on 31 stages that newly bind stochastic_assignment: frs_spine, frs_take_up, frs_person_draws, frs_household_draws, frs_brma, frs_annual_income_spells, spi_income_band_donors, hmrc_spi_income_spine, spi_housing_shell, was_wealth, was_lisa, property_components, property_purchases, nts_bus_travel, lcfs_consumption, etb_vat, etb_services, uc_reporter_redraw, spi_benefit_coherence, uc_capital_coherence, uc_take_up, uc_legacy_closure_take_up, pension_credit_take_up, child_benefit_take_up, uc_deduction_attributes, cgt_incidence_clone, cgt_incidence_anchor, salary_sacrifice, student_loans, isa_subscriptions and childcare_copy_take_up. Every spine node re-keys anyway through graph_kernels.py, as do the dense nodes that hash rowwise_dataset.py. Regenerated: source_stages.json (the stage-46 notes line), the coverage manifest's 25 source digests and the H2 fixture's notes line in fixture.json; uk_spine.json is unchanged. Tests: the 11 identity-tool tests that failed at 22bcdbe and 2f0f8ce pass again. The hand-built split frame is minted through copy_id_table; the other fixtures come from the real stages. New tests cover: - the old-spine refusal, through main and directly; - the arm decoder above 2**53; - a support copy at ordinal 9,999, which a float read misplaces; - the size evaluation's recorded, spine and clone-0 paths at 10**19, an older dense run at its own multiplier, the int64 refusal and the source-column stretch; - the count-only messages, with no id in them; - the executor's household_source_id branch; - the two static guards and the binding test. Synthetic mutations make each guard fail. The parent's weight_tables fails the four new size tests (OverflowError, TypeError). The 114 touched engine-free test files pass. The H2 fixture regenerates byte-identically, and H2 parity and the engine graph and ladder-clone tests pass in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 50b7ea1 found no must. This commit takes their should findings and the cheap nits, and corrects four claims 50b7ea1 made. - Bindings: stage 14 (frs_hmrc_spine_leaves) draws its employer pension contributions on keyed uniforms through employer_pension_contributions, but did not bind stochastic_assignment, because the derived test read direct imports only. It now binds it, and the test follows one more step: a stage draws when its module or a bound helper imports the helpers, or imports another first-party module's function that draws through them (a fixed point over the import graph, re-exports and module attributes included). spi_support_channel, which shares spi_spine with stage 14's draw and stage 18's SPI imputation but calls neither, is the one exemption, pinned to those two names. employer_pension_contributions itself stays unbound until the wider binding sweep. - Size evaluation: the spine-stretch refusal printed no count and chained a pandas KeyError listing every missing household id; it now counts the dense rows before the lookup and raises unchained. The misfit refusal says an inferred multiplier reads household ids only and names weight_tables(id_multiplier=), the one route for a recorded multiplier. - Count-only refusals: the row-wise tool's dry-run input checks now call the build's own (rowwise_dataset._require_unique and _require_person_links) instead of listing up to five ids, and the spine sampler's three refusals in national_sampling count families. - Guards: an AST guard refuses ten raised to a len(str(...)) power outside rowwise_geography (the dense rule), copy_id_table (field spans) and national_sampling's legacy candidate decoder, so the retired rule cannot come back written out inline. - Texts, correcting 50b7ea1: - a retired-rule decode would not replace two functions and nothing else; it would also thread the person and benefit-unit tables back through the two lineage functions and their four callers, and pair the twins in the tool at the three pair_clone_households call sites (the tool's comment now says so); - only the identity tool used the retired rule; size evaluation kept the dense line's own rule and overflowed at 10**19, and its non-integer id refusal names the dtype, not a count; - no caller passes a recorded dense multiplier yet: tools/evaluate_uk_dataset_size.py infers it from household ids (the docstrings say so); - the arm tests decoded a synthetic copy chain, not stage output; the stage-built tests below cover that now. The changelog names what each SPI and CGT stage binds and why, limits the count-only sentence to the UK modules (the shared graph executor's EXPAND checks are unchanged), describes the inferred multiplier, and says the identity tool refuses a pre-table spine with copy rows. What moves: no data value or id. Stage 14's key moves with its new binding; every spine node re-keys anyway through graph_kernels.py, and the dense population nodes that hash national_sampling.py re-key, as they already do through rowwise_dataset.py in this push. No generated file changes. Tests: new size tests decode an older dense run at K=15 on odd ids above 2**53 at all three bases (a float64 subtraction fails it), and pin the uncovered-spine refusal (no id in the message, its cause or the traceback) and the negative clone_index refusal. New identity-tool tests hold the old-spine gate to each stored ordinal exactly and to an empty vintage field, refuse a support copy whose parent the split did not copy, and run the gate on frames that stages 16, 17, 26, 42, 43, 45 and 46 built (stage 44's redraw done by hand) and the E8 residential, clone-pair and support-split receipts on the last of them. The dry run, the sampler, the CGT carrier and redraw checks, the SPI uniqueness checks and the support-copy refusals now have exact count-only messages pinned. Synthetic mutations make each new guard and test fail. The 30 touched engine-free test files pass (1,331 tests). The coverage manifest's check passes, the H2 fixture regenerates byte-identically, and H2 parity and the engine graph tests pass in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CGT support split divided each selected household into ceil(w / 60)
copies of its own weight. Once (b) clones households at the spine's start,
each geographic clone carries 1/K of its source's weight, so a clone's count
would shrink with K and the K=1 spine would stop nesting in the K=15 one.
As María ruled on 7 Nov, the count now reads the geographic clone count
times the row's own weight, so every clone of a household gets the count its
source gets at K=1. Nothing moves at K=1.
- Count: cgt_support_copy_counts(weights, maximum_copy_weight, *,
clone_count) returns max(1, ceil(K x w / 60)). For K > 1 the quotient is
snapped to the nearest integer when it is within K ulps of it, so an exact
multiple of 60 keeps its count at any K (at K=7, 7 x (60m / 7) lands above
60m for 626 of m < 20,000 in a synthetic sweep). The snap gives up the
converse case: a quotient within K ulps above an integer counts one copy
fewer than at K=1. At K=1 nothing is snapped, and the count is
ceil(w / 60) bit for bit.
- Stage: split_cgt_support_households takes a required clone_count, and the
transform reads it from the frame (geographic_clone_count): one more than
the largest household_clone_index, or 1 without the column. A non-integer
column, a missing or negative index, and indices that are not exactly 0
to K - 1 are refused by count. Copies keep w / n, so each weighs at most
60 / K. The selection walk still accumulates each row's own weight, and
nothing sums geographic siblings.
- Receipt and gate: the receipt's parameters gain clone_count and nothing
else. stage_health compares K x the heaviest copy and K x the heaviest
selected household with 60, so a receipt at K > 1 no longer fails
falsely. At K=1 both checks and their messages are byte-identical, and a
receipt without a positive integer clone_count is refused. The identity
tool's recompute reads K from the pre-split table and passes it on; its
receipt is unchanged.
- Texts: copy_rule in the code dict, sources.yaml and source_stages.json
now reads 'copies = ceil(geographic clone count x household weight /
maximum_copy_weight); every copy carries weight / copies'. The stage notes
give the cap as 60 divided by the geographic clone count before the
incidence clone. The docstrings, the cap's comment and a changelog
fragment are updated to match. The mass-change reason is unchanged.
What moves: no value at K=1. On synthetic frames the pre-commit and new
code give the same results: the stage's ids, copy counts, weights (bit for
bit), mass records and frame content identity, the identity tool's receipts,
and the gate's verdicts and messages. The only difference is that stage
evidence gains clone_count. Implementation hashes move for
cgt_support_split, and for high_value_home_support, which binds cgt_support.
So node keys move from stage 26 on. The spine gates hash stage_health, and
every stage after the assembled gate waits on its verdict, so in a gated
build every node from stage 14 on re-keys. cgt_support_split's stage
contract moves with its texts. Regenerated:
- source_stages.json (copy_rule and notes, re-derived from the typed
projection);
- the coverage manifest's 25 source digests;
- the H2 fixture: the two declaration lines in fixture.json, and the
stage's contract hash in uk_spine.json.
Tests:
- K=1 counts equal ceil(w / 60) bitwise on random and edge weights: exact
multiples of 60, one ulp either side, tiny and large.
- K=3, 7 and 15 replicas count as their sources, including the K=7 exact
multiples the unsnapped rule overcounts. The snap's documented trade is
pinned.
- The clone-count refusals ({0, 2}, negative, missing, float, bool) give
counts.
- On K=3 and K=7 clone-first frames, the transform gives every clone its
source's copies, with weights nested within rtol 1e-12 and every copy at
most 60 / K.
- The packaged gate passes a K=3 stage receipt with w <= 60 < 3w, and fails
it when the count is ignored. Hand-built receipts pin the K=1 messages and
both scaled checks.
- The identity tool's K=1 receipt equals the one ceil(w / 60) gives, with no
new key. At K=3 it receipts green, and it reports a recompute that ignores
K.
Eight source mutations each fail a new test. The 51 touched engine-free
files pass (2,317 tests). The coverage manifest's check passes, the H2
fixture regenerates and passes its check, and H2 parity passes in the
engine slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 914cae4 found no must. This commit takes their should findings and the cheap nits, and corrects three claims 914cae4 made. - Identity tool at K > 1: the E8 recompute counted each family exactly, on a weight rebuilt by re-summing the family's copies. At K > 1 that sum drifts by more ulps than the count's K-ulp snap absorbs. On the commit's own nesting scenario, the tool flagged the stage's correct count of the 7,620 family in every clone at K = 11 and K = 13. At K > 1 the stored count must now lie between the counts at the folded weight times 1 - 1e-9 and 1 + 1e-9, the tolerance the tool already gives family weights. At K = 1 the check stays exact, so K = 1 receipts and problems are unchanged. The tolerance cannot tell an exact multiple's n from n + 1. The count's unit tests pin the snap, and the flags and family weights still report a moved count. copies_recomputed stays the exact recount. - Gate tolerance: test_split_gives_every_geographic_clone_its_sources_copies now runs the packaged stage gate on the transform's receipt. At K = 7 it also asserts that 7 x the heaviest copy lands above 60, so the copy check's 1e-12 tolerance stays exercised. - Texts, correcting 914cae4: - Its changelog fragment and message said every clone of a household gets its source's count. That holds for a selected clone at exactly 1/K of its source's weight. The walk still accumulates each row's own weight, so the household at which a band's walk reaches its support mass, and households tied on wealth, can be selected in some clones and not others, lowest household id first. The fragment and the split's docstrings now say so, and a new test pins the boundary case. - Its Tests line omitted that the clone-first split scenario exhausts every pool, so its selection is the same at every K. - The spine gates re-key because they hash the country spec as well as stage_health; the fragment now names both. The fragment also says the snap adds a copy at K = 7 only for some exact multiples. The module docstring, the clone-split factor's comment and the split's refusal sentence name the incidence clone stage (cgt_incidence_clone). geographic_clone_count's docstring says a refusal gives a count or the column's dtype. Not changed here: the snap rule itself and a mention of it in copy_rule, which wait for a ruling on the snap; the stage notes' "the clone stage", which changes when the stage's declarations are next regenerated; and the tool's cross-check of K against the stage receipt, which lands in (b). What moves: no value, at any K. cgt_support.py changes only in docstrings and a comment, so its implementation hash moves again, and the stages that bind it re-key as in 914cae4. sources.yaml, source_stages.json, the coverage manifest and the H2 fixture are untouched; the fixture holds stage contract hashes, not code hashes. The tool's verdicts move only at K > 1, which no spine runs yet. Tests: the identity tool's nesting receipts are green at K = 7, 11, 13 and 15; a count moved off a non-multiple is reported at K = 1, 3 and 11; at K = 1 a moved count on an exact multiple is still reported by the count check. Six synthetic mutations each fail a new test: the gate without its tolerance, the tool exact at every K, the tolerance at K = 1, the tolerance ignoring K, a tolerance of 1e-1 and a strict walk. The touched and related engine-free files pass (243 and 275 tests). The coverage manifest's check and the source_stages projection check pass, and ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) The re-reviews of da5a350 and 9721448 found no must and no code should. This commit takes their four nits. - Bindings: the stochastic_assignment exemption of spi_support_channel now names the stage's transform, and the binding test checks that the transform names no drawing function, directly or through its module, as well as the drawing names the module imports. Before, a support-channel transform calling stage 14's employer pension draw, which spi_spine already imports, would have passed both binding tests. - The row-wise tool's --source-lineage-modulus help still described the spine's retired copy-id scheme. It now says spine inputs carry copy ids in the copy-id table's fixed fields. - Test comments on the CGT nesting scenario: the 9,000 household is the first candidate for the exact-total correction, not where it always lands. At 7 and 13 clones the shared fallback puts it on the last copy of the 7,620 family's last clone. At 13 clones an exact recount moves 12 of the 13 clones, not every one. - The identity tool's E8 docstring says copies_recomputed stays the exact recount, so at K > 1 it can exceed copies by one for each exact-multiple family while the count check passes. What moves: nothing. No runtime module changes, so no node key moves. Ruff is clean, the stochastic_assignment binding tests pass, and the re-review's mutant (the exempt transform naming the employer pension draw) now fails the binding test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
13 tasks
The land changelog said the two held rows "are measured on every evaluation", and 161c777's test was named for that. That phrase is the exclusions register's convention, and the register's entries keep it, but no build step reports a held row's estimate. As María ruled on 8 Nov, the changelog and the test now say what happens: the rows stay on the run's compiled register, and a tool that scores the un-excluded register, such as tools/evaluate_uk_incumbent_surface.py, measures them, as it does every held row. - The changelog fragment's sentence is reworded. - test_the_two_land_diagnostics_are_measured_but_not_fitted is renamed test_the_two_land_diagnostics_leave_the_solve_but_stay_on_the_compiled_register, and its docstring and the scoring comment say the same. What moves: nothing. Ruff is clean and the renamed test passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As María ruled on 8 Nov, national_sampling's compact sampler, sample_uk_national_frame, is retired with its family decoder, uk_source_family_units. No build or tool has called the sampler since 8adc836 (#762 PR A) moved the row-wise candidate onto the spine sampler. The decoder re-derived retired id arithmetic (a clone multiplier of ten to the digit count of the largest canonical id, and max + 1 SPI and capital-gains offsets), which kept the module on the copy-id tests' inline-rule allowlist. - national_sampling.py loses the two functions, their __all__ entries and the helpers only they used: the post-sample SPI replacement quota check, the required-column list and the clone-index column alias. The module docstring now describes the spine sampler. The spine sampler, the rung tokens, the default seed and every name other modules read from the module are unchanged. - test_uk_national_sampling.py drops the ten tests of the retired functions and their compact-frame builder. The numeric-region refusal, which only those tests covered, is now tested on the spine sampler. - test_uk_copy_id_table.py: national_sampling leaves _INLINE_DIGIT_COUNT_MODULES, so the inline-rule guard now admits only rowwise_geography and copy_id_table. - experiments/901-uk-main-rebase-receipts.md cited one of the removed tests in its row 4; the row gains a dated note. - A removed changelog fragment. What moves: no value. _PopulationKernel hashes national_sampling, so uk.full.sample, uk.full.normalize, uk.full.expand, uk.full.locations, uk.full.geography_mapping, uk.full.pool and uk.full.geography_gate re-key, and with them every node downstream of uk.full.sample. No other UK kernel binds the module, so no spine node re-keys. No generated file moves: the source_stages projection and coverage manifest checks pass, and the H2 fixture holds no code hashes. Tests: test_uk_national_sampling, test_uk_copy_id_table and test_uk_runtime_exports pass (57 tests). An AST scan finds no tracked Python file that names the two functions, and the inline-rule detector flags the module at the parent commit but not here. A permissive region check fails the new spine test. Ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll it (#931) A spine stage's key hashes its stage module and the helper modules listed for it, and source_hash does not follow imports, so a stage that called a function in an unlisted module replayed a stale output when that module changed. As María ruled on 8 Nov, this sweep binds those helpers, with a test that derives them. It only changes cache keys. - test_uk_graph gains the derived rule. From the transform graph_kernels builds for each stage, it follows every definition the code names (annotations included, through re-exports and module attributes) into the module defining it, within microcosm.build, to a fixed point, and requires the stage's hash to cover each module it meets. A check that returns and writes nothing, a function returning one column name, a record class of undefaulted fields, a constant of strings and the stage declaration need no binding; a numeric constant does. The other workspace packages stay outside the rule, and the docstring says why. A second test applies the rule to the seven dense kernels that share _PopulationKernel's hash, with the spine gate admission check listed as refusal-only, and a control test pins each form the rule follows and each kind it skips. - graph_kernels: national_frame joins the modules every stage hashes, and the helper table gains 219 bindings of 49 modules in 48 stages, with two shared groups (the vendored-fact reader with the chronicle feed and its schema epochs; frs_spine's pinned tab reader with trace). The six modules the ruling named are bound where stages 26, 42, 43 and 46 compute through them: property_localisation (26), spi_support (26, 42, 43, 46), cgt_imputation (42, 46), cgt_asset_type (46), advani_summers (43) and cgt_structure (42; stages 26 and 46 reach only its closed-world operations check). Stage 14 binds employer_pension_contributions. - The older import-level tests then ask for more: stochastic_assignment in 8 more stages (the support channel's exemption goes), the copy-id table in stages 14 and 18, 15 stages on the copy-id name-only list and four more counting stages. Their re-key tests get new controls and counts. - graph_population: _PopulationKernel hashes rowwise_geography, and the dense test also requires frame_sampling, national_frame, local_authority_input and gates. - Changelog: a new helper-bindings entry; the copy-id entry drops its sentences about the helpers that stayed unbound. What moves: node keys only. Every spine kernel hashes graph_kernels.py, so every spine node re-keys, and with it every node downstream. Every graph_population kernel re-keys (uk.full.sample, normalize, expand, locations, geography_mapping, identity, property.value, property.band, geography.local_authority, geography.brma, geography.rfdr, private_rent, pool and geography_gate), and so do the calibration and size kernels that hash the module (uk.full.dense, selected, calibrated, size_search, size_draw, size_refit and size_checkpoint_import). No value, declaration or generated file moves: uk_spine.json and the H2 fixture hold stage contract hashes, not code hashes, and the fixture check regenerates byte for byte; the source_stages projection and the coverage manifest are current. Tests: test_uk_graph (33 tests) and 15 related engine-free files pass, and ruff is clean. Mutations under scratch: a value helper newly imported into a stage (by name, by module attribute or through a re-export), a numeric constant newly read and a dropped binding (a helper, a group member, national_frame) each fail the stage test, while a helper the transform never names, a check and a constant of strings pass. Dropping rowwise_geography or frame_sampling from the dense hash, or a dense kernel newly naming an unhashed helper, fails the dense test. The H2 fixture check and the H2 parity test pass through the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erance (#931) This was to be the commit that keeps stage 42's exact-total correction off the support copies, as María ruled on 8 Nov: try the other unselected households, then the selected households left whole, never a root or copy. On synthetic frames that rule cannot always make the total exact. In the CGT nesting scenario at K = 7 and 13, every household the split leaves whole sits in the first half of NumPy's pairwise sum. That half lies in the total's binade and the second half carries exactly half an ulp, so round-half-to-even skips the target: no change to the whole households can reach it, and only a copy in the second half can. The rule as written refuses those frames, and 1 to 3 of 24 random 150-household frames at K = 1, 3, 7, 11 and 13. So the correction is unchanged here and goes back to María. This commit lands the two rulings that do not depend on it. - copy_rule names the K-ulp snap ruled on 7 Nov, identically in the code dictionary, sources.yaml and source_stages.json: "copies = ceil(geographic clone count x household weight / maximum_copy_weight), the quotient snapped to the nearest integer within clone-count ulps when the clone count exceeds one; every copy carries weight / copies". The stage notes say "the incidence clone stage (cgt_incidence_clone)" where they meant it. - The identity tool's _e8_support_split holds the stored count between the counts at the folded weight times 1 - 1e-9 and 1 + 1e-9 at K = 1 as well. At K = 1 the fold can land ulps from w: for a weight a few ulps above a multiple of 60 (about half of them in a synthetic sweep), or where the correction fell on a copy. On an exact multiple a moved count is then left to the flag and family-weight checks. The docstring, the receipt's tolerance policy and the test class docstring say so. - The split's docstring and _exact_support_weights' no longer claim the correction always keeps every copy at w / n. - test_at_one_clone_the_count_check_stays_exact becomes test_a_count_moved_on_an_exact_multiple_is_left_to_the_flags, at K = 1 and 3. A new test folds the 900 family just above 900 at K = 1: green, with the exact recount one copy higher. - The changelog fragment describes the snap clause, the K = 1 tolerance and the unchanged correction. What moves: no stage value at any K. The identity tool at K = 1: of 25 synthetic frames as built, the 2 it flagged are green now, both false positives; every tampered frame still fails, but a count moved on an exact multiple is reported by the flags and family weights, not the count check. Its K > 1 verdicts are unchanged. Re-keys: the implementation hashes of cgt_support_split and of high_value_home_support, which binds cgt_support; cgt_support_split's stage contract digest, with its texts; and every node keyed on the UK solve fingerprint, which covers sources.yaml, source_stages.json and the coverage manifest, the spine gates among them, so in a gated build every spine node from stage 14 on re-keys. Regenerated: source_stages.json (copy_rule and notes), the coverage manifest (25 source digests) and the H2 fixture (the two declaration lines in fixture.json and the stage's contract hash in uk_spine.json). Tests: 63 touched engine-free files pass (2,745 tests). The new K = 1 test fails against the parent's tool. The source_stages projection, the coverage manifest and the H2 fixture checks pass, and H2 parity passes in the engine slot. ci_test_plan verify and ruff are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rating left it (#931) The 1b decoders review found two identity-tool gaps on the synthetic H2 oracle frame. As María ruled on 8 Nov, they land as a tool-only commit before the arms. Both reproduce with origin/main's code and with the parent's. - E5's stored-column check re-applies the regional uprating (stage 25) to the stored values as a fixed point. regional_home_values (stage 27) then rewrites main_residence_value and property_wealth, so E5 was comparing stage 27's values. On the oracle, every mismatching row is one that stage 27 rewrote: 284 of 394 scoped rows on main, 294 of 404 at the parent. Stage 27 keeps the value it replaced in main_residence_value_rank_basis and moves the wealth by the value's change. The scoping now reverses that rewrite for every stage before 27. With the reversal, the mismatches drop to zero on both frames, so the stages are right and the tool was reading the wrong stage's output. - _STACKING_STAGE_BY_FLAG had no entry for household_is_high_value_support_copy (stage 26). The was_wealth scope kept the copies (105 of 394 rows on main, 115 of 404 at the parent), and E4's FRS-only frame kept any copy of an FRS household. The flag joins the map. Dropping the layer folds the copies onto their roots by family, after the CGT folds: each copy adds its w/n back, so the root returns to its pre-split weight, to rounding. A new decoder reads the copies at the copy-id table's high_value_support field. It refuses, by count, a copy without a root, a flag or index that disagrees with the id, and a family whose size disagrees with high_value_support_copies. Dropping the copies while a CGT layer stays in scope is refused. On the oracle every high-value root is an SPI row, so the copies never reach the uprating's FRS-base mean there. The new tests use FRS roots, where they do. What moves: tool verdicts only. On the oracle, through main() and the H5 loader, E5 fails with the parent's tool and passes now. E4 reads the same frame as before. The E6, E7 and E9 receipts are identical, and E8 stops at the same pre-existing error on the fixture with both tools. On a spine whose high-value roots include FRS households, E4's scope and weights change too. Nothing hashes the tool, so no stage, value, node key or generated file moves. The source_stages projection and coverage manifest checks pass unchanged. Tests: 12 new tests on frames the real stages build (stages 25, 26 and 27, then the CGT support split and the clone): - the was_wealth and FRS-only scopes drop every copy and give each root its pre-split weight; - E5 is green on that chain, red with either half of the old scope, and red when an uprated value is moved off its fixed point before stage 26; - the tool's entry point passes on that chain and fails on the moved value; - the reversal restores stage 26's values; - the lineage refusals print counts only. 9 of the 12 fail against the parent's tool. The 3 that pass are the mutation cases, which are red with both tools. The 9 touched engine-free files pass (218 tests). Ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts (#931) Three reviews of 2afda31, fe15413, e279b69 and bd292eb. Their one must, the stage-42 correction rule, needs a ruling and is left out with the other ruling items listed below. Bindings (fe15413): - The rule took any constant made only of strings as unable to change a value. A tuple, set or mapping of strings selects rows, columns or codes, so it is now a value, like a number. Only a single string, a function returning one column name and a record whose field types are of these kinds still stop the rule. Newly bound: tenure_constants in was_wealth, was_lisa, high_value_home_support and regional_home_values; council_tax_constants in frs_spine; rowwise_geography in lcfs_consumption; etb_vat in etb_services, which joins the counting stages through it; cgt_imputation in cgt_incidence_clone; frs_annual_income and hmrc_source_contract in hmrc_spi_income_spine; household_income in was_wealth and was_lisa. The dense population hash adds spi_support. - Where the rule stops at a name, a new test requires the name's module to be hashed by the stage or by a node upstream of it, since node keys chain on their predecessors', or the name to be listed with a reason. One stage name is listed (the geographic clone index columns, which no spine node writes, checked) and two dense ones (the LAD23 lookup's pin). - The derived check kind goes: a check is skipped only when listed with its reason, now also cgt_structure's closed-world check and spi_support's entity check. _writes_nothing, which tests each entry, also counts an augmented assignment, out=, copyto, ufunc.at, shuffle, reverse and the like as writes, and lets only a one-parameter function return an argument. A record's field types must be harmless too. - The spine root's rung sampler moves from spine_build, which no node hashes, to national_sampling (UKSampledSpineRootTransform and sample_uk_spine_root, behaviour unchanged). It declares national_sampling and frame_sampling through graph_implementation_dependencies, so the production root node hashes them. A test holds every transform spine_build installs to its stage's module, declared dependencies or a listed adapter. - The scope comment and changelog said microcosm.fit's fitters are bound by hand; three are, in two stages. They now name what stays unbound outside microcosm.build and the dense kernels the rule is not yet applied to, and the changelog names stage 18's employer pension binding. Fallback (e279b69), texts only: the tool's docstring, comment and tolerance policy, the receipts test class and the changelog say that within 1e-9 of a multiple of 60 the count check accepts n or n + 1, so a family split consistently into one copy more goes unreported at every K; a new test pins it. The changelog's what-moves names the tolerance policy and the country- and solve-fingerprint kernels that move, and puts the correction row about an ulp of the frame's total off w / n. Stage 26's docstring names the shared fallback as stage 42's does. Tool (bd292eb): the E5 reversal refuses, by count, a roster where a stage after regional_home_values declares main_residence_value or property_wealth. A test holds the stacking map to the structural stages' declared flags, with two listed exceptions. Sampler (2afda31): the seed comment drops the retired draws, and a test pins uk.full.sample's f100 branch, which experiments/901's row 4 now cites. Left for María: the stage-42 correction rule (on the synthetic nesting scenario no whole household can carry it at K = 7 and 13), an exact K = 1 count check, binding the unbound modules outside microcosm.build, binding cgt_structure in stages 26 and 46 instead of listing its check, and copy_rule's wording, which waits for the correction ruling. For the PR body: 2afda31 adds a changelog fragment of type removed; fe15413 binds employer_pension_contributions in stages 14 and 18; e279b69 also re-keys the spine, full, preflight and holdout gates and the target compilation, target selection, problem, measures, national targets, national problem and national gate kernels. What moves: node keys only. graph_kernels.py changes, so every spine node re-keys; the 12 new bindings would re-key their 9 stages by themselves. national_sampling.py and graph_population.py change, so every graph_population kernel and the calibration and size kernels that hash it re-key. The production root node hashes the rung sampler at every rung. No value, declaration or generated file moves: the source_stages projection, coverage manifest and H2 fixture checks pass, and H2 parity passes. The identity tool's receipts change only in the tolerance policy text. Tests: 34 touched and related engine-free files pass (1,115 tests), and ruff is clean on the 11 edited Python files. In-memory mutations under scratch turn the new tests red: a dropped tenure_constants or cgt_imputation binding; a stage newly reading a collection of strings, calling an np.add.at helper in an unbound module or reading an unhashed single string; an emptied or padded names table; spi_support or rowwise_geography dropped from the dense hash; the rung sampler declaring nothing or only its own module; an unlisted source adapter; the stage-26 flag dropped from the stacking map; uk.full.sample sampling at f100; and the tool before the K = 1 tolerance on the n + 1 pin. The source_stages projection and coverage manifest checks pass, the H2 fixture regenerates byte for byte and H2 parity passes in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 16 copied exactly round(10,000 x the sample fraction) households,
picked by row position with rng.choice, so whether a household was copied
depended on every other row. Once (b) clones households at the spine's
start, the K=1 spine's copies would not be clone 0's copies at K=15. As
María ruled on 7 Nov, the stage now copies a keyed Bernoulli share whose
probability it computes itself.
- Share: the transform passes expected = count x sample fraction, so the
manifest's 10,000 is the share's numerator. create_uk_spi_support_tables
takes expected_spi_households in place of spi_household_count (None
still copies every household). It counts N, the distinct
source_household_id values entering the stage, and copies a household
when pi = min(1, expected / N) is one or the household's uniform falls
below pi. The uniform is keyed on the household's own household_id,
under the stage's seed (42) and the new salt
spi_support_channel:inclusion. The decision is spi_share_inclusion,
which takes pi explicitly; at pi = 1 it reads no uniform, since an
integer over 2**64 can round to one. Copies keep frame order. The
positional draw and the count-versus-rows refusal are gone. The
tests-only replace path keeps its own draw.
- Gate: e7_spi_synthetic_preclone is renamed e7_spi_synthetic_share. It
declares maximum_zero_weight_rows_rule {rule: binomial_share_band,
sigma: 4} in place of the integer. After the draw, the stage resolves the
rule to min(n, ceil(n x (pi + band(pi, n)))) over the n household rows
entering it, then runs the unchanged zero-weight gate.
stage_health.latent_attribute_tolerance moved into stochastic_assignment
as binomial_share_tolerance(*, target, rows, sigma). The stage_health
name stays as an alias at four sigma, with unchanged values.
_support_stage_parameters checks the draw, salt, gate name, selector,
rule and reason, and returns the declaration with its rule. Its
hmrc_source_contract caller needs no change.
- Per-clone strata (ruling 2): where household_clone_index is present,
_allocate_spi_prior_mass adds a derived clone stratum, which is never
written to the frame. The manifest declares it as
geographic_clone_stratum, and spi_spine checks it. The empty-stratum
refusal now counts strata and names none. Without the column, as on
every national spine today, the allocation is unchanged bit for bit.
- Manifest and schema: the stage has a new source sentence, draw text,
salt, gate reason and notes. In sources.schema.json,
stack_zero_weight_donors requires salt, the allocation accepts
geographic_clone_stratum, and a zero-weight declaration takes the
integer maximum or the rule (oneOf).
- Receipt: new keys expected_spi_households, inclusion_probability
(receipt_counts.floored_share), source_households (rounded_count), and
the gate's name, rule and sigma, not its resolved maximum.
spi_households stays exact, because the stage gate reads it.
- Bindings: the receipt's counting sits in spi_spine. That adds one
counting stage (frs_hmrc_spine_leaves), against three for spi_support
(cgt_support_split, cgt_incidence_clone, cgt_residential_split), which
ten stages bind. Node re-keys are the same either way: the root already
binds receipt_counts. high_value_home_support and cgt_residential_split
now bind stochastic_assignment, as they bind spi_support.
What moves: at K=1 the SPI copy set is redrawn. Its count is Binomial(N,
pi) around 10,000 x f instead of exact: about 1,000 at f010, by design
arithmetic. Per-stratum SPI weights move with it, and so do every
SPI-channel value from stage 18 on, the high-value and CGT selections that
rank SPI rows, and calibration. The claim that stage 17's seating does not
move holds on synthetic frames. On 8 frames run through the parent's stage
16 and this one, stage 17 seated the same households, carriers and bands,
with equal seating scales, mass scales and funding factors. Stage 16's
base weights were bitwise equal except for one row in one frame (relative
7e-14, the exact-total correction). The H2 fixture's 135 households at
f = 135/10,000 give pi = 1 (10,000 x 0.0135 is 135.0 in float), so its
copies are unchanged. Every node re-keys, since graph_kernels,
stochastic_assignment and spi_support changed and the root binds
stochastic_assignment; so does every kernel keyed on the spec or the
coverage manifest. Regenerated: source_stages.json, the coverage
manifest's 25 source digests, the SPI declarations in the H2 fixture
(fixture.json), and the stage's contract hash (uk_spine.json).
Tests:
- With pi held fixed, permuting rows or appending other households leaves
every household's draw unchanged. A row permutation leaves the stage's
selection unchanged.
- Over 200 seeds the count stays inside the 4-sigma band.
- pi = 1 copies every household, even with uniforms patched to 1.0.
- The resolved gate passes at its maximum and refuses one more row. The
stage gate refuses an injected copy above the bound, and an unmatched
zero-weight base row still fails.
- Malformed rules, and drift in the draw, salt, rule, sigma, name or clone
stratum, are refused.
- On clone-first frames at K=3 and 7, N is the same as at K=1, and clone
0's copies are exactly K=1's. The other clones draw their own, and K x
clone-0 weights equal K=1's to rtol 1e-12.
- Each clone keeps its own region masses. An empty clone stratum is
refused by count, and so are a non-integer, missing or negative clone
index. A one-clone index allocates bit for bit as no index.
- The band-donor fixture moves to expected_spi_households=10, and a
comment records that seed 42 keeps copies in both regions at 60 and 200
households. Its weight test now takes the median over the FRS base the
seats are drawn from: over every row, the redrawn copies moved the
median and the statistic fell just under its 0.53 bar, while the
seating itself is unchanged.
- Ten in-memory mutations each fail a new test: keying on the source id,
a row-count denominator, no pi = 1 short-circuit, no clone stratum, the
positional draw, a gate without its band, a three-sigma alias, an exact
share or source count in the receipt, and another salt.
The touched engine-free files pass, with ruff. The coverage manifest is
current. The H2 fixture regenerates and passes its check, and H2 parity
passes in the engine slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 17 seated its donors in one walk over every candidate, and sized the seats on the eligible households counted in rows. Once (b) clones households at the spine's start, the walk would pool the clones, and the seats and the 15% household-share cap would grow with K, so clone 0 of a K=15 spine would not seat the K=1 donors. As María ruled on 7 Nov, the stage now runs today's walk inside each geographic clone, with one common mass scale. - Seating: where household_clone_index is present, the candidates are grouped by their household's clone; without it there is one group. The eligible households are counted in distinct source_household_id values, so the seats and the cap (spi_income_band_donor_seats) are the same at every K. Today's walk runs once per clone with those seats and the same person-keyed uniforms, Efraimidis-Spirakis keys and lexsort, so clone 0, whose ids are the one-clone ids, seats exactly the one-clone donors. The per-band eligibility check runs per clone and refuses by count. selected_household and carrier_person stay shared, since ids differ across clones. A non-integer, missing or negative clone index, or a gap in 0 to K - 1, is refused by count. - Weights and funding: each donor weighs (T_b / count_b) / K. Its funding stratum is its region within its clone: _funding_strata appends the clone where the column is present, and the manifest's funding_strata stays ['region']. So every clone's regions conserve their own mass. One mass scale, the minimum over every clone-region stratum, serves every clone (ruling 2), so each band keeps one donor weight, and stage_health's band-mass reconciliation and the identity tool's E8 replay hold unchanged. The exact-total correction still moves one incumbent row and never a donor, so a band's donors stay equal bit for bit at any K. With exact replicas it lands on clone 0, whose weights then nest to rtol 1e-12, not bitwise. The MassChangeRecord reason is unchanged, so uk_frame_content_identity and the coverage manifest's required_mass_change_reason do not move. - Receipt: new key clone_count, and funding_strata lists household_clone_index where the column is present (the stratum labels read region|clone). expected_donor_households, expected_donor_count and the donor counts cover every clone; donor_weight and planned_donor_weight are one clone's. At the receipts' floor of three, as the design review asked, both eligible counts go through rounded_count over distinct true sources in place of the exact ones; seating_scale reads 'full' or 'capped', since an exact scale below one gives back the eligible count; a band's carrier_mean_age reads '<3' below three carrier source households; and carrier_region_counts and the funding rows' donor_households are rounded. Besides the tests, stage_health reads seating_scale and the identity tool copies it; nothing else reads the changed fields. - Gate: the band-donor check refuses a receipt without a positive integer clone_count. It requires donor weight x K within the maximum, donors that split evenly across the clones, and at full seating donors equal to the receipt's plan (K x one clone's) and at least K x 120. The band-mass check is unchanged. At K = 1 the checks and messages are as they were, except that the seating scale is read as 'full' or 'capped' rather than a number in (0, 1]. - Texts: the draw, weight_rule, funding, seating_scale and mass_scale constants and the stage dictionary, sources.yaml (and the stage's notes) and source_stages.json. The design's draw wording named "geographic_expand", which nothing in the code defines, so the draw names the node that writes the clone index, uk.full.expand. - Bindings: the derived tests in test_uk_graph.py pass with no change. The stage already counts by lineage (it is in _COUNTING_STAGES) and binds spi_support and stochastic_assignment, and the clone column's name is a string constant. What moves: no value at K = 1. On 48 synthetic stage-16-then-17 runs (8 frames, 3 band configurations, 2 seeds; two of the frames carry a one-clone index), the seating, carriers, every household weight (hex), donor weights, mass and seating scales, funding factors, mass log and content identity after the stage equal the parent commit's. The receipt gains clone_count and takes the floor-3 forms above. spi_income_band_donors and every node after it re-key, the spine gates included, as does any kernel keyed on the spec or the coverage manifest. Regenerated: source_stages.json, the coverage manifest's 25 source digests, the stage's declaration in the H2 fixture (fixture.json) and its contract hash (uk_spine.json). Tests: - K = 1: the existing seating, carrier, weight and receipt expectations hold, with seating_scale now 'full' or 'capped'. A pinned test checks uk_frame_content_identity after stage 17 on three synthetic frames (full seating, capped seating, scaled mass) against the parent commit's values. A one-clone index seats, weighs and funds bit for bit as no index. - K = 3 and 7, with clone-first frames run through stage 16: clone 0 seats the K = 1 households, bands and carriers; every clone seats the same seats, and the other clones seat their own. K x clone-0 weights, donors included, equal K = 1's at rtol 1e-12, and so do clone 0's funding factors. - A capped seating scale is the same at K = 1 and K = 3. Each (clone, region) conserves its mass. Where one clone's region binds, its factor is 0.5, the one scale serves every clone and band, and the gate passes the scaled receipt. - Refusals: a clone without candidates, and a float, missing, negative or gapped clone index, each by count. A row permutation at K = 3 leaves the seating unchanged. - Receipt: the eligible counts are rounded, checked against an independent recount; the funding and region counts are rounded; the key sets are pinned. The mean age reads '<3' at K = 1 and where three clones' carriers share one source household. - Gate: the K = 3 receipt passes, a dropped clone_count raises, and a misread one fails. Synthetic K = 3 evidence passes and fails on weight x K, uneven splits and the scaled minimum, with the K = 1 messages pinned. - The identity tool's E8 replays a K = 3 seating with no problems and flags a reweighted clone-2 donor. - 17 mutations on scratch copies of the source each fail a test. The touched engine-free files pass, with ruff. source_stages.json and the coverage manifest are current. The H2 fixture regenerates and passes its check, and H2 parity passes in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 50b6e62 (stage 16's keyed share) and cd43acf (stage 17's per-clone seating) found no must. This takes their should findings and the cheap nits, all inert at K = 1. The ruling items, and the text changes the reviews placed at the next regeneration, are left out and listed below. Band donors (cd43acf): - Each band's rounded eligible count is clone 0's, in distinct source households. Pooled over the clones it was a union, which grew with K in every band below the top. Clone 0 walks what one clone walks, so the count is now the same at every K. The per-clone refusal is unchanged. - A band's carrier_mean_age is also withheld where the band seats fewer than three households per clone. At K = 1 a band's carriers rest on at most its seats in source households, so no K = 1 receipt changes. - The result docstring and the receipt comment name the fields that are rounded, and say that the band rows' donor and carrier counts and donor_count stay exact row counts, because the gate reads them. The module docstring says the seats are sized on eligible households counted in source households, and when clone 0's weights and funding factors nest. - Gate: a seating_scale in neither form is named by its type, or as missing, in the failure, the full-scale message and the details, so a replayed old receipt's exact scale does not reach the verdict. Share (50b6e62): the spine gate kernel hashes stochastic_assignment beside stage_health, since the latent-attribute checks' band moved there. Changelog fragments: stage 16's says what stays bit for bit without the clone column (the strata, given the same copies), that stage 17's funding factors and mass scale move by rounding under the redraw, and that the empty-stratum refusal reads the new draw. Stage 17's drops the clone-0 premise for the exact-total correction and states the nesting condition, the per-band counts, the mean-age rule and the gate message. Corrections to the earlier messages, which stay as they are: - 50b6e62: under the redraw, stage 17 keeps its seating, carriers, bands and seating scale, but its mass scale and funding factors are not equal. The funding strata's masses sum the redrawn copies' weights, so the factors, and the mass scale where it binds, move in their last bits, and the incumbent and donor weights with them (the reviews' synthetic probes: within about 1e-11 relative). The empty-stratum refusal reads the new draw, so a frame that built before can refuse, and one that refused can build; the reviews found both on synthetic frames. - cd43acf: the seating nests exactly at any K, but clone 0's weights and funding factors nest only while the common mass scale is the one K = 1 takes (one at both, or bound by a clone-0 region). The synthetic tests hold it at one. After stage 16 the clones are not exact replicas, and in the reviews' synthetic probes the exact-total correction often landed outside clone 0, at stage 16 too. - Both: rtol 1e-12 is the small synthetic frames' figure. A corrected row misses by a few ulps of the total, so its relative miss grows about as rows x K x 2.2e-16 (design arithmetic). What moves: no value at K = 1. On six synthetic frame and plan configurations at K = 1, the stage-17 evidence and the frame content identity equal cd43acf's; at K = 3 and 4 only carrier_mean_age differs, withheld in more bands. Every UK stage kernel hashes graph_evidence, so every spine node re-keys (54 of the spine graph's 79 kernel hashes change, the root's among them; 50b6e62 had re-keyed every node already), as do the dense population kernels and the spine gates. No declaration changes: source_stages.json and the coverage manifest are current, and the H2 fixture is untouched. Tests: - K = 3: the carriers per region are recounted in source households (SOUTH_EAST reads '<3' over three or more rows). With the rounding patched out, each band's eligible count equals clone 0's recount and is below the pooled one; the rounded counts equal K = 1's. - K = 3 and 4: the one-seat bands withhold the mean age, though their carriers come from three or more source households. - The K = 1 per-band refusal message is pinned. - The share gate resolves a sigma-2 rule below sigma 4, and refuses a NaN or infinite sigma. A sample fraction whose 10,000 x f is not whole keeps the fraction in the expected count and the share. - The gate names a float or a missing seating scale by type. The gate kernel's hash includes stochastic_assignment. - Ten mutations on scratch copies of the source each fail the intended test: the six the reviews found missed, a pooled per-band count, and one each for the mean-age rule, the gate message and the gate hash. Left out: the draw text's "(one before uk.full.expand)", which waits for María; clone_count in the stage-16 receipt, with (b); and the declaration texts the reviews put at the next stage-16 and stage-17 text regeneration, so that H2 is not regenerated for text alone (the weight rule's band donor count, the stage-17 notes' seats counted in source households, the stage-16 notes' draw on the id alone). Disclosure-control items from the reviews went to María separately. The touched engine-free files pass, with ruff. source_stages.json and the coverage manifest are current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #931. This is the output-moving half of the geography-first UK spine, stacked on #1519, which changes no outputs.
The spine clones every FRS household K times at its start: K=15 on the dense line, K=1 on national. Each clone gets its own small area and its own later draws (wealth, consumption, SPI copies and so on). The K=1 national spine nests in the K=15 dense one wherever draws are keyed: it is the clone-0 slice.
This is a draft, opened early so the commits can be followed as they land. The base is
uk-931-geography-first-prep(#1519). CI runs only on PRs to main, so this PR gets none until #1519 merges.Commit groups
frs_brmafold, the SPI-copy area draw, G7, G8 inside the WAS chain, gates, dense re-basingGroup (0)
After the move, the dense build's per-clone engine blocks are no longer identical copies. policyengine-uk's
shareholding,corporate_land_valueandconsumption_shareholdingdivide by a weighted total over the simulated households, so a block would read its own total instead of the pool's (PolicyEngine/policyengine-uk#2896 asks pe-uk to fix the cause).microcosm.frame.adapters.engine_population, and policyengine-uk's declaration of its 19 whole-population formulas. The shared module does the pool-share correction, a recorder of formula runs, and a guard that judges them. Nothing imports it yet.exactnow certifies corrected blocks plus a clean guard (exact_basis), equal to one pooled engine to float32 rounding. Today's identical copies take a correction of 1 and give the same matrix and metrics, bitwise; national runs one block and records none of the new keys.household_clone_indexcarries more than one clone. National never batches, and today no spine frame carries the column, so every call still takes one simulation.engine_batchesonly when the final frame carries more than one clone. A follow-up commit reads that record from the frame, so a resumed spine reports what a fresh run does, and holds each batch's parameter tree to the one the stage-52 restatement reads.rank_route,engine_parity).Data changes in group (0): only the annual-income factor's float32 step, kept by ruling on 7 Nov. Everything else is inert until G1.
What moves without being an output:
measure_simulationand the two frame modules);The spine graph declaration (
uk_spine.json),source_stages.jsonand the coverage manifest are unchanged.For Max:
engine_populationmodule (pool shares, the recorder,judge,EngineBlock,materialize_in_batches,frame_key_sums);exactand whether the preflight, assembler or microcosm-data mirror should also readexact_basis;WAS and land
was_wealthandwas_lisa, match the donor's household income to the engine's HBAI net income before negative incomes are reset to zero, under one predictor name,hbai_household_net_income. The donor's is WASDVTotinc_bhcR8, net regular income before housing costs less council tax. The LCFS and ETB services stages already read the engine's concept (UK: move the LCFS stage to the Living Costs and Food Survey 2024-25 #1164). Before, both stages readhousehold_net_income, which also takes off stamp duty, the TV licence, and the business rates and corporate stamp duty thatshareholdingspreads over the simulated households.ons.land.corporate_land_value(E43B) andons.land.land_value(E44R) out of the solve as measured diagnostics, inuk/calibration_measure_exclusions.json, tracked to workstream 2 of Epic: UK land value data for land value tax results by constituency #1521.corporate_land_valuehands out the engine's own aggregate by each household's share of corporate-sector wealth, so a released file meets it at any weights.land_valuedoes not hold.ons.land.household_land_value(E44N) stays bound, and its binding note now says households alone.national_contract_registry.json).tools/evaluate_uk_incumbent_surface.py, measures them, as it does every held row.What moves:
ons.land.household_land_value, now theons_landfamily's only bound row, carries three times its loss weight before. No other row's weight changes. Weights move slightly through both effects, and a calibrate-only rerun attributes them.uk_target_fitandcalibration_diagnostics.json.Group (a)
Commits 1a, 1b, 2, 3 and 4 are in, each with its review follow-up, and so are the 8 Nov follow-ups. Still to do: the 9 Nov follow-ups (stage 42 and 26's correction rule, the identity tool's exact count at K = 1, and the binding rule over every first-party package and UK graph kernel), then commits 5 to 7.
uk_runtime/copy_id_table.py, so a copy id depends only on its source id and its copy path, not on K.kin a field takesid + k × radixon all five id columns. The fields, least significant first:vintage(reserved, 10^8),geographic_support(10^9),spi_support_channel(10^11),high_value_support(10^12),cgt_support_split(10^13),cgt_incidence_clone(10^17) andcgt_residential_split(10^18).cgt_support, adds tests for the dense line, stage-16 nesting and the executor, and corrects texts.tools/verify_uk_identity_stability.py) still decodes the old rule, and 11 of its tests fail there. Don't run it from those two commits. They were pushed together with 1b.weight_tables(id_multiplier=)), though none does yet.tools/evaluate_uk_dataset_size.pyinfers the multiplier from the spine's household ids or the clone-0 household ids. It refuses, by count, a dataset the inferred multiplier does not fit, where it used to miscount silently.microcosm.build.stochastic_assignmentnow binds it, so an edit to the keyed draws re-keys the stages it moves. That is 32 more stages, 37 in all.ceil(K × w / 60), as ruled on 7 Nov. K is the frame's geographic clone count, 1 on every national spine today.ceil(w / 60)bit for bit.clone_count, and the stage-health gate and the identity tool scale their checks by K.national_sampling's legacy compact sampler (sample_uk_national_frame) and its family decoder (uk_source_family_units). No build or tool has called them since 8adc836. It adds a changelog fragment of type removed. No value moves. The full build's population nodes that hash the module re-key, and so does every node downstream ofuk.full.sample.test_uk_graphderives the bindings. From each stage's transform it follows every definition the code names, withinmicrocosm.build, and requires the stage's hash to cover each module it reaches.employer_pension_contributions.rowwise_geography, with a dense test.copy_rule, and gives the identity tool's recount its 1e-9 tolerance at K = 1 as well.main_residence_valueandproperty_wealthbefore it checks stage 25's uprating. So it no longer fails on spines built sinceregional_home_valuesjoined the roster.national_sampling. It declares the modules it samples through, so the root node hashes them.spi_support_channel:inclusion), so a household's draw depends neither on row order nor on the other households. At π = 1 every household is copied without reading a uniform.e7_spi_synthetic_share. Its maximum is the repo's 4σ binomial band around the drawn share. The band moves tostochastic_assignment.binomial_share_tolerance, with a sigma argument, andstage_health.latent_attribute_tolerancestays as an alias.household_clone_index, the prior-mass allocation adds a per-clone stratum. It is inert at K = 1.clone_count. Its eligible counts are rounded,seating_scalereads full or capped, and a band's carrier mean age is withheld below three carrier source households. The gate scales by K.stochastic_assignment, where the latent-attribute band now lives.What moves in (a) so far:
*_build_idcolumns carry the new copy ids, so copy rows join across versions by lineage (source id and copy path), not by build ids.clone_count.Rulings applied
exact.copy_rulesays so;copy_rulenames that one exception (the 8 Nov rule of never correcting a copy cannot always keep the total bitwise exact: in the synthetic nesting test at K = 7 and 13, no whole household can carry the correction);cgt_structurein stages 26 and 46 stays a listed refusal-only check;Proof plan
🤖 Generated with Claude Code