Repository navigation
Conversation
REFUGEE / ASYLEE / PAROLED_ONE_YEAR (per program origin) / TPS (per country) are now selected between the ASEC-UA indicators and the EAD spill, each to a manifest-cited target, so the H.R.1 $71109/$71301/ $71302 and SNAP $10108 eligibility channels stop computing $0 on the release. DEPORTATION_WITHHELD carries an explicit cited zero (no published stock). Residual-pool parole/TPS draws receive EAD SSN cards, keeping the NONE-undocumented invariant; the composition gate now bands each category against its target. Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thirteen stable blake2b sites (per-origin parole, refugee, asylee, deportation-withheld, per-country TPS) join the seed protocol; the seed-owner ledger binds them to the immigration_status stage; the sources schema gains the required humanitarian_status_stocks block; the bundle regenerates with the new source_stages attestation. Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h tests Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restorations of the pre-H.R.1 Medicaid, SNAP, and ACA lawful-presence lists and the below-FPL PTC exception must each move their budget measure on the export; a ~$0 score means the humanitarian statuses regressed and blocks publication. Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n digests test_us_hr1_immigration_channels builds each probe's reform against the locked engine and shows nonzero, sign-correct movement per channel (medicaid, aca_ptc twice, snap) so the release smoke gate's probes have a PR-CI early warning. The thirteen humanitarian draw sites join EXPECTED_LEGACY_V1_SITES and the audited-source manifest; BE/UK envelope digests re-pin because the shared seed-protocol source is attested. Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ration-statuses-767 # Conflicts: # packages/microcosm-build/tests/test_spec_engine_country_bundles.py
… draws The thirteen humanitarian sites move every closed-world census the fail-closed machinery keeps: protocol site counts (53 -> 66), the owner-link sum (112 -> 125, one owner each, distinct owners unchanged), the configuration field universe (42,120 -> 42,419: +13 legacy-behavior target fields, +286 attestation-only resolved seed fields), four pointer claims (source_stages, spine/resolved seed-site bindings, resolved seed protocol), and the source_stages byte attestation in the bundle tests. Refs #767 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ration-statuses-767 # Conflicts: # packages/microcosm-build/tests/test_spec_engine_country_bundles.py
…ration-statuses-767 # Conflicts: # packages/microcosm-build/tests/test_spec_engine_country_bundles.py
Add source-aware ASEC and ACS immigration evidence, pooled control scaling, deterministic humanitarian reconciliation, calibration targets, terminal gates, authenticated receipts, and release regressions. Refs #767
…ration-statuses-767
MaxGhenis
marked this pull request as ready for review
August 28, 2026 21:18
MaxGhenis
marked this pull request as draft
August 28, 2026 23:36
MaxGhenis
marked this pull request as ready for review
August 29, 2026 01:39
MaxGhenis
marked this pull request as draft
August 29, 2026 07:49
…ration-statuses-767 # Conflicts: # docs/evidence/spec-engine/us-f0-coverage.json # packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py # packages/microcosm-build/tests/test_spec_engine_country_bundles.py # packages/microcosm-build/tests/test_spec_engine_loader.py # packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py # packages/microcosm-build/tests/test_us_multispine_pool_tool.py
Resolves the six pin collisions against current main: BE/UK envelope digests, the loader golden vector, the US spec digest in the multispine tool test, the nine inventory digests plus the producer-input count the A_LFSR column adds, and the regenerated coverage evidence. The h5_io test is a real three-way merge (branch kept its stacked-pool immigration receipt tests; main's failed-gate fixture helper replaces the inlined block). Re-applies the branch's seed-group entries and site counts on main's inventory module. Refs #767 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The branch's scoring loader authenticates the canonical stacked terminal gates, so main's minimal fixture gate is refused; the helper keeps its API and adopts the branch's construction (real gates, completeness flipped). The denied-pool test now derives the fixture's household count from the positive canonical humanitarian draws instead of main's three-row literal. Refs #767 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main added the am country bundle this week; the envelope digests move with the shared seed protocol, so all three pins are recomputed by country rather than by position. Refs #767 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ration-statuses-767 # Conflicts: # docs/evidence/spec-engine/us-f0-coverage.json # docs/us-multispine-operator-ordering.md # packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py # packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py # packages/microcosm-build/src/microcosm/build/us/spec/sources.yaml # packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py # packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py # packages/microcosm-build/src/microcosm/build/us_runtime/us_late_producer_registry.py # packages/microcosm-build/tests/test_spec_engine_country_bundles.py # packages/microcosm-build/tests/test_spec_engine_coverage_tool.py # packages/microcosm-build/tests/test_spec_engine_field_usage.py # packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py # packages/microcosm-build/tests/test_spec_engine_loader.py # packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py # packages/microcosm-build/tests/test_us_bundle_core_contracts.py # packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py # packages/microcosm-build/tests/test_us_multispine_pool_tool.py # packages/microcosm-build/tests/test_us_spec_bundle.py # tools/build_us_multispine_pool.py # tools/generate_us_bundle_from_constants.py # tools/spec_engine_coverage.py
Main's two new US package resources and this branch's frozen schedule together move the configuration universe to 42,549 fields; the seed protocol/map digests, pointer inventory, coverage evidence, loader golden vector, and US spec sha mirror are recomputed on the merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sealed Round-2 review: legacy scoring accepted the frozen schema-16 schedule but still checked every execution row against today's registry, so a real schema-9 pool's immigration source row (57 inputs) and transfer row (93) were refused against today's 56 and 100. The registry builder now takes its source/transfer inventories as arguments; legacy_us_late_producer_contracts rebuilds the historical contracts from the frozen schedule's own inventories and refuses unless that derivation reproduces both the frozen schedule_sha256 (which hashes every contract's full inputs and outputs) and the sealed payload hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Sixth sync. Pin literals take the branch side and are recomputed next; the fiscal-refresh tool and its test layer main's newer changes onto the branch's humanitarian stock targets (materializer 13 -> 14 stays this branch's bump); sources.yaml is regenerated from the merged stages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Sep 26, 2026
Native commit 071f563 ported #779's paired immigration transfer, which requires exactly one row-level citizenship source (ASEC PRCITSHP or ACS CIT) with its birth-country and entry-year partners on every donor and recipient row. It did not port #779's ACS loader change (3c3bdbf) that carries CIT/POBP/YOEP, so any multispine pool with ACS rows refused (test_every_pool_transfer_family_accepts_its_produced_physical_dtype on every lane: missing_rows=2, the two ACS-channel clones). acs_pums now reads CIT/POBP/YOEP as optional person columns, like #765's hours columns. Every ACS PUMS person file carries them. A spine loaded without them stays loadable, and a paired immigration transfer onto it still refuses; nothing is defaulted. The reviewed acs_pums.py fingerprint in acs_native_coverage_binding is re-pinned with that review noted. The pool fixture's ACS rows carry the evidence, as #779's did, and the transfer gains its three evidence predictors (32 -> 35). A new loader test checks carriage when supplied and absence otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…partial) Observed on the merged tree: 42,567 configuration fields, 38 source stages, and the source-stages, generated-authorities, predictor-tuple, resource semantics, stacked checkpoint base identity, and seed digests. The ledger mode counts, pointer inventory, coverage evidence, loader golden and spec sha mirror follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Seventh sync, across main's test-tier reorganization. Pin-only files are main's versions with this branch's identities recomputed on the merged tree (42,571 fields, spec 1e3125e9, source stages e2cda1e3, schedule 88bc9243, 66 seed sites / 125 owner bindings, 2,750 producer inputs); the frozen schema-16 schedule resource and the 13 humanitarian seed sites are re-declared. Branch test changes are ported onto main's engine_free/engine split (helpers in test_support/microcosm_build), each port adversarially verified; the three new test modules move into tiers, the H.R.1 channel test to engine/us without its module-level engine guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rged tests The immigration-drift case now expects the SPM composition gate's degraded-mode line main's cofailure contract appends on the fake export frame; main's prepared-pool SPM case gets the immigration statuses every post-#767 pool carries; the shipped reform-coverage baseline plans count the three keys the four H.R.1 probes add (medicaid 2027, aca_ptc 2027, snap 2026). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… within tolerance Two defects the invariant tests surfaced (#767): - us_immigration_composition_gate re-derived the source-aware cohort evidence at a hard-coded 2024 while the stage derives it at the build's time_period, so a non-2024 build would be gated against a different DACA cohort than it assigned. The gate now takes a required time_period, and both builders pass their own period (PERIOD, POOL_TIME_PERIOD). - _spill_pew_unauthorized_excess compared the recomputed excess to zero exactly; a selection that closes the gap exactly can leave ~1e-15 of summation residue, which opened the next tier and spilled a row inside the other controlled margin — the ordering the round-1 fix promised to keep. Closure now allows a billionth of the in-scope weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e for the immigration fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
28 Hypothesis properties over synthetic ASEC/ACS person tables (6-32 rows, a third humanitarian-profile or DACA-cohort rows, periods 2024/2025/2030) for every invariant immigration.py states: NONE <=> UNDOCUMENTED; mutually exclusive humanitarian draws that never select an excluded cohort; the weight-to-target selection contract; seed determinism and lineage keying; the tiered spill (non-increasing Pew count, closure within tolerance, never touching the other margin while an outside row remains); and a differential between the spill's Pew projection and the composition gate's, with the gate evaluated at the stage's own period. Two example regression tests pin the period and closure-tolerance fixes of 5a89fed; each fails with its fix reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Astra round 2 (second half): the positive schema-9 tests kept today's declared-input surfaces, so nothing proved a real schema-9 pool's historical rows authenticate. Every producer whose schema-16 contract differs from today's now has its row sealed by the production readiness, absence, input-evidence and output-evidence builders, run against the historical contract on the fixture's own late-DAG input frame (with the WSAL_VAL/SEMP_VAL inputs the schema-16 immigration source read). The positive test asserts the historical surfaces (57 and 93 inputs, WSAL_VAL and SEMP_VAL present, no A_LFSR); a legacy pool whose immigration source row carries today's 56-input surface is refused, and a current pool whose rows carry the historical surfaces is refused, with a control proving the re-seal itself reproduces a loadable pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Ninth sync, across main's engine test split (engine_contract / engine_scenario / engine_workflow). The H.R.1 channel test moves to engine_scenario/us (it runs small household engine calculations); the loader golden follows main into engine_free/shared. Pin hunks resolve to main's side and are then recomputed on the merged tree: spec 4ae5e2e6, producer registry 2c71f35d, full checkpoint 6a6a3f74, loader golden 646929f6; coverage evidence regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Tenth sync, onto main 1acd884. Main's ACS-local immigration lane (#1052) now runs on the #767 kernel: the donor's delivered worker and student mass counts the Pew universe the spill binds, each humanitarian draw targets its own donor residual, ACS ESR maps onto the A_LFSR codebook, and the signal gate grades the build's period. The attested schema-9 legacy path keeps main's design, which rebuilds the schema-16 schedule and v11 authority from today's registry. Only #767's two immigration producers keep their schema-16 inventories: the legacy contracts take them from the frozen schedule and check each rebuilt contract against the digest the schema-16 registry derived. The legacy schedule payload, authority and all 38 legacy contracts equal main's, and the live schedule payload does not move. Two fixes ride along. The 2026 ASEC adds PEINUSYR code 29, which the default pool had started to refuse: this restores main's 29: 2024 midpoint, widens the domain to 0..29 and names each program's arrival-code window. The ACS-transfer contract pins move with #767's schema-3 contract. Git merged two edits to the ACS loader's column lists without a conflict: the PR had made CIT, POBP and YOEP required, and main's #1020 made them optional. Main's design wins. The loader leaves an absent column absent, and the immigration consumers refuse a row with no citizenship source; new tests check the refusal through the kernel's evidence profile and through the evidence features the ACS transfer reads. The late-producer scope test now names the three columns itself, and the transport battery's binding closure counts us_runtime.acs_pums, whose ACS_2024_1YR_VINTAGE the transfer imports. Recomputed on the merged tree: spec 49d7b804, producer graph 2606b852, full checkpoint d7829268, loader golden 7dd74e95, late schedule 46cea046; coverage evidence regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration-statuses-767 Eleventh sync, onto main 764ae3a. One file conflicted: the transport battery's binding closure list, where main's #1405 added the microcosm.data modules and the tenth sync added us_runtime.acs_pums; the list keeps both. Git merged main's SCF wealth gate record (#1216) beside this PR's humanitarian target materializer in the fiscal refresh release tool and its tests without a conflict; those tests pass. The spec-engine, bundle, coverage and pool pin checks pass unchanged (spec 49d7b804). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release-build rule asks this PR for a from-scratch build from its own tree; d2729 asks Max whether stage receipts stand in for it, on the terms of d2647. The receipts run the immigration stage alone on main's default pool (the 2024-2026 ASEC, the pool d2603 gives the next certified build) at this PR's merge of main 764ae3a, at that main, and at the pre-sync head 42bb166, then run the composition gate. At the merge the gate passes with no failures: refugees, asylees and TPS emit within 3% of their cited stocks, one-year parole reaches 0.90 and citizens match main. The pre-sync head refuses the 2026 ASEC's PEINUSYR code 29. The receipts hold aggregates only; the record-level labels behind the README's row-by-row table stay out of the repository, and scripts/xtab_labels.py rebuilds the table from two runs' labels. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The independent review of bad5cb6 found that the tenth merge kept main's stacked-authority pins (receipt and late-producer schedule component) in test_spec_engine_stacked_authority_semantics.py, so engine-US failed. Pin the merged tree's live values. The review also found that the Operation Allies Welcome parole window on exact-year (ACS YOEP) rows has no end year. Keep it open and say why above _PAROLE_ACS_MIN_ARRIVAL_YEAR: the stacked transfer asks the ACS arm for its own share of each national target and raises on a candidate shortfall, and the ACS 2024 1-year holds 70,123 weighted Afghan-born non-citizens who arrived in 2021-2022, fewer than the 73,566 target. The comment moves the attested kernel identities: seed protocol and map digests, the US spec (49d7b804 -> c414c985), the coverage evidence and the minimal-loader golden. Name the legacy path's second reach in the multispine design doc: post-clone source rows bind today's whole-file source_stages.json hash, which #767 changed for the fifteen source rows that bind a stage spec. The stage receipt script now records the key each draw call uses instead of a constant, and --draw-keys full|legacy forces one key so two trees' runs share their draws. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…changes The review of bad5cb6 found that every receipt reported this PR's draw key, although main keys its draws on the income year and PERIDNUM alone. The script now records each draw call's key. Regenerate the head, main and pre-sync receipts with it, and add receipt.main_full_keys.json: main's method run on this PR's key. The README credited the method with all the rows the two trees label differently. The key change alone moves about as many: on this PR's key, main's own method moves 2.400 million people from undocumented to LPR and 2.392 million back. Holding the key fixed, the method moves 0.319 million and 0.706 million. Rewrite the README to name both causes, with a row-by-row table for each. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y candidate The delta review of e3b2c8c and 9d16436 (wf_5bb29ab5-2eb) found that the post-export scorer cannot score the H.R.1 Medicaid probe. PolicyEngine-US prices each Medicaid enrollee through medicaid_slcsp_state_denominator, an aggregate over the whole file, which the household-batched scorer refuses and a reform engine refuses even in one batch (#956). Engine-US failed, and a release build would have aborted at the reform-coverage smoke. The probe now counts weighted people in medicaid_enrolled, with a 10,000-person floor. Probes carry a measure_unit, usd by default or persons; the smoke gate states effects and floors in it, and the manifest check refuses a budget measure that is not a live variable of that unit. medicaid_reporters.py counts the people in the restored statuses who report Medicaid at interview, from the head run's labels. The review also found that the receipt script said main's tree has no composition gate. Main has one, so the script now runs each tree's own gate. The changelog fragment no longer says schema-9 pools authenticate against the contract they sealed. The review found that the OAW kernel comment misstated the stacked transfer's rule: the reconciliation asks the ACS rows for each draw's target less the ASEC rows' labelled mass. Under that rule, U4U, Nicaraguan parole and Honduran TPS hold too few ACS candidates at the stacked ACS arm's weights, and the reconciliation refused any shortfall, so every stacked build would have stopped at U4U. A draw short of candidates now takes every one, as the ASEC stage's draws do, and its receipt records candidates_exhausted and candidate_shortfall_population. The reconciliation refuses only a draw that ends below 0.35 of its target, the composition band's floor, and the stacked validator replays the rule. The transfer contract's candidate_shortfall value changes, which moves its three identities. The Operation Allies Welcome window on exact-year rows now ends in 2022. stacked_supply.py projects each draw's ACS supply against its residual. The kernel edits move the attested identities: seed protocol and map digests, producer resource semantics, the stacked checkpoint base identity, the US spec (c414c985 -> 783ff9a4), the coverage evidence and the minimal-loader golden. The coverage manifest's ACA probe reasons now cite the default-pool receipt's TPS and humanitarian totals. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… receipt The receipt script reran each tree at the code commit. The head run labels every row as before and its composition, gate and draw results equal the previous head receipt's; only provenance moves. Main's two receipts now carry the result of main's own composition gate, which passes. The Medicaid-reporter receipt names the new head's labels file. receipt.stacked_supply.json projects each humanitarian draw's ACS supply in a stacked pool from receipt.head.json and the ACS 2024 1-year PUMS. Four draws run short of ACS candidates and end at 0.718 to 0.971 of their targets; in the worst case none ends below half its target. The README's receipts table, stacked-pool section and rerun instructions describe it. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stacked-supply section listed four draws that run short of ACS candidates without saying the projected share assumes no earlier draw takes a draw's candidates. Define the projected share, note that none of the four shares a candidate with an earlier draw, and add TPS Venezuela: its supply exceeds its residual by 38,923, but earlier draws could take up to 214,138 of its candidates, which would leave it at 0.710 of its target (receipt.stacked_supply.json, worst_case_share_of_target). Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The third delta review of #779 (wf_06fd9b1f-5b7) confirmed two defects in the stacked-supply prose. Honduran TPS does share candidates with an earlier draw: every one also falls in the deportation-withheld draw's pool, whose zero target takes none, so its worst case still equals its projected share. And TPS Venezuela does not just "may" run short: Venezuelan parole, the fourth draw, takes 58,115 from a pool that shares 0.848 of its mass with TPS Venezuela's, about 49,300 of it, more than TPS Venezuela's 38,923 margin. stacked_supply.py now projects an expected share for each parole and TPS draw: each earlier parole or TPS draw takes its residual from the candidates still free, each with the same chance, as the reconciliation's person-keyed hash draws do. It leaves out refugee and asylee takes, which PUMS cannot show. TPS Venezuela's expected share is 0.983; every other draw's equals its projected share. The receipt gains that one field per draw and the script's new hash; every other value is unchanged. The README table gains the column, and its bullets count five short draws. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review of a5a1dda (wf_78421e89-31e) found two docstring claims in stacked_supply.py that its code contradicts. The expected share sits between the worst case and the projected share, not between the two lower bounds: for TPS Venezuela it is 0.983 against a worst case of 0.710. And leaving refugee, asylee and withholding takes out does not overstate a later draw's share by all they take of its candidates: those takes can only lower the share, and only once the draw's free supply falls below its residual, as TPS Venezuela's does. The receipt changes only in script_sha256. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The confirmation of d12edbd (wf_abe4b4fc-c03) found that a draw that does not exhaust its candidates also takes the tie group that crosses its residual, and a refugee, asylee or withholding take can move that overshoot either way, by less than one row's weight. The docstring now takes the reviewer's wording: those takes can only lower a later draw's share, up to the weight of that tie group, and lower it by more only once the draw's free supply falls below its residual. The receipt changes only in script_sha256. Refs #767 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Hand-off to the Microcosm hub (2026-11-10). The owner session hands this PR to the hub, which lands it.
|
The engine-free (3.14) leg of this head's run drew an ubuntu-26.04 runner twice (jobs 130198769545 and 130281464672). uv took that image's Python 3.14.4 there, and the head-to-head scorer's worker-recycling test needs 3.14.7. Main pins every Tests job to ubuntu-24.04 since #1576, and a re-run keeps the old workflow, so this empty commit starts a run that reads main's. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
Author
|
Hub note: I pushed |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #767.
Implements the design proposed in the issue thread, plus one follow-on redesign of the spill controls (Aug 29, "Align immigration controls with Pew methodology") that this description now reflects. The US
immigration_statussource stage draws humanitarian/temporary-protection statuses to externally cited stocks, so the enacted H.R.1 eligibility channels stop computing ≈$0 on the next build.What changes
Humanitarian draws (
us_runtime/immigration.py). Between the ASEC-UA legal-status indicators and the EAD spill, thirteen seeded draws run — sequential, mutually exclusive, in decreasing order of target precision:PAROLED_ONE_YEARper program origin: OAW Afghans (73,566), U4U Ukrainians (158,000), CHNV Nicaraguans (93,070) and Venezuelans (117,330), with program-specific arrival windows (U4U/CHNV cannot draw from the 2020–21 bin);REFUGEE(160k ≈ FY2023+FY2024 OHSS admissions) andASYLEE(155k ≈ FY2022–24 OHSS grants as revised) from the indicator-documented pool only;TPSper country from the CRS RS20844 table (Venezuela 605,015; El Salvador 170,125 with the 2001 continuous-residence cutoff binding at PEINUSYR granularity; Honduras/Nicaragua/Nepal/other-designated likewise);DEPORTATION_WITHHELDcarries an explicit cited zero (EOIR grants ~2.2k/year and publishes no stock).Every draw excludes the Cuba/Haiti-born (the
CUBAN_HAITIAN_ENTRANTclass is the statutorily favorable label H.R.1 preserves — Haiti TPS rides it) and the DACA statutory cohort, and the parole draw carries Ukraine/Afghanistan TPS registrants. Residual-pool draws (parole/TPS only) flipssn_card_typeNONE →NON_CITIZEN_VALID_EAD, preserving theNONE ⇔ UNDOCUMENTEDgate invariant.Spill controls re-founded on Pew's universe. Pew's 2025 report supersedes the earlier series with a coherent 2023 pair — 14.0M unauthorized immigrants, 9.7M in the labor force — whose modeled universe deliberately retains DACA, parolees, TPS holders and similar temporary protections (and excludes refugees and granted asylees). The worker/student EAD spills now bind that broad universe rather than the bare residual pool: residual undocumented rows, DACA and residual-cohort Cuban/Haitian rows that keep protected engine labels after an EAD, and the Pew-included humanitarian marks. The worker margin uses measured current labor-force status (
A_LFSR, age 16+) restored from the pinned Census ASEC archives by exact PERIDNUM join (checkpoint schema v4) instead of prior-year earnings. The composition gate anchors the same broad universe against 14.0M — identifying residual-derived entrants by the(CUBAN_HAITIAN_ENTRANT, NON_CITIZEN_VALID_EAD)pair, which is exactly the set the spill counts, since indicator-documented entrants carryOTHER_NON_CITIZEN.Manifest, spec, attestations. Every target lives in
source_stages.jsonwith one citation per number (URLs verified live, the DHS OAW/U4U reports by exact byte match); the sources schema gains the requiredhumanitarian_status_stocksblock; thirteen_stable_sitedeclarations join the seed protocol with owner-ledger bindings; every closed-world census the fail-closed machinery keeps (site counts, owner links, field universe, pointer claims, inventory digests, envelope digests, coverage evidence) is re-pinned on the merged tree.Gates & probes. The composition gate bands each category's emitted mass against its cited target (explicit-zero categories must emit zero) and records achieved-vs-target for certification notes. Four release-blocking reform-coverage probes pin the channels end-to-end — restoring the pre-H.R.1 Medicaid list (§71109, 2027), the below-FPL PTC exception (§71302, 2026), the ACA lawful-presence list (§71301, 2027), and the pre-§10108 SNAP list (2026) must each move their measure on the export, binding only through
immigration_status_str. The Medicaid probe counts the people it enrolls (medicaid_enrolled, at least 10,000); the other three score dollars.Evidence
The stage receipts and the script that writes them live in
docs/evidence/us-immigration-statuses-767/. The script runs the immigration stage alone on main's default pool, the 2024, 2025 and 2026 ASEC, which d2603 gives the next certified build. It runs at this PR's4f94e489aand at main764ae3aa7, at seed 0 and period 2024, then runs each tree's own composition gate. Later commits change only that folder. Weighted persons, in millions:ssn_card_typeisNONEexactly on the undocumented rows in both trees.receipt.pre_sync_42bb166b.json); see "The 2026 ASEC's arrival code 29" below.The two trees label different rows, for two reasons. A fourth receipt,
receipt.main_full_keys.json, runs main's method on this PR's draws and so separates them.PERIDNUM; this PR adds the household'sPH_SEQ. In these files the old key already tells all 421,119 rows apart, but the new key changes every hashed string and so every row's draw, and the EAD spill takes candidates in draw order. On this PR's key, main's own method moves 2.400 million people from undocumented to LPR and 2.392 million back, and 0.220 million from undocumented to DACA and 0.244 million back.A_LFSRat age 16 and over instead of prior-year earnings (WSAL_VAL,SEMP_VAL), and spills until Pew's broad universe matches the controls instead of the undocumented rows left in the residual pool. With both runs on one key, the method moves 0.319 million people from undocumented to LPR and 0.706 million back, and 0.031 million from undocumented to DACA and 0.105 million back.The two effects don't add, since both reshuffle one pool. Together they move 2.021 million of main's undocumented rows to LPR and 2.415 million the other way, and 0.172 million of main's undocumented rows to DACA and 0.271 million of its DACA rows to undocumented. The README has a row-by-row table for each comparison.
The checks tab is authoritative for CI. Locally, the four read-only pin checks pass after the tenth and eleventh merges (spec
49d7b804), after the first 2026-11-10 review fixes (specc414c985) and after the second (spec783ff9a4): coverage 42,781 of 42,781 configuration fields and 41 of 41 inventory checks, bundle, spec contracts and pool contracts.Independent review round (2026-09-19)
A second-reader audit of head
9622beb5requested three changes; all are on the current head._spill_pew_unauthorized_excessentered the other controlled margin before exhausting rows outside it, so the student control could remove a worker the worker pass cannot restore (reproduced in 4 of 6 seeds). Each tier is now exhausted before the overlapping one;test_student_spill_exhausts_nonworkers_before_touching_a_workerpins it.2f61b55e…instead of the sealed02e618cc…), and it required the 12-step operator order and immigration reconciliation evidence of pools that predate them. History is now data: a frozenus/legacy_schema16_late_producer_schedule.jsonverified against its sealed hash, a literal 11-step order, and ordinary immigration transfer receipts — all only underlegacy_worker_authentication. The legacy digests now equal main's byte for byte (test_us_legacy_schema9_contract.py), the positive fixture is genuinely historical, and two layers refuse a version-downgraded current artifact. (Main has since moved its legacy path; the tenth sync narrows the frozen schedule's role to US: humanitarian immigration statuses absent from the imputation — H.R.1 §71109/§71302 eligibility channels compute ≈$0 #767's two producers, below.)Second review round and invariant tests (2026-09-19 to 2026-09-29)
legacy_us_late_producer_contracts()now rebuilds the schema-16 contracts from the frozen schedule's own input inventories. It refuses unless the rebuild reproduces both the frozenschedule_sha256and the sealed payload hash. (Main's later contract changes broke that rebuild; the tenth sync replaces it, below.) The positive legacy fixture now seals those rows with the production readiness and evidence builders under the historical contracts. Two negative tests pin both directions: neither a legacy pool that carries today's surfaces nor a current pool that carries historical surfaces passes. A control shows the re-seal itself is faithful.us_immigration_composition_gatere-derived the DACA cohort at a hard-coded 2024. It now takes the build'stime_periodas a required argument.Tenth main sync (2026-11-09)
This sync merges main
1acd884fe. It ports main's ACS-local immigration lane onto this PR's kernel, keeps main's design for the attested schema-9 legacy path while isolating #767's two producers on it, fixes the stage for the 2026 ASEC's arrival code 29, moves the ACS-transfer contract pins, and settles a conflict git merged silently in the ACS loader's column lists.ACS-local immigration on this kernel
Main's ACS-local immigration lane (#1052, for #1020) runs this module's kernel on ACS rows. It aims each national control at the residual the donor spine leaves, measured at pooled weights. Taking either side's version would have broken the other, so this merge ports main's lane onto the #767 kernel:
_pew_unauthorized_rows, the universe the kernel's spill binds, instead of bareUNDOCUMENTEDrows. A donor DACA, TPS or parole worker now counts toward the 9.7M control, as it does nationally.max(0, national target − donor-emitted mass). The kernel's own emission mask (_humanitarian_emitted_mask) measures the donor side. The receipt records target, donor-delivered, residual, ACS-assigned, pooled total and excess for each draw.acs_local_immigration_receipt_failuresrefuses a receipt whose draw set differs from the packaged draws, or whose residual or pooled identities fail. The method string moves tonational_control_minus_donor_pooled_delivered_v2, so the release tool refuses a staging run from before this change.ESRmaps onto the CPSA_LFSRcodebook (1→1, 2→2, 3→3, Armed Forces 4 and 5→0, 6→7, blank→0; the mapping refuses any other code). The kernel's worker test (age 16+,A_LFSR1–4) then reads ACS rows directly._donor_status_masksusesA_LFSRwhenever the donor carries it. In production the donor doesn't:tools/build_us_puf_support_base.pygives the immigration stageA_LFSRas an ephemeral input and drops it from its output. So the donor's delivered worker mass keeps main's earnings-pair test, and the receipt names that test indonor_status_columns.worker. ACS-local donor worker count uses the earnings proxy because the support base drops A_LFSR #1554 tracks restoringA_LFSRon donor rows.acs_local_immigration_signal_gatetakes a requiredtime_period(the composition gate has required it since the round-2 fix) and optionalcontrols. Both callers pass their period: the release tool and the legacy multispine builder.snap2026 key with the Texas vehicle-exemption probe. The engine-US drift test now accepts the list-valued restored-eligibility probes.Property tests (Hypothesis, in the same module) cover the port's invariants:
max(0, target - donor)and the emitted ACS rows weigh exactlyacs_assigned;A mutant that draws the ACS rows toward the full national stock fails the residual property at once.
The attested schema-9 legacy path
Since the last sync, main's #1239 and #1258 changed the primary-PUF S-corporation outputs and moved the PUF itemization transfer targets between batches. Main rebuilds its legacy schedule and v11 authority from today's registry and surfaces and relabels them at schema 16, so at main
1acd884fethey hash to2868139b…and3860e1c3…, not the sealed02e618cc…ande660a8ce…. This branch had rebuilt every legacy contract from the frozen schema-16 inventories and refused unless the rebuild reproduced the sealed hashes, so on the merged tree it refused every legacy load. The merge keeps main's design and isolates only what #767 changed:7c4a5ac0, whose registry reproduces the sealed schedule);On the merged tree the legacy schedule payload, the legacy authority and all 38 legacy contracts equal main's exactly. The parameterized payload builder leaves the live schedule payload byte-identical (
3f1daff0…).test_us_legacy_schema9_contract.pypins this in 16 tests, including a Hypothesis round trip of the inventory payloads. Two #767 changes still reach the legacy path, through two bindings it takes from today's code, as main's does. Late-transfer rows bind today'sacs_transfer_execution_contract_identity, which main's own post-transfer changes since7c4a5ac0already moved for all nineteen groups. Post-clone source rows bind today's source-stage binding, which hashes the whole packagedus/source_stages.jsonand carries each bound stage's resolved spec. #767 rewrote theimmigration_statusstage, so its change reaches the fifteen source rows that bind a stage spec (direct housing assistance binds none), and main has changed that file in fourteen commits since7c4a5ac0. Restoring main's whole legacy path to sealed history is separate work.The 2026 ASEC's arrival code 29
On 2026-09-27 (e518fb8), main pinned the 2026 ASEC and set the default US pool to the newest three income years (
asec_sources.ASEC_DEFAULT_POOL_INCOME_YEARS: the 2024, 2025 and 2026 ASEC). An earlier sync brought that onto this branch. Census re-bins the newestPEINUSYRcodes each survey year. Code 27 is 2020-2021 in all three files. Code 28 is 2022-2024, 2022-2025 and 2022-2023 in turn, and the 2026 ASEC adds code 29 for 2024-2026. Foreign-born infants (age 0-1) sit only in the newest code of each file (28, 28 and 29). This branch had removed main's29: 2024midpoint and refused code 29 (3c3bdbf, Aug 28), which held while the stage ran on the 2024 ASEC alone. Since that sync, the stage has refused the default pool outright:PEINUSYR carries value(s) outside the 2024 ASEC domain 0..28: [29.0](docs/evidence/us-immigration-statuses-767/receipt.pre_sync_42bb166b.json). The single-file receipts never saw it. Main's ACS-local_asec_entry_yearalso refuses any foreign-born donor whose code is missing from that table, so on the merged tree it refused every code-29 donor. The fix:29: 2024and widens the domain check to 0..29;_POST_2021_ARRIVAL_CODES), never 27.test_2026_asec_peinusyr_29_is_in_the_domain,test_peinusyr_past_the_2026_asec_domain_raisesandtest_2026_asec_code_29_draws_only_programs_running_in_2024;ACS-transfer contract pins
Main's #1056 review pinned the shared ACS-transfer execution contract with no person predictor extensions. #767 moves that contract to schema 3, adding its immigration predictors, the baseline-excluded statuses and the humanitarian post-transfer block to every group, so the three pins move: default
4403aefd…, declared plance27d251…, pregnancy061d8f10….ACS loader column lists
Git merged two edits to the ACS loader's column lists without a conflict. This PR had moved CIT, POBP and YOEP into
_PERSON_REQUIRED, and main's #1020 added the same three columns to_PERSON_OPTIONAL. The merge keeps main's design: the loader leaves an absent column absent, and the immigration consumers refuse a row with no citizenship source. Two new tests intest_us_acs_pums.pycheck that refusal through the kernel's evidence profile and throughus_immigration_evidence_features, which the ACS transfer reads. The late-producer scope test names the three columns itself, and the transport battery's binding closure addsus_runtime.acs_pums, whoseACS_2024_1YR_VINTAGEthe transfer imports.Eleventh main sync (2026-11-09)
This sync merges main
764ae3aa7, 98 commits past1acd884fe. One file conflicted: the transport battery's binding closure list (BINDING_MODULES_ADDED_AFTER_BASEintest_transport_evidence_kernels.py), where main's #1405 added themicrocosm.datamodules and the tenth sync addedus_runtime.acs_pums. The list keeps both. Git merged main's SCF wealth gate record (#1216) beside this PR's humanitarian target materializer intools/build_us_fiscal_refresh_release.pyand its tests without a conflict, and those tests pass. The four pin checks pass unchanged. The stage receipts above run at4f94e489a, four commits past this merge, and at764ae3aa7.Local runs (
pytest -n 3; the checks tab is authoritative for the full tiers):4b3b6a7a6: the immigration, immigration pool-scaling, immigration property, ACS-local immigration and legacy schema-9 contract files (270 passed), and the transport evidence-kernel, fiscal-refresh builder and plan files (569 passed, 2 skipped).f5491c5f9: the ACS loader, ACS-local hours, income, receipt-anchor and work-disability, ACS multispine, ACS multispine legacy builder, ACS release-predictor, ACS transfer and late-producer files (452 passed).ruff check .passes, and the four pin checks pass after each merge.Review fixes (2026-11-10)
An independent review of
bad5cb667(a native Opus 5.5 workflow) found one blocking defect, one major and three minor. The current head answers all five:test_spec_engine_stacked_authority_semantics.py. Engine-US failed in a contract batch, which stops at its first failure (--maxfail=1); the runner then admits no new task and skips every workflow file, so most of the tier never ran. Both pins now hold the merged tree's live values (7cdedd5b…,bdc3ab40…).PERIDNUMalone, and the README credited the method with every row the two trees label differently. The script now records each draw call's key and takes--draw-keys full|legacy. The receipts are new, and a fourth runs main's method on this PR's key: the key alone moves about as many rows as the full comparison (Evidence, above).source_stages.jsonhash, which US: humanitarian immigration statuses absent from the imputation — H.R.1 §71109/§71302 eligibility channels compute ≈$0 #767 changed.docs/us-multispine-operator-ordering.mdand the legacy-path paragraph above now name it.YOEP) rows. A kernel comment above_PAROLE_ACS_MIN_ARRIVAL_YEARsaid the stacked transfer asks the ACS arm for its own mass share of each national target and refuses a candidate shortfall, so the window also took 2023-2024 arrivals. The comment moved the attested kernel identities (US spec49d7b804→c414c985). The second review found that the comment misstated the transfer's rule, and the window now ends in 2022 (below).Local runs at
e3b2c8ce9(the checks tab is authoritative):pytest -n 3(598 passed);--maxfail: the 67 contract and scenario files in onepytest -n 3 --dist loadfilesession (485 passed, 6 skipped), then each of the 22 workflow files in its own process: 20 passed,test_us_puf_support_base_equivalence.pyskipped both its tests, andtest_us_post_export_scoring.pyfailed on the Medicaid probe, the second review's blocking finding (below);tools/repin_us_spec_engine_contracts.py --check,tools/repin_us_pool_engine_contracts.py --check, coverage--checkand bundle--checkall exit 0.Second review fixes (2026-11-10)
A native Opus 5.5 delta review of
e3b2c8ce9and9d164367f(workflowwf_5bb29ab5-2eb, one finder and one skeptic per dimension) left six findings standing, which name four defects: the two blocking findings name one, and four minor findings name the other three.4f94e489aand2f70d5c3canswer all four, and the work on one of them found a fifth defect, in the stacked transfer.medicaid_slcsp_state_denominator, an aggregate over the whole file. The household-batched scorer (microcosm#956) refuses it, and a reform engine refuses it even in one batch; main's own guard test pins both refusals. Engine-US failed attest_shipped_baseline_plans_match_whole_file_policyengine_us, and a release build would have aborted at the reform-coverage smoke. The probe now counts weighted people inmedicaid_enrolled, with a 10,000-person floor. Probes carry ameasure_unit(usdby default, orpersons), the smoke gate states effects and floors in it, and the manifest check refuses a probe whose budget measure is not a live variable of that unit: acurrency-USDvariable, or a person-level boolean. On the default-pool stage receipt, 0.391 million of the 0.714 million people in the restored statuses report Medicaid at interview (receipt.head_medicaid_reporters.json), so the floor is 2.6% of the reporters.The stacked transfer refused U4U in every stacked build. The
--exact-karm, the only one that runs the improvement gate, takes a stacked pool. Checking the OAW finding against the ACS 2024 1-year showed the U4U draw short of candidates under the real rule. The ASEC arm carries 56,467 of its 158,000 target, the reconciliation asks the ACS rows for 101,533, and at the ACS arm's weights (scale 0.478) the Ukraine-born non-citizens who arrived from 2022 on hold about 57,046. The old reconciliation refused any such shortfall, so every stacked build would have stopped at U4U, the second draw in manifest order. The ASEC stage already handles the same shortfall: it takes every candidate (U4U ends at 0.715 of its target there), and the composition gate's band, 0.35 to 1.5, decides. The reconciliation now follows that rule:candidates_exhaustedandcandidate_shortfall_population;candidate_shortfallfield now readstake_every_candidate_error_below_composition_band_floor, which moves the three transfer-contract identities.With that rule, the OAW window ends where the program does: exact-year rows take Afghan arrivals from 2021 and 2022 only. ASEC rows keep their coarser codes.
scripts/stacked_supply.pyprojects each draw's ACS supply from the PUMS without building a pool (README, "The stacked pool's ACS residuals"). Five draws run short. U4U, Nicaraguan parole, Honduran TPS and OAW run short even if no earlier draw takes their candidates, and end at 0.718 to 0.971 of their targets; no earlier draw that takes any rows shares a candidate with them. TPS Venezuela's supply exceeds its residual by 38,923, but Venezuelan parole, the fourth draw, takes an expected 49,300 of its candidates, so TPS Venezuela ends at an expected 0.983 of its target before refugee and asylee draws take any of them. In the worst case, where every earlier draw takes as much of a later draw's candidates as its own residual allows, no draw ends below half its target: TPS Venezuela ends at 0.710, and the lowest, refugee and asylee, at 0.505. The ASEC arm alone carries 0.357 of U4U's target, above the floor even with no ACS rows.A third delta review, native Opus 5.5 workflow
wf_06fd9b1f-5b7at5610bde70(it continued two runs that a usage limit stopped), found no code defect at4f94e489aand three prose defects, whicha5a1dda0band this body answer. Honduran TPS's candidates all fall inside the zero-target deportation-withheld draw's, so the four short draws share no candidate with an earlier draw that takes rows, rather than with none. TPS Venezuela runs short in every stacked build on these inputs, not just in the worst case, sostacked_supply.pynow projects an expected share for each parole and TPS draw. And the eleventh-sync section still named the receipts' old commit.The new rule is this PR's call, not main's: main has no humanitarian draws, and the reconciliation, its shortfall refusal and the band all arrive here. Apart from the OAW window, a draw that does not exhaust its candidates runs the old statements and selects the same rows, and its receipt gains only the two new fields. The property tests check the old bounds on those draws, that an exhausted draw takes exactly the candidates no earlier draw took, that a draw left below the floor refuses by name, and that the replay agrees with the reconciliation. The kernel edit moves the attested identities: seed protocol and map digests, the US spec (
c414c985→783ff9a4), the coverage evidence and the minimal-loader golden.Local runs at
4f94e489a(the checks tab is authoritative):pytest -n 3(1,021 passed);tools/repin_us_spec_engine_contracts.py --check,tools/repin_us_pool_engine_contracts.py --check, coverage--check(42,781 of 42,781 configuration fields, 41 of 41 inventory checks) and bundle--check(spec783ff9a4) all exit 0;--maxfail: the 67 contract and scenario files in onepytest -n 3 --dist loadfilesession (489 passed, 6 skipped), then 11 of the 22 workflow files, each in its own process, before a session restart stopped the run. Ten passed, among themtest_us_post_export_scoring.py, which failed ate3b2c8ce9, andtest_us_puf_support_base_equivalence.pyskipped both its tests. CI's engine-US jobs ran the whole tier at2f70d5c3cand pass.Release-build rule
docs/us-release-build-rule.mdasks a pull request "large enough to change how the dataset is built" to carry a from-scratch build and certification receipt from its own tree before it merges. This PR changes how the build draws immigration statuses: the ASEC source stage, the transfer of those statuses onto ACS-transferred rows, and the ACS-local lane. In place of a from-scratch run it carries stage receipts on the inputs the next certified build uses, the first on main's default pool (Max's ruling d2603). The receipts cover the source stage (Evidence, above). The transfer and the ACS-local lane run only in a full build, where the property tests below and the release gates check them: the release tool refuses a base whose composition gate fails and grades that gate again at the delivered weights, and the four channel probes must each move their measure on the export. For the stacked transfer, the evidence folder also projects each draw's ACS supply against its residual ("The stacked pool's ACS residuals" in its README). Whether the receipts may stand in for the from-scratch run is Max's call. d2729 asks it on the terms of d2647 for #880: the next certified US build on main becomes this PR's from-scratch run.Invariants (property-tested)
engine_free/us/test_us_immigration_properties.pyholds 30 Hypothesis properties. Each one quotes the code comment or docstring that promises it. Every property below holds against the current code. The two example regression tests fail with their fix reverted._select_weight_to_target:_stable_person_draws: draws lie in [0, 1), key only on the most complete source lineage, and reproduce exactly._spill_pew_unauthorized_excess:ssn_card_typeNONE ⇔ status UNDOCUMENTED;engine_free/us/test_us_acs_local_immigration.pyadds six Hypothesis properties for the ACS-local port, listed in the tenth-sync section above.Independence limits, stated in the test module: the humanitarian candidate pools and some receipt identities are only checked against the module's own helpers or guards. The property tests do not independently establish them.
axiom: n/a: microsimulation data imputation (immigration-status inputs to existing policyengine-us eligibility rules), no policy encoding.
Deliberately out of scope (filed)
years_since_us_entry(blocked on the engine's CHIPRA-214/ICHIA option).This PR touches no certified release; the imputation activates in the next build lane, where the smoke gate enforces the four channels permanently.
🤖 Generated with Claude Code