Repository navigation
US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578
Description
The acceptance rule (stated once, enforced everywhere)
A candidate release ships iff:
- It beats the incumbent on the frozen comparison register (same targets, both artifacts rescored on it — never each build's own register), and
- The invariant battery passes — the codified list of everything the loss cannot see: input-mass parity, tail concentration, effective sample size, band delivery, coverage/degeneracy, spine agreement (new here). Every battery entry was purchased with a specific incident; the battery is versioned, machine-checked in one batched pass, and recorded in both manifests.
Nothing else gates a ship. Adjudications and fences are receipts inside rule 2, not extra rules; the per-arm sprawl this epic retires was rule 2 metastasizing across two lineages. Loss is the objective the optimizer games — it can be rule 1 only because rule 2 checks what it can't say.
The design (ratified direction, 2026-07-29)
One base pool, one lineage, every dataset a record count:
- Two household spines + one detail channel. CPS-ASEC and ACS PUMS as peer household spines; PUF stays a cloned tax-detail channel on household-shaped shells (PUF records are filers — no household structure — so it cannot be a peer spine; today's
puf_tax_detailchannel is already this). - Dense is not a category — it's k = N. The full pool with no selection, calibrated
dense_no_l0. - Sparse is not a category — it's an exact record count. Releases are labeled by cardinality (
…-k57240,…-k250000, …), and the artifact carries exactly that many households, gate-asserted (len(support) == k). "sparse"/"dense" retire as dataset names; future sizes are just new k values.
Exact-k selection (three stages, all seams existing)
- Stage 1 — L0 to the ballpark, keeping probabilities. The hard-concrete gate machinery (
populace-calibrate/gates.py) already yields a per-record open-probability π_i; today we threshold it. Instead, stop at π. - Stage 2 — certainty units + fixed-size πps draw. Records with π_i ≥ π_hi are force-included (certainty units, standard survey practice); the remaining budget k − |certainty| is drawn from the boundary mass with a fixed-size inclusion-probability-proportional design (conditional Poisson / Sampford), seed pinned in the release manifest. Deterministic given (solve state, seed); no per-target knobs anywhere.
- Stage 3 — the existing frozen-support weight refit + full gate battery. Unchanged (
calibrate_l0_refitsemantics). Cell-coverage gates catch any unlucky boundary draw; a k below the feasibility floor for the target surface fails loudly, never ships thin.
Why now — the measured defects this must clear (the bar, not a hope)
The buildl ACS-multispine staging (#403; 1.61M households, ASEC+ACS) measured three defects that are the acceptance criteria for increment 2:
- Spine-conditional agreement. Baseline SSI incidence 2.59% (ACS spine) vs 1.85% (ASEC); $10k/$20k delta $821 vs $433 per baseline recipient. NEW GATE: spine-conditional distributions of transferred inputs must agree (declared tolerance) before calibration is attempted.
- Effective-support collapse. buildl calibration landed ess_fraction 0.003 (≈4,854 effective of 1.6M) with persons-per-household 1.50 (ASEC) vs 2.93 (ACS). A dense arm that is effectively 5k records is worse than an honest exact-k release. Gate on ess and per-spine composition.
- Baseline SSI overshoot vs SSA on that surface (no SSI target there then; the v9.x register carries the SSA bands now).
The transfer machinery these gates need did not exist in June and does now: donor joint-vector fidelity (#570's construction assertion + bijection), the deterministic mass-conserving tail-transfer stratum (#568), stage-attributable concentration gating (#571), reviewed-exclusion discipline (#564/#574), and arm-fence receipts in both manifests (#577).
Increments
- Canonical multispine operator ordering: assemble spines first, then clone/impute/derive/seed/simulate/calibrate #395 operator ordering — assemble spines first; clone/impute/derive/seed/simulate/calibrate as spine-blind operators downstream. (Existing charter; becomes increment 1 of this epic.)
- ACS transfer re-run under the CG depth charter: close the dense netCG gap (−30.8%, B) so the dense arm can publish #567-era machinery, gated on the ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403 defects above (spine-agreement + ess + composition gates).
- Exact-k selection (stages 1–3 above) from the healthy pool; first ladder: k = N (dense), k ≈ today's 57k contract, and one smaller demo-class k. Spine-aware near-duplicate handling in stage 2 so support composition is not solver noise.
- Lineage unification: the acs_local overlay (Rebuild the US ACS local-area artifact on the certified Build O lineage (buildl 36de5d9a is stale) #512) re-bases onto this pool (local = filter/reweight the one national dataset); per-arm adjudication registers re-adjudicate against the unified pool (SSI fences Dense arm fences its SSI adult bands: no in-band adult pair after the one permitted recompute (#566/#567) #577, QRF registers, STCG parity Reviewed parity exclusion: short_term_capital_gains (untargeted vs incidental reference) #574 all RE-EXAMINED, none carried forward silently).
Non-goals / doctrine carried forward
- No per-target calibration knobs; no loss shaping; gates are never loosened to fit outputs.
- Raw-only doctrine intact: ACS-spine records carry transferred inputs with full provenance; measured frame values untouched.
- Selection is deterministic given the manifest (solve state + pinned seed); the notary/CI split unchanged.
- Release ids state the exact count; nothing labeled "sparse" or "dense" going forward.
Context receipts
- Certified national default today: 57,240 = 35,040
puf_tax_detailclones + 22,200asec(channel crosstab of the released H5); base pool 352,932 = 184,080 + 168,852 — zero ACS units in either arm;--acs-h5is input transfer only. - CD carrying capacity is the motivating example: ~130 households/district on the 57k (47.4% within-10 on the CD surface) vs 65.1% on the full pool.
- Relates: Canonical multispine operator ordering: assemble spines first, then clone/impute/derive/seed/simulate/calibrate #395 (increment 1), ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403 (acceptance bar), Rebuild the US ACS local-area artifact on the certified Build O lineage (buildl 36de5d9a is stale) #512 (overlay re-base), CG depth charter: close the dense netCG gap (−30.8%, B) so the dense arm can publish #567/Dense arm fences its SSI adult bands: no in-band adult pair after the one permitted recompute (#566/#567) #577 (dense-arm adjudications to re-examine), Refit crushes keogh carrier weights to $6.1M of $148.97M — smoke floor unreachable on the sparse arm (attempt 15) #445 (frozen-support discipline), Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462/CG tail transfer: field-local donor quarantine + mass-conserving own-tail stratum + E01000 receipt (#567) #568/Tail assignments deliver the donor's joint vector verbatim (fixes the recipient-value overwrite) #570/Scope the tail stage's frame gate to stage-attributable worsening #571 (transfer machinery).
Sequencing: starts after the dense-P3 diagnostic publishes (#567 close-out). Build-Q-scale epic.
Scope hardening (Max ratified 2026-07-29 evening — this section governs)
- One suite per country. The separate local-area artifact is RETIRED, not re-based.
populace_us_2024_acs_local(and the Rebuild the US ACS local-area artifact on the certified Build O lineage (buildl 36de5d9a is stale) #512 lane) ends as a product: there is no "local file." Every release of the one pool carries the full geography ladder and geo targets; every file is local — the difference between releases is fidelity, and fidelity is published, not implied by a product name. The k-ladder's scoreboard reports within-tolerance by geography level (national / state / CD) per release, so choosing a k IS choosing a local sample size with its measured consequences in front of you. - Record count is the only within-country variation. No sparse/dense, no local/national, no arm names.
populace-us-2024-k57240,-k353000,-k5000— one lineage, one register, one battery, k values. - Non-US source tiers (the one extra axis, named by provenance, never by quality). Where a country has a licensed-microdata build and a public-derivable build (UK: FRS-licensed vs the public basis), the tier is named by its spine source license class: proposal
populace-uk-2023-frs-k<N>(restricted distribution) andpopulace-uk-2023-public-k<N>. Rules: the tier token names what the spines are built from, never adjectives ("true", "full"); both tiers run the same register and battery; the public tier's scoreboard shows its fidelity gap to the frs tier as a published number. Tier token per country ratified with that country's onboarding. US ships tierless (single public-safe basis). - UK is in scope for the same end state: one UK pool, exact-k ladder, OA-ladder geography on every file, both tiers, same two-line acceptance rule. The UK lineage (June 19 vintage) gets the O/P-era battery and doctrine as part of this epic, not as a separate arc.
Increment 4 is amended accordingly: not "overlay re-base" but overlay retirement — the acs_local consumer surface (policyengine.py named access) migrates to a k-release of the one pool, then the artifact family is closed out with a deprecation note in the bundle manifest.
Activity
MaxGhenis commentedon Jul 30, 2026
UK parity audit (sol recon, 2026-07-30 — full receipts in the lane log)
Tier naming settled by lineage facts: the bundled UK suite is UKDS-licensed FRS-derived microdata in
populace-uk-private— there is no public UK entry in today's bundle at all. Token for the bundled suite:frs. The public basis is a separate Enhanced-CPS-to-UK transfer lineage (maps eligible US ECPS households into UK entities, calibrates to UK targets; its own docs say it does not represent native UK survey data) — per the provenance-token rule, that suite should becps-transfer, notpublic(a license adjective). Adoptingfrs/cps-transferunless overruled.June UK release facts (the current certified private default): 149 targets (national 22 / region 119 / country 8 — zero LA or constituency targets), 146/149 within 10%; 535,080 rows of which only 335,080 carry weight (the 200k SPI-synthetic stratum ships entirely at zero weight); Kish ESS 21,378; max weight 18,653. Two landmines the ladder work must clear:
Exact-k readiness (both countries): not ready anywhere — the existing
target_recordsAPI is explicitly approximate (±5% or closest run), and June UK never invoked L0 at all. The #578 exact-k design (certainty units + fixed-size πps boundary draw + fixed-support refit +n_nonzero == kassertion + full battery per rung) is new machinery, as chartered.UK increment ordering (from the audit's effort-ranked backlog):
frs,cps-transfer) — low.Items 1–6 are the launchable-parity path and don't wait on 7; the
frsk-ladder can bridge on the June-derived base with the battery enforced, with raw-only (#144) completing underneath and re-cutting the suite when done.Assemble spines first: pre-operator multispine seam, spine-blind oper…
One-pool multispine builder: assemble-first build path with the agree…
MaxGhenis commentedon Jul 31, 2026
Increment 1 (#395 ordering) is fully landed.
assemble_spinespre-operator seam, PUF rejected as a peer spine (clone-index routed as the tax-detail channel), 27 operators spine-blind with an AST guard, and the spine-agreement gate — incidence band [0.8, 1.25], quantile envelope 0.25, default registry covering the take-up contract + SSI. The ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403 repro fails the gate at ratio 0.714, as it should.tools/build_us_multispine_pool.py— assemble-first build path (producer split at the pre-operator boundary), tombstone-first publication, agreement gate terminal; plus the spine-blindness guard hardened through 22 adversarial review rounds (guard battery grew 61 → 495 tests; final rounds verified by independent case oracles with zero divergences). Cumulative post-round-2 diff was guard-test-only.Next: increment 2 — the ACS transfer re-run under the current machinery (#568/#570/#571-era), executed through the new pool builder with the agreement gate adjudicating the #403 defects on real data. First run stops at the gate (transfer + agreement adjudication, no calibration spend) so the gate verdict is the deliverable.
MaxGhenis commentedon Jul 31, 2026
Demo-class ladder point set: k = 20,000 (Max, 2026-07-31).
The first exact-k ladder is now: k = N (full pool), k = 57,240 (today's contract), k = 20,000. No architecture is involved in adding it — k is a release parameter of the stage-2 selection (#585), and every ladder point runs the same frozen comparison register, invariant battery, and per-geography scoreboard. Whether 20,000 records carry the state surface is a number the scoreboard publishes, not a promise: the cell-coverage gates fail loudly if a k is below the feasibility floor for a target surface, per the charter.
Exact-k selection: pi seam, certainty units + fixed-size Sampford dra…
MaxGhenis commentedon Jul 31, 2026
Increment 3 machinery landed: exact-k selection (#585, merged 46e2e41).
select_exact_k— certainty units (π ≥ π_hi) + fixed-size Sampford πps boundary draw (exact O(Nk) dynamic program / accept-reject with fail-closed cap / complement design, all three paths cross-authenticated statistically and by mutation tests: stripping the Sampford tilt or substituting SRSWOR fails the committed joint-law and marginal tests), returning(support, six-scalar manifest receipt, aligned q). Infeasible designs error rather than clamp; complex/non-finite inputs rejected by name; deterministic given (π, k, π_hi, seed).The frozen-support refit takes the stage-2 support with the fail-closed
len(support) == kgate before and after subsetting, and benchmarks selected records at w/q (Horvitz–Thompson, unbiased for pool mass in design expectation) so the weight-cap's feasible region doesn't punish low-π records — the review's concrete adjudication case (a −16.67% target miss under a uniform rescale becoming +0.07% under w/q) is a committed test. Legacy paths byte-identical throughout.Review trail: 3 adversarial rounds + 3 fix lanes; enumerated-subset χ² verification of the joint law; 40k-case summation audits. Remaining for the increment: the k-ladder launcher + release plumbing, which land against the increment-2 pool once it exists (first ladder: k = N, 57,240, 20,000).
Multispine pool: enforce pre-clone source contracts (prior-year, hous…
UK parity increment 1: frs/cps-transfer tier identity, standardized d…
MaxGhenis commentedon Aug 1, 2026
UK backlog items 1–3 landed (#588, merged 1de826a).
frs/cps-transferenforced fail-closed at every assembly surface (formatter, manifest, publication contract); quality adjectives rejected; canonical idspopulace-uk-<year>-<tier>-k<N>; exact-k ids reconcile record count/year/country across manifest, diagnostics, and target surface; the June release is grandfathered against its REAL schema-2 artifacts (committed real-shape fixtures, all 149 legacy selectors) and lineage-bound (tierfrsor absent —cps-transferon the June id rejects).Review trail: 3 adversarial rounds + 2 fix lanes; the wheel gate caught a missing pytables skip guard post-approval (fixed per the tables-extra convention). Operational note: release automation needs
POPULACE_UK_TERMINAL_GATE_SIGNING_KEYminted before any UK release cut. Items 4–6 (weighted integrity gates, take-up anchors, OA-ladder wiring) are the next UK increments.ACS transfer accepts object-backed semantic booleans; restore hours-w…
Transfer targets must be engine input leaves: drop formula-owned week…
ACS transfer predictors accept semantic booleans; adjudicated null ha…
Consumer-existence guard: every pool input must earn its imputation; …
MaxGhenis commentedon Aug 2, 2026
The pool input surface is now self-auditing (#597, merged 268594a; supersedes #594).
Three CI guards, all in the ordinary secrets-free PR gate, all mutation-authenticated:
Also in #597:
has_marketplace_health_coveragedropped from the plan (engine-deprecated, zero live consumers;_at_interviewemission unchanged) — plan is 114 targets + 3 receipted deferrals.Net effect: an engine pin bump that orphans a variable (as one did
weeks_worked), claims one, or changes a dtype now fails CI by name instead of failing a billed build. Drop receipts and the default-hole complement live in #591.7 remaining items
juaristi22 commentedon Aug 4, 2026
Filed #611: consolidate gating and reporting into one country-agnostic module.
Working item 4 (#609) surfaced a structural finding worth acting on before more gates land: the UK and US have converged on the same discipline by two different mechanisms, and a third country is about to need it.
Three gate-report formats exist today — UK
terminal_gates.json(schema-versioned, HMAC-attested, batched, membership-checked), US national per-gate keys insidecalibration_diagnostics.json(no aggregate verdict, no attestation, no membership check), and US local-areagate_summary.json(retiring withacs_local). The gap is not gate count; the US carries 66 distinct gate objects against the UK's 11. It is that a US gate's identity lives in three hand-maintained places, so nothing forces consolidation. The UK has the better container with fewer things in it, and the US has the better placement — each gate where its evidence appears, which is what lets a build fail before paying for a downstream stage.The reconciliation is a phase field: batched within a phase, fail-closed at the phase boundary.
SourceOperatorContract(multispine_pool.py:324-330) already does exactly this for operators; nobody has applied it to gates.Two things make now the moment rather than later:
build/country_spec.pydefinesGateSelectionSpec(id, gate, criticality, parameters, notes)with anALLOWED_GATE_FUNCTIONSallowlist and per-country selection —test_country_spec.py:122already asserts BE deselectsparity/export_surfacefor having no incumbent. Nothing loads a spec and runs its gates.ALLOWED_GEOGRAPHY_SPINE_METHODSis documented in-file as "the UK OA pattern, generalized," so the module was written country-agnostic and has been waiting for a second consumer.build/be/gates.jsonexists with 9 gates and needs per-country applicability by construction. Consolidating now costs one issue; discovering it afterpopulace-beships costs three migrations.Increment 1 is the shared module with the UK as first consumer, and it is behaviour-preserving: the UK is already
phases = (preflight, terminal), so declaring that changes nothing about when gates run and the PR can assert identical per-gate verdicts against the committed June fixtures. #611 records the full consolidation goal — pool builder, then the national release tool, thengate_summary.jsonretirement — as named later increments.Two design calls in #611 worth flagging here since they touch shared doctrine:
marks_artifactsemantics (H5 written, manifest records the verdict) since its output is an intermediate, so the executor carries both modes.contract.pylearns battery membership from a pinnedspec.fingerprintper country, not a mirrored gate list. It cannot importpopulace-build(dependency cycle, and a verifier that derives expectations from what it verifies checks nothing), and a pinned digest keeps the verifier flat as countries are added. Same doctrine as the existing policy pin — "a threshold outside this hash is not attested" — applied to membership.Against the UK 1–8 backlog: #611 sits under item 3's container and is a prerequisite for doing items 5–8 without re-cutting the battery each time. In particular item 5 (delivered take-up) lands as a declared
not_applicablewith a reason today — the UK has no take-up assignment surface at all (grep takes_upoveruk_runtime/returns nothing) — which turns a silent gap into a receipt. Items 1–4 are unaffected.Canonicalize string dtypes at every pool serialization boundary (#578:…
MaxGhenis commentedon Aug 4, 2026
Increment 2's agreement-gate verdict is in: FAILED — and the failure is informative. Run 7c completed the full pool build (assembly → QRF → banked ACS transfer → derive/seed/simulate → agreement → publication; pool h5 + manifest + agreement diagnostics all on the volume,
simulation_ready: false).104 of 131 registered spine comparisons violate tolerance (incidence ratio bounds [0.8, 1.25], TVD ≤ 0.25, quantile envelope ≤ 0.25). The signature:
puf_tax_itemizationfamily is effectively missing from the ACS spine: ASEC-over-ACS incidence ratios of 160,667× (taxable_interest_income), 9,336× (qualified_dividend_income), 4,169× (STCG), 2,244× (LTCG-before-response) — despite these being declared ACS-transfer targets whose per-target banks completed. The values die somewhere between the transfer bank and the assembled pool, or the transfer produced near-zeros.simulated_output/ssiinverts: ACS incidence ≈ 3.6× ASEC — consistent with hollow ACS income inflating simulated eligibility, i.e. plausibly downstream of the same root cause.A read-only forensic lane is on the artifacts now (bank-vs-pool value tracing for sentinel targets, SSI decomposition, and a review of whether the [0.8, 1.25] contract is the right registration for heavy-tailed QRF-transferred detail). Fix charter + phase-survival analysis (what banked state survives the fix) follows. The k=N ladder build holds until the pool passes.
MaxGhenis commentedon Aug 4, 2026
Increment-2 revision charter: the stacked spine (root cause in hand)
The forensic audit of run 7's agreement failure (full report in the review trail; receipts embedded) found the values never died in transit — the ACS transfer never ran for the PUF family:
acs_interest_dividend_rental_income, nonzero for 16.4% of ACS tax units) sits unused.0.0, so the ACS transfer's null-only activation judges all 58 PUF targets "complete" and silently skips the family — 59 of 117 targets banked, indices 25–76 and 105–110 never requested.Every one of these lives in the two-spine / two-pipeline seam — clone selection ignoring source spine, one pipeline's zeros masquerading as the other's observations. Per Max's direction, the revision removes the seam rather than patching it:
Design: stack, gap-fill, one PUF pass
Gates (the statistics survive; the frame retires)
Execution
Pilot at 10% ACS (fast pipeline validation, agreement-battery verdict same day) → scale per the memory envelope (observed peak 241 GB vs 330 GiB ceiling). Banked-state survival per the audit's matrix: assembled + primary-QRF banks survive where semantics allow via scoped identities; transferred/simulated rebuild (they were wrong anyway).
The stacked spine: one origin-labeled spine, banked gap-fill, null-vs…
MaxGhenis commentedon Aug 6, 2026
The stacked-spine revision is on main: #616 merged (86f6c83) after four adversarial rounds (r1: 3 critical fail-opens + 6 findings; r2: the caller-authority class; fix2: the canonical-authority doctrine; r3: APPROVE with every case re-executed independently and a stronger adversary — rebinding all fourteen module globals — failing closed). The architecture Max ratified is now real: one origin-labeled spine, banked cross-origin gap-fill under declared activation authority, the null-vs-zero doctrine with run 7's failure pinned byte-for-byte as a regression, one uniform PUF pass, the completeness gate, and the by-origin battery with per-column declared metrics under live-digested canonical authority.
Next: the production adoption lane (stacked path into the pool tool, the full battery registry, clone-2 tail, single
sample_fractionwith the #624 rungs), then the first 1% smoke build — local, sub-hour, end-to-end through export.UK parity increment 4: weighted integrity gates (input-mass parity + …
UK carrier swap, increment 1 of #612: carry the national build on pop…
Delete the stacked local-solve research harness (#612 inc 2)
Production adoption: the pool tool builds the stacked spine by default (
Delete the stacked local-solve research harness (#612 inc 2)
State the promotion rule main actually enforces, and record the 18 Se…
Score the UK national candidate on the surface both artifacts can mat…
juaristi22 commentedon Sep 22, 2026
Evaluation rule on the UK national line (microcosm#967, 2026-09-22): rule 1 (#578) is decided on the surface both artifacts can materialize. A target whose measure the incumbent cannot compute is pruned from BOTH arms only under a signed, in-force entry of the reviewed incumbent-unresolvable register (
uk/incumbent_unresolvable_measures.json); an unlisted measure refuses the evaluation. The receipt names every pruned row, its measure and the entry it stands on, and the release-cut certifier re-checks the pruned measures against the committed register. At v21 the six measures the enhanced FRS never carried remove 120 of 644 targets (UC administrative family 93, two-child limit 15, ONS household type 10, CGT asset type 2); the six entries are signed on the register as of 2026-09-22 (a8aa8684on #967), in force to 2026-12-22.(Cross-posted from #823; the register and its refusals live in the UK build package.)
Update (2026-09-22, after the rebase over #965): the signing commit on the branch is
2f97e087; the register ispackages/microcosm-build/src/microcosm/build/uk/incumbent_unresolvable_measures.json.Score the UK national candidate on the surface both artifacts can mat…