Skip to content

Carry SIPP/SCF assets on the declared household head - #1221

Open
MaxGhenis wants to merge 5 commits into
mainfrom
us-wealth-carrier-household-head
Open

MaxGhenis wants to merge 5 commits into
mainfrom
us-wealth-carrier-household-head

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1215 (the selector that contradicts its own declaration; the ownership problem stays open there).

The mismatch

us/source_stages.json declares the wealth stage's carry as head_carry with condition: is_household_head. relationship_inputs.py sets is_household_head from P_SEQ == 1, the CPS reference person, and the housing stages use it as the head. The runtime selector (scf_wealth._household_head_mask, and its SIPP twin sipp_financial_assets._household_head_mask) took the lowest A_LINENO instead, and a test pinned that rule.

On the certified populace-us-2024-spm-20260909 the two rules disagree in 243 of 57,240 households. In the 2024 income year, the one vintage whose CPS file reports A_EXPRRP (the 2022 and 2023 frames synthesize it from line 1), is_household_head picks the reference person in every household, and the lowest-line rule misses 73. For example, in household 884 (#1215), the 71-year-old wife on line 1 carries $17,500 and the line-2 reference person carries nothing.

Change

  • sipp_financial_assets.household_head_carrier_mask picks one carrier per household. Where is_household_head marks exactly one person in the household, it takes that person. Otherwise (no flag column, or zero or several flagged people), it falls back to the lowest A_LINENO, then row order, as before. It reads missing flag values as False and returns an empty mask for an empty frame.
  • scf_wealth._household_head_mask calls it, so the SCF and SIPP halves, and the SCF-wealth gate's audit check, use the same person.

#1215's larger problem stays open: one household-level vector still sits on one person, and fixing it takes a change of draw unit and a validation pass.

Invariants

Hypothesis tests each of these with 100 examples:

  • Exactly one carrier per household, for any mix of flags and line numbers.
  • Where exactly one person in a household carries the flag, the selector picks that person.
  • Permuting rows moves the carrier with its row (line numbers unique within a household, as in the CPS).

Example tests pin that the flag beats a lower line number, that zero or two flags fall back to the lowest line, that both halves return the same mask, and that the selector handles nullable-boolean flags with NA and an empty frame. The existing test_household_head_mask_is_one_per_household_at_lowest_line (no flag column) still passes unchanged.

Deliberate re-pins

The seed protocol attests the source bytes of seeded stage modules, so editing them moves these pins. On main at 3f46d5958, tools/repin_us_spec_engine_contracts.py recomputes them:

Pin Main This PR
EXPECTED_HASHES["seed_protocol"] 5840ac6c…ff53 3e432297573840bbae497370efe3e7cafed5c2c571db029208d10722035a7470
EXPECTED_HASHES["seed_map"] fa48af06…6b37 5f8b7c620abee639d31f3dd50ed61e9c9a0daad89b8b4886d485b71d5cb882bd
Synthetic loader spec_sha256 golden 5b39d83c…d36e ddb925d9603cd269c358f46ff433d5c4adf67991aa17782331a23c0b55767a05
Real US spec_sha256 (engine-workflow test) 4ccbd017…38b2 915d29b0a5419d8e9ff4c9945fb5a92fb18576440dbbbcef2a7680546cfc4441

tools/spec_engine_coverage.py regenerates docs/evidence/spec-engine/us-f0-coverage.json and reports 42,394/42,394 configuration fields and 41/41 inventory checks. The PR changes no RNG rule, seed or QRF target.

Effect

Future builds only. The 243 households' vectors move to their reference person, and both the asset draws and the household net-worth draw condition on that person's predictors (the net-worth QRF reads the same head mask). A frame that already carries an old blend audit can pass through the stage unchanged (the stage's healing rule), so a reused in-memory frame keeps the old carrier; canonical builds draw fresh. The PR runs no release or build.

Release-build rule

docs/us-release-build-rule.md asks a pull request "large enough to change how the dataset is built" to carry a from-scratch build and certification receipt from its own tree before it merges. This PR changes which person carries the asset vectors in future builds, and it carries no such receipt. d2781 asks whether its tests may stand in for that run on the terms of d2647 and d2729: the next certified US build on main would become this PR's from-scratch run.

Verification

At eff5d8343, which merges main at 3f46d5958 and re-pins:

  • The spec-engine loader and field-usage tests, test_us_scf_wealth.py, test_us_sipp_financial_assets.py and the multispine constants test pass (68 tests).
  • ruff check ., tools/ci_test_plan.py verify and verify-workflow, and tools/graph_acceptance_burndown.py --verify pass.
  • tools/repin_us_spec_engine_contracts.py --check timed out in its 300 s clean-worker import while host load sat near 150. The reviewer recomputed all four pins from the head's bytes instead, and CI's engine-us job runs the coverage test.

5932199a5 restores a comment above the seed pins and changes nothing else; the loader tests pass at it (10 tests).

At db53b5f6b, the engine-free US and shared suites passed (-n 4), with the telemetry subprocess test deselected locally because it hangs on this host; CI runs it.

Review

axiom: n/a: microdata construction only; no policy rule changes.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 12, 2026 05:46
us/source_stages.json declares the wealth stage's carry as head_carry with
condition is_household_head (P_SEQ == 1, the CPS reference person), and the
housing stages use that flag. The runtime selector instead took the lowest
A_LINENO. They disagree in 243 of the certified frame's 57,240 households;
on the 2024 income year, where A_EXPRRP is observed, is_household_head is
the reference person in every household and the line rule misses 73.

household_head_carrier_mask (sipp_financial_assets) now uses the flag where
it marks exactly one person in a household and falls back to the lowest
line, then row order, otherwise; scf_wealth's _household_head_mask calls
it, so both halves of the blend carry the same person. Re-pins the
attested digests the change moves: seed protocol and seed map, the
synthetic and real US spec goldens, and the coverage receipt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Only true booleans and nonzero numbers flag a head; strings such as
  "False" or "" never do.
- Flags are counted within the same groups the lowest-line fallback sorts,
  so a missing household id is its own household in both rules.
- New blend-plus-gate test with the declared head on line 2: all three
  leaves land on the head, other members hold zero, and the gate's
  blend-audit checks agree.
- Re-pins the seed protocol and map, synthetic and US spec goldens, and the
  coverage receipt for the new source bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Independent review (GPT-6.1 Sol) of 0b15232c: APPROVE WITH NITS, no blocker; it confirmed caller availability, fresh blend/audit consistency, the four re-pins (all 53 draw-site contracts unchanged), and that the archived pipeline took the first stored row and never minimized A_LINENO. Dispositions in db53b5f:

  • String flags ("False", "") no longer count as heads; only booleans and nonzero numbers do.
  • Flags are counted within the fallback's own sort groups, so a missing household id is its own household in both rules.
  • New blend-plus-gate test with the declared head on line 2 (all three leaves on the head, zeros elsewhere, no blend-audit disagreement).
  • PR body now says net-worth conditioning follows the same head, and that a reused frame with an old audit keeps its old carrier (canonical builds draw fresh).
  • sipp_vehicles still picks the lowest A_LINENO on its own; that predates this PR and is noted on Liquid assets sit on the lowest-line person, against the stage's declared is_household_head carry; 13.1M SSI-relevant adults test at $0 resources by construction #1215.
    Digests re-pinned for the new bytes; spec_engine_coverage --check 41/41; wealth tests 42 passed; loader tests 10 passed.

MaxGhenis and others added 3 commits November 10, 2026 22:50
Main changed the US spec-engine pins in the same four files this branch
re-pinned: us-f0-coverage.json, inventory_coverage.py and the two pin
tests. The merge takes main's side in all four, and the next commit
recomputes the pins with tools/repin_us_spec_engine_contracts.py. Git
merged everything else without conflicts; main has no commits that
touch scf_wealth.py, sipp_financial_assets.py or test_us_scf_wealth.py
since the branch point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the merge, main's pins no longer match this branch's
scf_wealth.py and sipp_financial_assets.py bytes.
tools/repin_us_spec_engine_contracts.py recomputes them:

- EXPECTED_HASHES seed_map and seed_protocol, and the matching rows in
  us-f0-coverage.json;
- the synthetic loader golden and the real US spec_sha256 (915d29b0...).

tools/spec_engine_coverage.py reports 42,394/42,394 configuration
fields and 41/41 inventory checks.

The repinner's first pass also rewrote the coverage test's
covered_item_count from 41 to 40, because it reads that count from a
report built against the stale seed hashes. A second pass, run once the
hashes matched, restored 41 and changed nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The re-pin rewrote the seed_map and seed_protocol values and dropped the
comment that names this PR among the changes that move them. One comment
line now records it beside the other PRs that moved both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

I'm handing this PR to the Microcosm hub (session local_86a67d6b-abf3-47b6-8f7b-6247f14e222a) from session local_ccd7a569-162a-4b4c-a12a-81027d77e07e.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant