Skip to content

PSID income and wealth readers and field registry (D1a) - #575

Merged
MaxGhenis merged 7 commits into
masterfrom
ir/d1a-20261024
Oct 25, 2026
Merged

MaxGhenis merged 7 commits into
masterfrom
ir/d1a-20261024

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This is the first part of milestone D1 of the income-and-resources design (#559: the D1 row of the milestone table): readers for the PSID family-file waves already staged. The wealth supplements and pension-section items come in later parts. The tier counts were updated at the coordinator's slot, and the independent reviews approved (see below).

PSID use follows the project's data-access record. This build read PSID metadata only, namely setup files and codebooks. It opened no microdata and computed no statistic from PSID records.

What it adds

  • src/populace_dynamics/data/psid_income_wealth_registry.py: 1,463 verified fields. Waves covered:
    • income: 1999, 2001, 2003, 2015, 2017, 2019, 2021 and 2023;
    • wealth: 2015, 2017, 2019, 2021 and 2023.
  • What each field records:
    • variable, label and inclusive column positions, from the wave's pinned .sps setup (with its SHA-256);
    • value domain, missing codes, top codes and definition notes;
    • its page in the pinned codebook PDF (with its SHA-256).
  • What it leaves out: codebook frequencies.
  • src/populace_dynamics/data/psid_income_wealth.py: income and wealth readers, labelled like family_income.py. Before reading, they verify the setup bytes, labels and columns, and they refuse unregistered waves, concepts, setups, labels and columns.

Invariants

  • Unregistered waves, concepts, setup bytes, labels and columns raise.
  • Families keep input order and appear exactly once.
  • Documented missing codes, top codes, losses and cents keep their specified handling.
  • Income identities use each wave's own components and the legacy $10 tolerance.
  • WEALTH1 equals assets less debts. WEALTH2 adds home equity.

Open questions

  • Missing annual amounts in early releases. Some early releases lack generated annual SSI amounts and recipient-specific Social Security splits. The raw amounts keep their reporting periods, and the readers refuse unavailable annual concepts.
  • WEALTH2 accuracy descriptions. All five cross-reference WEALTH1 despite their WEALTH2 labels. The flags stay raw, and named alternatives remain open.
  • 2001 aggregate description. Its prose says seven variables but enumerates five. The identity uses the five.
  • Spouse "other assets," 1999 and 2001. These may contain rent; this stays an open source question.
  • Private helpers. The reader imports family_income's private _identity_counts and _normalized helpers. The review should decide whether to make them public.

Tests

Tier Added
unit 94
integration_psid 21
  • Unit tests: 94 passed (-n 4). They build invented records.
  • integration_psid tests:
    • they verify the staged setup files and build invented records from their layouts;
    • they skip cleanly when the metadata root is absent (21 skipped);
    • they passed against the staged metadata in the build.
  • Lint: black -l 79 and ruff are clean.
  • Tier counts: updated at the slot. In tests/tier_counts.json, unit goes from 18,339 to 18,433 and integration_psid from 1,450 to 1,471. The table in tests/README-tiers.md matches (total 25,003).

Since the build

  • 0d704e36 merges origin/master at fe5ab5ab cleanly.
  • 1ec99fb0 keeps the birth-evidence seal. The two new modules are listed in POST_REVIEW_SOURCE_EXCLUSIONS and in the test's expected tuple, and a new assertion checks that neither is reachable from the birth-evidence reducer. That fixes test_reducer_input_identity_matches_reviewed_branch.
  • The seal, U2-isolation and invented-record tests pass locally (111 passed). tests/test_tier_policy.py was expected to fail until the tier-count slot.
  • A fresh independent review (GPT-6.1 Sol) approved 1ec99fb0 with no high or medium findings. Its prescribed runs passed 111 and 21 tests. It judged the private helper imports safe as used.
  • CI at 1ec99fb0 passed lint and the seal. Two failures remained, both expected and outside this PR's code: test_tier_policy, which waits for the tier-count slot, and Track B's holdout-registry append-only check, which waits for Track B's fix on master.

At the slot

  • 4f2dc624 merges master at 9a344a1a cleanly. Its tree equals git merge-tree --write-tree 1ec99fb0 9a344a1a.
  • 5dd8464b recounts the tiers: 94 more unit tests and 21 more integration_psid tests, which are the two new modules. 7ca55400 updates the README tier table to match.
  • CI passes at 7ca55400 (Tests run 40180470423): all four shards, both graph-integration jobs, pytest (3.14), lint and build. The tier-policy test is in the suite and no shard failed.
  • An independent delta check (Opus 5.5) approved 7ca55400. It checked the merge tree, that D1a's own files are unchanged since 1ec99fb0, the counts against a static count of the two modules, the README table and CI's head.

Description updated by the Microcosm Dynamics coordinator at landing (10:59Z, 2026-10-25) to match head 7ca55400; the lane's text is otherwise unchanged.

axiom: n/a: survey-file readers and metadata registry for the microsim; no PolicyEngine country-model or RuleSpec encoding changed

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 24, 2026 11:00
A registry of 1,463 PSID family-file fields across the requested income
waves (1999-2023) and wealth waves (2015-2023): each field's variable,
label and column positions from the wave's pinned setup file, and its
value domain, definition notes and codebook page. It holds no codebook
frequencies. The readers verify setup bytes, labels and columns before
reading, refuse unregistered waves and concepts, and reconcile income
and wealth identities.

PSID use follows the project's data-access record. Built by a Subfleet
build lane in a standalone clone from PSID metadata only; committed by
the coordinating session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests build invented records and check refusals, conservation of
families, missing and top codes, and the income and wealth identities
with Hypothesis. The integration_psid tests verify the staged setup
files and build invented records from their layouts; they skip when
the metadata root is absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
social-security-model Ready Ready Preview Oct 25, 2026 9:24am UTC

Request Review

MaxGhenis and others added 2 commits October 24, 2026 16:52
The PSID income and wealth readers are opt-in source readers that nothing
in the historical projection imports. List their two source files as
post-review exclusions in the birth-evidence script and its expected
tuple, and assert that neither module is reachable from the
birth-evidence reducer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 25, 2026 05:15
D1a adds 94 unit and 21 integration_psid tests in its two PSID income
and wealth modules; its seal edits add no tests. Counted by collecting
those named modules per marker, added to master's counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The table in tests/README-tiers.md now matches tests/tier_counts.json at
this head: unit 18,433, integration_psid 1,471, total 25,003.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 25, 2026 11:00
@MaxGhenis
MaxGhenis merged commit 1ec57a5 into master Oct 25, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the Microcosm Dynamics coordinator at 11:00:20Z on 2026-10-25, as merge commit 1ec57a5d, with --match-head-commit 7ca55400ac0da4e16bb63aa87f85626e33bbdf16. This PR held the tier-count slot.

Gates at the merged head:

  • CI: gh pr checks 575 exited 0. The latest Tests run, 40180470423, is for 7ca55400 and passed: all four shards, both graph-integration jobs, pytest (3.14), lint and build. Marking the PR ready started no new run.
  • Merge state: MERGEABLE and CLEAN. It was not a draft, and no review requested changes.
  • Review: a fresh GPT-6.1 Sol review approved the substantive change at 1ec99fb0 (see the description). An independent Opus 5.5 delta check then approved 7ca55400, after seven checks:
    1. the merge tree equals git merge-tree --write-tree 1ec99fb0 9a344a1a;
    2. after the merge, only the two tier-count files change;
    3. D1a's own files have the same blobs as at 1ec99fb0;
    4. the +94 unit and +21 integration_psid counts match a static count of the two new modules and CI's per-shard totals;
    5. the README table matches tier_counts.json;
    6. CI's run is for this head;
    7. master's later move to 7e050df5 touched only docs/parity-ledger.md.

The description. The coordinator updated it at landing to match the head. The tier counts were described as unchanged, which was no longer true, and the PR was still labelled WIP. The coordinator also added an "At the slot" section. The lane's other text is unchanged.

The slot now passes to #573.

MaxGhenis added a commit that referenced this pull request Oct 25, 2026
Master gained D1a (#575). Both branches extended the birth-evidence
check's post-review exclusions and its reachability test, so the merge
keeps D1a's two PSID income and wealth readers and D4's five
aggregate-table modules, each with its own reachability assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 25, 2026
Master brings #575 (D1a: PSID income and wealth readers). It adds two
new modules under src/populace_dynamics/data/ and changes no existing
source file, so nothing B0.2 imports changes.

The tier file and the README's tier table conflicted. Both are resolved
to master's counts plus this branch's unit delta: 18,433 + 233 = 18,666
unit, with master's integration_psid 1,471, for a total of 25,236. The
branch's README table had not been updated since a3b6236 (18,514);
it now matches the tier file. This is derived; CI's collection checks
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 25, 2026
Master brings #587 (ledger delta), #575 (D1a) and #570 (SM1). None of
them changes the registry module, its tests or its mutation specs, and
the registry module imports only the standard library and yaml.

The tier conflict resolves to master's counts plus this branch's nine
unit tests: unit 18,549 + 9 = 18,558, total 25,128. The README tier
table now matches the tier file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 7ca55400 Deployed Oct 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant