From c904cfa9512d2e06e9643ff3c6ca3042f0edaa50 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 09:58:50 -0400 Subject: [PATCH 01/26] Track B B0.2: plan (d1802) B0.2 freezes the source, exposure, access and power records for B3, B3L, B4, I's observational timing gate, B5-empirical RET, B6a, B6b and J, including J's joint estimands and limits for blinded ratification before B3 registers, and prices the design's still-unpriced items. This first commit sets out the deliverables, the separation from the B2 successor design and the order of work. Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_plan.md | 139 +++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 docs/design/track_b_b0_2_plan.md diff --git a/docs/design/track_b_b0_2_plan.md b/docs/design/track_b_b0_2_plan.md new file mode 100644 index 000000000..45fa75589 --- /dev/null +++ b/docs/design/track_b_b0_2_plan.md @@ -0,0 +1,139 @@ +# Track B B0.2: plan + +**Status: plan, 2026-10-21. This is not an audit record, and no gate can +register on it.** Max ruled in d1802 (2026-10-21) that B0.2 starts now, +on the Track B lane, after the B2 successor memo. This file sets out what +B0.2 must deliver, the order of work and the rules that keep it apart +from the successor design. It changes as the work goes; the audit +records that follow it are the frozen artifacts. + +"The design" means +`microcosm-launch-evidence/dynasim-parity-20260909/trackb-design-20260928.md` +(revision 3), which sits outside this repository. Line numbers refer to +that file. + +## 1. What B0.2 must deliver + +- **One frozen record per gate** (design line 288): B3, B3L, B4, I's + observational timing gate, B5-empirical RET, B6a, B6b and J. Each + record lists the ten §5.2 elements, read as B0.1 read them + (`docs/design/track_b_b0_1_audit.md`, section 2): source release, + variables, population, exposure history, holdout partition, critical + cells, numerical tolerances, uncertainty treatment, pass conjunction, + and reference dates kept separate from interview and publication + availability. Each record also names its power procedure. A gate + cannot register until its record is frozen. +- **A structural-audit protocol frozen before any count or power + summary** (line 286). The record keeps reference dates apart from + availability. Outcomes that were already inspected stay regression + evidence, and "later" or "external" does not by itself establish an + unused holdout. +- **A blinded review of prospective feasibility and power** for the + §5.3 limits (line 120). It may find a limit infeasible before any + candidate result, which leads to prospective rescoping; it may never + loosen a limit after candidate results (line 308). +- **J's joint estimands and numerical limits**, frozen in B0.2 and + ratified by a blinded reviewer before B3 registers (lines 230, 320, + 365-368; Q10, line 615). J's holdout must be independent of B2's + evaluation data. +- **Exposure.** B2's result is recorded as exposed before J's holdout is + chosen (lines 297, 368). So are the SSA Annual Statistical Supplement + pages that Track B participants have already opened (lines 290-295). +- **Pricing of the still-unpriced items** (§3.3, line 198; Q13, line + 618): data access, loaders and gate work for I's observational timing + gate; the HRS loader and access; linked-withholding data access and + loaders for empirical RET; entrant initialization; B6b's access time; + and the 2032 and 2050 horizon extensions. Q4 (line 606) sets the order + for data: public HRS and SSA sources are evaluated first, and + restricted linked data are requested only if public sources cannot + supply the risk denominator, the timing observations or the + withholding observations. +- **B0.1's handoffs** (`docs/design/track_b_b0_1_addendum.md`, "To + B0.2"): + - the marital selectors' pseudo-boundaries 2008 and 2010 bear on B4; + - B3L's 2006 boundary is a q\* and ρ\* pseudo-boundary whose floors + and outcomes are in the ledgers, and protocol v2 projected candidate + 3's law from 2006 to 2008 and 2010 at ages 25-64, which includes + B3L's 62-64 band; + - §5.3 feasibility is reviewed once, for B3 and B3L, with B0.1's + section 11.3 figures as inputs; + - H's spec names and counts its 2010 start roster, and B4's coverage + declaration cites it; + - the addendum's design rule (its section 16, item 10: household + resampling stands only if every cell's design-ratio 95th percentile + is at most 1.10) did not hold at the 2006 pseudo-origin, so the + choice between inflating standard errors and resampling design + clusters arises again for every later gate. +- **B2's medians.** Under d1606 option C1, B2 gates the 90th percentile + and the zero-earnings rate by cohort; its two median cells are + report-only (`src/populace_dynamics/track_b/b2.py`, `GATE_SURFACE` and + `HEADLINE_SCOPE`). B0.2 records which later gate, if any, covers the + medians, and states that nothing downstream may rely on B2 for them. + +## 2. Separation from the successor design + +Under d1659 a B2 successor is designed on evidence independent of B2's +outcome. Its design inputs live in +`microcosm-launch-evidence/dynasim-parity-20260909/trackb-b2-successor-20261018/`. +B0.2 and that lane must not see each other's evidence. + +- No B0.2 lane or reviewer reads the successor folder, or anything + derived from it. +- The successor lane must not read B0.2's counts, planning values or + records. Before any B0.2 file holding real-data counts reaches + master, B0.2 adds an exclusion inventory in the format + `scripts/track_b_b0_1_review_root.py` reads (`q6_exclusions`), which + successor-lane review roots name. +- B2's outcome is not restricted for B0.2. The 2026-10-18 entry in + `RESTRICTED-FILES.md` restricts B2's outcome files for roles building + or proposing a successor blind, and B0.2 must record that exposure. + B0.2's blinded roles still get roots built by the review-root + builder, which always drops the standing B2-outcome record + (`docs/design/track_b_b2_outcome_exclusions.json`); they read B2's + exposure from B0.2's own record. + +## 3. Order of work + +1. **Landscape (no counts).** Separate lanes, none of which sees + another's output before synthesis: + - a repository and staged-data inventory: loaders, staged sources + and what each gate needs against what exists, from listings and + code only, with no survey values read; + - external sources and access: public and restricted HRS, SIPP + public-use and SSA-linked files, SSA public-use files, sources of + withholding and event-timing observations, and inputs for entrant + initialization, each with access steps, lead time, cost and the + gates it could serve; + - a mechanical exposure inventory for the candidate holdouts of each + gate, extending B0.1's inventory approach; + - a per-gate requirements table drawn from the design text. +2. **Protocol.** One B0.2 structural-audit protocol, reviewed by GPT-6.1 + Sol (`--tier hard`), then committed and hash-pinned before any count + runs, as B0.1's section 16 was. +3. **Counts and power.** One-shot count scripts, run once from a clean + worktree at the protocol commit, writing exclusive outputs. Tests + include value-invariance tests in the style of + `test_counts_are_invariant_to_valid_earnings_values`, Hypothesis + property tests and mutation checks. +4. **Records.** The per-gate records, the pricing table and questions + for Max with recommended defaults, queued with `cos add-decision`. +5. **Reviews.** A Sol review of each record; the blinded + feasibility-and-power review; the blinded ratification of J's record + before B3 registers. + +The work may split into several pull requests (plan and protocol, +counts, records). This draft pull request opens with the plan. + +## 4. Rules + +- Design §5.1: no edits to `engine/loop.py`, `engine/steps.py`, + `gates.yaml` or committed `runs/*.json`. +- B0.2 fits no model, runs no projection and computes no candidate + outcome. What it may compute from real data is fixed by its protocol. +- Local runs use at most four BLAS and OpenMP threads with + `OMP_WAIT_POLICY=PASSIVE`, and at most four test workers. +- Any change to `tests/tier_counts.json` waits for a merge slot from the + Microcosm dynamics coordinator. +- B0.2's 5-7 engineer-day estimate (line 120) predates the timing gate + and is not re-priced in the design (line 201). B0.2 reports its actual + effort beside that estimate. From 42ac50fe13d357b2f6525f79c740773441164684 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 10:03:52 -0400 Subject: [PATCH 02/26] B0.2 plan: name the working folder and the lane root Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_plan.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/design/track_b_b0_2_plan.md b/docs/design/track_b_b0_2_plan.md index 45fa75589..1849a0a29 100644 --- a/docs/design/track_b_b0_2_plan.md +++ b/docs/design/track_b_b0_2_plan.md @@ -78,12 +78,16 @@ outcome. Its design inputs live in B0.2 and that lane must not see each other's evidence. - No B0.2 lane or reviewer reads the successor folder, or anything - derived from it. + derived from it. B0.2's own working files live in + `microcosm-launch-evidence/dynasim-parity-20260909/trackb-b0-2-20261021/`, + and its lanes run in a standalone clone with the design and + `RESTRICTED-FILES.md` copied in, and nothing else from the evidence + directory. - The successor lane must not read B0.2's counts, planning values or - records. Before any B0.2 file holding real-data counts reaches - master, B0.2 adds an exclusion inventory in the format - `scripts/track_b_b0_1_review_root.py` reads (`q6_exclusions`), which - successor-lane review roots name. + records, or B0.2's working folder. Before any B0.2 file holding + real-data counts reaches master, B0.2 adds an exclusion inventory in + the format `scripts/track_b_b0_1_review_root.py` reads + (`q6_exclusions`), which successor-lane review roots name. - B2's outcome is not restricted for B0.2. The 2026-10-18 entry in `RESTRICTED-FILES.md` restricts B2's outcome files for roles building or proposing a successor blind, and B0.2 must record that exposure. From e1c3c1e603e21eb6bb1b69825d16fd8ed9105dc7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 10:32:57 -0400 Subject: [PATCH 03/26] B0.2: start the held-out data registry for Track B and open-population gates The dynamics coordinator made B0.2's exposure record the single registry of held-out years for every lane. The registry starts with the five prior uses verified in gates.yaml and b2.py (gate_m6, B2, gate_epuf_fill, gate_m4, gate_w1); B0.2's mechanical inventory adds the rest before any B0.2 record freezes. Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_plan.md | 8 +- docs/design/track_b_holdout_registry.json | 123 ++++++++++++++++++++++ 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 docs/design/track_b_holdout_registry.json diff --git a/docs/design/track_b_b0_2_plan.md b/docs/design/track_b_b0_2_plan.md index 1849a0a29..fe76d89c4 100644 --- a/docs/design/track_b_b0_2_plan.md +++ b/docs/design/track_b_b0_2_plan.md @@ -37,7 +37,13 @@ that file. 365-368; Q10, line 615). J's holdout must be independent of B2's evaluation data. - **Exposure.** B2's result is recorded as exposed before J's holdout is - chosen (lines 297, 368). So are the SSA Annual Statistical Supplement + chosen (lines 297, 368). +- **The held-out data registry.** `docs/design/track_b_holdout_registry.json` + is the single registry of held-out years, editions, tables and + partitions for every Track B gate and every open-population gate + (coordinator, 2026-10-21). J's candidates are registered first; the + open-population lane's OP0.2 registers its picks for OP12 before it + freezes them; no unit is held out by two gates silently. So are the SSA Annual Statistical Supplement pages that Track B participants have already opened (lines 290-295). - **Pricing of the still-unpriced items** (§3.3, line 198; Q13, line 618): data access, loaders and gate work for I's observational timing diff --git a/docs/design/track_b_holdout_registry.json b/docs/design/track_b_holdout_registry.json new file mode 100644 index 000000000..e827c2ae5 --- /dev/null +++ b/docs/design/track_b_holdout_registry.json @@ -0,0 +1,123 @@ +{ + "schema": "track_b_holdout_registry.v1", + "title": "Held-out data registry for Track B and the open-population lane", + "purpose": "The single registry of held-out source years, editions, tables and partitions for every Track B gate and every open-population gate. It is part of B0.2's exposure record (design revision 3, section 5.2, elements 4 and 5). The dynamics coordinator set it up on 2026-10-21. Cite it by path, commit and SHA-256.", + "rules": [ + "A gate registers every source year, edition, table or partition it holds out here before its record freezes.", + "J's candidate holdouts are registered first, because J is on Track B's critical path. OP0.2 registers its picks for OP12's gate_ssa_totals here before it freezes them, and picks from the years J has not taken.", + "No unit may be held out by two gates silently. If two entries share a unit, each lists the other's id in disclosed_overlaps, and both gates' registrations name the overlap.", + "A unit is one source at one reference year, or one edition's table, or one partition. Reference years and publication or interview availability are recorded separately (section 5.2, element 10).", + "An entry whose gate has scored has status 'used'. Its units are exposure for every later gate and never again an unused holdout (section 5.2: previously inspected outcomes remain regression evidence).", + "Entries are only appended. A status change is recorded in the entry's history, with its date and reason. Ids never change or get reused.", + "This registry starts with the uses verified below. It is not yet complete: B0.2's mechanical exposure inventory adds the remaining prior uses before any B0.2 gate record freezes." + ], + "status_values": { + "candidate": "proposed by a gate's B0.2 or OP0.2 record; not yet frozen", + "frozen": "fixed in a frozen gate record before that gate's candidates run", + "used": "scored by a registered gate; exposure for every later gate", + "withdrawn": "released before use, with the reason recorded" + }, + "entries": [ + { + "id": "H001", + "gate": "gate_m6", + "lane": "M6 (locked gate)", + "status": "used", + "source": "PSID family and cross-year individual files", + "unit_kind": "reference_years", + "units": { + "boundary_T_star": 2014, + "held_out_window": "2015-2022", + "gated_earnings_reference_years": [2016, 2018], + "gated_flow_event_years": [2015, 2016, 2017, 2018, 2019] + }, + "evidence": [ + "gates.yaml:5374-5376 (fit <= T*=2014; score the held-out PSID window 2015-2022)", + "gates.yaml:5496-5501 (fit_holdout)" + ], + "disclosed_overlaps": [], + "history": [ + {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32"} + ] + }, + { + "id": "H002", + "gate": "B2", + "lane": "Track B", + "status": "used", + "source": "PSID family files FAM2013ER and FAM2015ER, with the cross-year individual file", + "unit_kind": "reference_years", + "units": { + "reference_years": [2012, 2014], + "collection_waves": [2013, 2015], + "ages": "25-64 at the collection wave", + "population": "2011 full-anchor persons with a valid 2010 row" + }, + "evidence": [ + "src/populace_dynamics/track_b/b2.py:64-65", + "docs/design/track_b_b0_1_audit.md, section 16 (definitions)", + "registration: issue #42 comment 6252388925" + ], + "disclosed_overlaps": [], + "history": [ + {"date": "2026-10-21", "status": "used", "note": "B2 scored on 2026-10-18; seeded at 1e739d32"} + ] + }, + { + "id": "H003", + "gate": "gate_epuf_fill", + "lane": "EPUF career fill (locked gate)", + "status": "used", + "source": "SSA 2006 Earnings Public-Use File (EPUF)", + "unit_kind": "person_partition", + "units": { + "partition": "EPUF2006_TEST_by_salted_person_hash" + }, + "evidence": [ + "gates.yaml:5918 (holdout_basis)" + ], + "disclosed_overlaps": [], + "history": [ + {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; open_population.md rev 2 plans gate_young_earn on a new, disjoint salt"} + ] + }, + { + "id": "H004", + "gate": "gate_m4", + "lane": "M4 disability (locked gate)", + "status": "used", + "source": "PSID cross-year individual file employment status; SSA Annual Statistical Report on the SSDI Program, 2023", + "unit_kind": "publication_edition_tables", + "units": { + "holdout_basis": ["ind2023er_employment_status", "di_asr_2023"] + }, + "evidence": [ + "gates.yaml:2945 (holdout_basis)" + ], + "disclosed_overlaps": [], + "history": [ + {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; tables to be itemized by B0.2's inventory"} + ] + }, + { + "id": "H005", + "gate": "gate_w1", + "lane": "W1 frame transport (locked gate)", + "status": "used", + "source": "SSA Annual Statistical Supplement 2023, section 6.B; SSA Annual Statistical Report on the SSDI Program, 2023; Census CPS tables", + "unit_kind": "publication_edition_tables", + "units": { + "holdout_basis": ["populace_us_2024_certified", "ssa_supplement_6b", "di_asr_2023", "mermin_smith_committed_orderings"], + "tables_named": ["Supplement 2023 table 6.B5.1", "Supplement 2023 tables 6.B4 and 6.B3", "DI ASR 2023 table 19", "Census HH-4, reference year 2023", "Census AD-3, reference year 2023"] + }, + "evidence": [ + "gates.yaml:3735-3736 (holdout_basis)", + "gates.yaml:4582-4584 (sources)" + ], + "disclosed_overlaps": [], + "history": [ + {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; whether each named table was gated or report-only is itemized by B0.2's inventory"} + ] + } + ] +} From 530ac6961edf23c702b6d3f76014ff8eda35f2b2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 10:41:31 -0400 Subject: [PATCH 04/26] B0.2 plan: the held-out data registry moves to its own PR (#560) Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_plan.md | 12 ++- docs/design/track_b_holdout_registry.json | 123 ---------------------- 2 files changed, 7 insertions(+), 128 deletions(-) delete mode 100644 docs/design/track_b_holdout_registry.json diff --git a/docs/design/track_b_b0_2_plan.md b/docs/design/track_b_b0_2_plan.md index fe76d89c4..63dda3cba 100644 --- a/docs/design/track_b_b0_2_plan.md +++ b/docs/design/track_b_b0_2_plan.md @@ -39,12 +39,14 @@ that file. - **Exposure.** B2's result is recorded as exposed before J's holdout is chosen (lines 297, 368). - **The held-out data registry.** `docs/design/track_b_holdout_registry.json` - is the single registry of held-out years, editions, tables and - partitions for every Track B gate and every open-population gate - (coordinator, 2026-10-21). J's candidates are registered first; the + (PR #560, on its own so that every lane can cite it on master) is the + single registry of held-out years, editions, tables and partitions for + every Track B gate and every open-population gate (coordinator, + 2026-10-21). J's candidates are registered there first; the open-population lane's OP0.2 registers its picks for OP12 before it - freezes them; no unit is held out by two gates silently. So are the SSA Annual Statistical Supplement - pages that Track B participants have already opened (lines 290-295). + freezes them; no unit is held out by two gates silently. B0.2's + mechanical exposure inventory, `docs/design/track_b_b0_2_exposure_inventory.json`, + completes it. - **Pricing of the still-unpriced items** (§3.3, line 198; Q13, line 618): data access, loaders and gate work for I's observational timing gate; the HRS loader and access; linked-withholding data access and diff --git a/docs/design/track_b_holdout_registry.json b/docs/design/track_b_holdout_registry.json deleted file mode 100644 index e827c2ae5..000000000 --- a/docs/design/track_b_holdout_registry.json +++ /dev/null @@ -1,123 +0,0 @@ -{ - "schema": "track_b_holdout_registry.v1", - "title": "Held-out data registry for Track B and the open-population lane", - "purpose": "The single registry of held-out source years, editions, tables and partitions for every Track B gate and every open-population gate. It is part of B0.2's exposure record (design revision 3, section 5.2, elements 4 and 5). The dynamics coordinator set it up on 2026-10-21. Cite it by path, commit and SHA-256.", - "rules": [ - "A gate registers every source year, edition, table or partition it holds out here before its record freezes.", - "J's candidate holdouts are registered first, because J is on Track B's critical path. OP0.2 registers its picks for OP12's gate_ssa_totals here before it freezes them, and picks from the years J has not taken.", - "No unit may be held out by two gates silently. If two entries share a unit, each lists the other's id in disclosed_overlaps, and both gates' registrations name the overlap.", - "A unit is one source at one reference year, or one edition's table, or one partition. Reference years and publication or interview availability are recorded separately (section 5.2, element 10).", - "An entry whose gate has scored has status 'used'. Its units are exposure for every later gate and never again an unused holdout (section 5.2: previously inspected outcomes remain regression evidence).", - "Entries are only appended. A status change is recorded in the entry's history, with its date and reason. Ids never change or get reused.", - "This registry starts with the uses verified below. It is not yet complete: B0.2's mechanical exposure inventory adds the remaining prior uses before any B0.2 gate record freezes." - ], - "status_values": { - "candidate": "proposed by a gate's B0.2 or OP0.2 record; not yet frozen", - "frozen": "fixed in a frozen gate record before that gate's candidates run", - "used": "scored by a registered gate; exposure for every later gate", - "withdrawn": "released before use, with the reason recorded" - }, - "entries": [ - { - "id": "H001", - "gate": "gate_m6", - "lane": "M6 (locked gate)", - "status": "used", - "source": "PSID family and cross-year individual files", - "unit_kind": "reference_years", - "units": { - "boundary_T_star": 2014, - "held_out_window": "2015-2022", - "gated_earnings_reference_years": [2016, 2018], - "gated_flow_event_years": [2015, 2016, 2017, 2018, 2019] - }, - "evidence": [ - "gates.yaml:5374-5376 (fit <= T*=2014; score the held-out PSID window 2015-2022)", - "gates.yaml:5496-5501 (fit_holdout)" - ], - "disclosed_overlaps": [], - "history": [ - {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32"} - ] - }, - { - "id": "H002", - "gate": "B2", - "lane": "Track B", - "status": "used", - "source": "PSID family files FAM2013ER and FAM2015ER, with the cross-year individual file", - "unit_kind": "reference_years", - "units": { - "reference_years": [2012, 2014], - "collection_waves": [2013, 2015], - "ages": "25-64 at the collection wave", - "population": "2011 full-anchor persons with a valid 2010 row" - }, - "evidence": [ - "src/populace_dynamics/track_b/b2.py:64-65", - "docs/design/track_b_b0_1_audit.md, section 16 (definitions)", - "registration: issue #42 comment 6252388925" - ], - "disclosed_overlaps": [], - "history": [ - {"date": "2026-10-21", "status": "used", "note": "B2 scored on 2026-10-18; seeded at 1e739d32"} - ] - }, - { - "id": "H003", - "gate": "gate_epuf_fill", - "lane": "EPUF career fill (locked gate)", - "status": "used", - "source": "SSA 2006 Earnings Public-Use File (EPUF)", - "unit_kind": "person_partition", - "units": { - "partition": "EPUF2006_TEST_by_salted_person_hash" - }, - "evidence": [ - "gates.yaml:5918 (holdout_basis)" - ], - "disclosed_overlaps": [], - "history": [ - {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; open_population.md rev 2 plans gate_young_earn on a new, disjoint salt"} - ] - }, - { - "id": "H004", - "gate": "gate_m4", - "lane": "M4 disability (locked gate)", - "status": "used", - "source": "PSID cross-year individual file employment status; SSA Annual Statistical Report on the SSDI Program, 2023", - "unit_kind": "publication_edition_tables", - "units": { - "holdout_basis": ["ind2023er_employment_status", "di_asr_2023"] - }, - "evidence": [ - "gates.yaml:2945 (holdout_basis)" - ], - "disclosed_overlaps": [], - "history": [ - {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; tables to be itemized by B0.2's inventory"} - ] - }, - { - "id": "H005", - "gate": "gate_w1", - "lane": "W1 frame transport (locked gate)", - "status": "used", - "source": "SSA Annual Statistical Supplement 2023, section 6.B; SSA Annual Statistical Report on the SSDI Program, 2023; Census CPS tables", - "unit_kind": "publication_edition_tables", - "units": { - "holdout_basis": ["populace_us_2024_certified", "ssa_supplement_6b", "di_asr_2023", "mermin_smith_committed_orderings"], - "tables_named": ["Supplement 2023 table 6.B5.1", "Supplement 2023 tables 6.B4 and 6.B3", "DI ASR 2023 table 19", "Census HH-4, reference year 2023", "Census AD-3, reference year 2023"] - }, - "evidence": [ - "gates.yaml:3735-3736 (holdout_basis)", - "gates.yaml:4582-4584 (sources)" - ], - "disclosed_overlaps": [], - "history": [ - {"date": "2026-10-21", "status": "used", "note": "seeded from gates.yaml at 1e739d32; whether each named table was gated or report-only is itemized by B0.2's inventory"} - ] - } - ] -} From 207f408efdeb47110a5c7adf6f8c664c55a53d6a Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 06:46:57 -0400 Subject: [PATCH 05/26] B0.2: landscape synthesis of the five lanes Sources per gate (staged, missing, external candidates and their access), the exposure catalogue and every accidental exposure the lanes recorded, the per-gate requirements and J's candidate estimands, H's roster options, pricing of the design's still-unpriced items (entrant initialization cited from the open-population lane), what must be verified before the protocol relies on it, and the scope of protocol version 1 (structural counts on staged PSID). Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_landscape.md | 282 ++++++++++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 docs/design/track_b_b0_2_landscape.md diff --git a/docs/design/track_b_b0_2_landscape.md b/docs/design/track_b_b0_2_landscape.md new file mode 100644 index 000000000..51aa1758a --- /dev/null +++ b/docs/design/track_b_b0_2_landscape.md @@ -0,0 +1,282 @@ +# Track B B0.2: landscape + +**Status: synthesis of five landscape lanes, 2026-10-23. It computes +nothing and freezes nothing.** It feeds B0.2's structural-audit +protocol, which is frozen before any count, and the questions for Max. + +Five read-only lanes ran on 2026-10-21 and 2026-10-22, as set out in the +B0.2 plan (`docs/design/track_b_b0_2_plan.md`, section 3, step 1): + +| Lane | Subject | Output | +|---|---|---| +| a1 | Repository and staged-data inventory | `lanes/a1-repo-inventory-out.md` | +| a2 | Microdata sources and access | `lanes/a2-microdata-access-out.md` | +| a3 | Public aggregate and administrative sources | `lanes/a3-aggregate-sources-out.md` | +| a4 | Exposure catalogue for each gate's candidate holdouts | `lanes/a4-exposure-catalog-out.md` | +| a5 | Per-gate requirements and J's joint record | `lanes/a5-gate-requirements-out.md` | + +The outputs are in +`microcosm-launch-evidence/dynasim-parity-20260909/trackb-b0-2-20261021/`. +No lane read survey microdata, computed a count or ran a script. Where a +lane could not open a source page, it marked the fact **UNVERIFIED**, and +this synthesis keeps that label. + +## 1. What each gate can draw on now + +**Staged and readable on this machine** (a1, section 1): +- PSID family files 1968-2023, the cross-year individual file, the + marriage history, childbirth history and relationship matrix; +- SSA's 2006 Earnings Public-Use File (EPUF), hash-pinned; +- committed SSA, Trustees, CBO, NCHS and DI ASR tables under + `data/external/`. + +**Not staged:** SIPP, CPS ASEC and CPS tenure files. **Absent:** any HRS +file or HRS loader, and any reader for SSA's Benefits and Earnings +Public-Use File (BEPUF), the Master Beneficiary Record or another +SSA-linked file. + +| Gate | What exists | What is missing | +|---|---|---| +| B3 | Head and spouse earnings at all ages from the family files; odd income years; EPUF through 2006; B2's and M6's cell and support helpers | A cell family for ages 62-69; other family members' earnings; an HRS loader if B3 uses HRS; evidence of an unused partition | +| B3L | The panel through income year 2022; boundary and origin already parameters | A 2006-origin domain reaching ages 62-69; everything B3 lacks | +| B4 | Marriage histories and transitions; candidate 16's port; roster-only widowhood; the DI engine; self-reported disability; deaths by year; forward support | A person-level administrative DI holdout; deaths separated from attrition; widowhood linked to the spouse's death; entrants | +| I's timing gate | Marriage months; DI award year; aggregate tables by month of entitlement; a SIPP job-month reader (data not staged) | Dated person-level death, award, claim and conversion events; a monthly-receipt loader; an event journal | +| B5-empirical | R1, R2 and R3 statutory modules; annual PSID Social Security amounts | Any withholding or deduction observation; RET-countable earnings | +| B6a | A marginal claiming distribution; first observed receipt year; receipt history; years of coverage | Insured status; an insured, unclaimed person-month risk set with entitlement months | +| B6b | The FRA schedule and B6a's pieces | Identifying policy variation, and B6a's gaps | +| J | Self-reported benefit types; opening status; spouse links; the G layer; exposed aggregate SSA tables | Estimands and limits; a holdout independent of B2; a joint truth builder; administrative payment and recipient denominators | + +**External candidates** (a2, a3), all still to be chosen: + +| Source | Strongest use | Access | Status | +|---|---|---|---| +| HRS public files and the RAND HRS longitudinal file | B3 at ages 62-69; B4 deaths separated from attrition; self-reported claim and DI dates for I; joint states for J | Registration; whether an agent lane may work on files a person registers is UNVERIFIED | Pages refused automated fetches | +| HRS files linked to SSA records | Administrative earnings, insured status and entitlement dates for B3, B6a and I | Virtual-desktop enclave run by a named person; 6-8 week decision (search snippet); likely an institutional sponsor and IRB | UNVERIFIED | +| SIPP public-use panels | Monthly receipt, earnings and marital status | Public download | Variables verified through the Census API | +| SIPP Synthetic Beta, validated on the Gold Standard File | Insured status and claim timing for older cohorts, to 2014 | Application approved in about five business days; real results only through Census validation | Verified on census.gov | +| FSRDC files linking SIPP, CPS or ACS to SSA records | Everything, including possible deduction fields | Person with Special Sworn Status on site; 6-12 months; $15,000-20,000 a year per project | Verified on the SAP catalog and RDC pages | +| BEPUF 2020 | Linked earnings and benefit records, for J and recipient-side checks | Public CC0 download | Field list UNVERIFIED: the user guide was not opened | +| Disability Analysis File | Monthly disability participation, with birth and death dates | "Restricted public" | Access UNVERIFIED | +| SSA Supplement, DI ASR, OACT series, Monthly Snapshot | Aggregate stocks, flows, awards, withholding (6.E) | Public | Section and table indexes verified; several sections already exposed (section 2) | +| NCHS life tables, mortality files and HMD | B4 mortality | Public; HMD needs registration | HMD and NCHS share vital-registration inputs, so they are not independent of each other | +| ACS and CPS marital and earnings tables | B4 state shares; B3 employment margins | Public | Metadata verified | + +**Gaps no verified public source closes:** +- the insured, unclaimed person-month risk set (B6a); +- observed annual deductions and withholding (B5-empirical); +- a complete joint holdout for J. + +Each of these needs either a restricted linked source or the design's +narrower fallback: +- B6a: a marginal claiming distribution, under `NO_RISK_DENOMINATOR`; +- B5: statutory conformance and sensitivities only; +- J: `JOINT_VALIDATION_FAIL` or a declared narrower scope. + +## 2. Exposure + +**Prior uses recorded by a4.** Each gate's candidate data has already +been used as follows. None of these can be an unused holdout (design +section 5.2). + +- **M6 candidates and floors:** + - PSID earnings reference years 2016 and 2018, anchored 2014; + - flows 2015-2019; + - diagnostics on earnings 2020 and 2022 and flows 2020 and 2021; + - cohorts 25-44 and 45-64. + + These bear on B3 (its 62-64 band), B3L, B4 and J. +- **The q\* and ρ\* selectors:** pseudo-boundaries 2006, 2008 and 2010, + scored on reference years b, b + 2 and b + 4 at ages 25-64. These bear + on B3L and J. +- **B0.1's protocol v2:** fitted at origin 2006 and projected to 2008 + and 2010, ages 25-64, with derived summaries published. This bears on + B3L's 62-64 band. +- **Marital selectors:** first marriage and remarriage at boundaries 2006, + 2008 and 2010. This bears on B4 and J. +- **M4 disability floors:** PSID waves through 2023, ages 20-66. +- **Mortality floors:** waves 1969-2023, ages 25-85+. +- **The DI-entitlement diagnostic:** validated against the DI ASR 2023. +- **The claiming reference:** fitted 1998-2019, evaluated 2020-2022. +- **EPUF and `gate_epuf_fill`:** earnings 1951-2006, ages 22-74, with DEV + and TEST scores published. +- **B2:** PSID income years 2012 and 2014, cohorts 25-44 and 45-64; gated + p90 and zero-earnings rate. +- **The SSA Supplement:** design participants opened 2023 §6.B, 2024 §4.B + and §6.E, and 2025 §6.E, including numeric tables (design lines + 290-297). +- **SSA tables committed in the repository:** a1 adds those not on the + design's list: + - Supplement 6.B for 2014, 2025 and 2026; + - the 2025 4A, 4B, 5A and 6A snapshots; + - 2008 4C; + - DI ASR tables for 2007, 2008 and 2023; + - 2024 Supplement captures staged under + `psid-data/legal-capture-staging/`. + +The held-out data registry (PR #560, +`docs/design/track_b_holdout_registry.json`) records the scored gates +among these. B0.2's mechanical exposure inventory completes it. + +**HRS exposure is UNVERIFIED.** No executed HRS exercise was found, but +issue and PR threads were not searched. + +**Accidental exposures during the lanes.** Each lane recorded them. No +value was copied into any output. +- a1 and a2: source files print microdata or codebook counts (for + example `deaths.py`'s codebook tallies and `epuf.py`'s row counts). The + lanes read them and copied nothing. +- a3: four web pages showed numbers before the lane stopped reading: + - the Monthly Statistical Snapshot for September 2026; + - the DI ASR 2025 highlights; + - an OACT chart's alternative text; + - an NSFG methods page. + + A numeric erratum also appeared on the 2024 Supplement contents page, + and some search snippets showed outcomes. +- a4: a content search with no exclusions (`git grep -il HRS`) read the + restricted B2 outcome files' bytes. It displayed file names only. + Masking also failed on several files, showing outcome counts or figures: + - `docs/amendments/gate_epuf_fill_registration_proposal.md`; + - `docs/design/di_entitlement.md`; + - comments in `harness/m6_cells.py`; + - assertions in `scripts/build_m6_holdout_floors_v4.py`. +- a5: excerpts of the B0.1 audit and addendum showed planning and power + figures, and `b2.py:229-235` showed inherited outcome text. + +These lanes are author-side landscape roles, not blinded reviewers, so +none can serve as J's blinded reviewer. B0.2's exposure record carries +each incident. The exposed web pages (the Snapshot for September 2026 +and the DI ASR 2025 highlights) are exposure for any gate that would use +them. + +**What the mechanical inventory must add** (a4, section "A B0.2 +mechanical inventory"): +- **A candidate registry per gate:** source, reference year, interview + wave, event period, age, partition, denominator, concept and estimand. +- **Exclusions first:** apply them before any content is read. +- **Pattern families:** B0.1's families, plus families for earnings, + marital events, disability, mortality, claiming, RET, dates and joint + states. +- **Classification of each artifact,** separately by evidence content + and by role. Content is one of names or rules, observed values, + projected values, derived values, transient computation, or + unverified. Role is one of training, selection, floor, development, + diagnostic, regression or holdout. +- **Integers and code in scope:** cover hard-coded counts and code, not + only decimals. +- **An off-repository register** supplied by outcome-aware holders, for + issue and PR threads and external directories. + +## 3. Requirements per gate + +a5 sets out, for each gate, the ten section 5.2 elements and the power +procedure. For each element it gives what the design fixes, what B0.1's +rulings carry over, and the open options with a recommended default. +Its summary table marks almost every element of every gate "open: needs +Max" or "open: needs counts". Only exposure history is "carried over". + +Its main recommendations: +- **A common power procedure,** for Max's approval: prospective + whole-gate equivalence power. The operating characteristic is the + probability that a faithful candidate passes the entire registered + conjunction, evaluated over a frozen grid of nuisance assumptions, with + a conservative dependence treatment and finite-design and estimation + uncertainty included. +- **Permissible structural counts and forbidden outcome disclosures,** + per gate. a5's table "What the structural-audit protocol should allow + and forbid" becomes the protocol's core. Missingness and availability + flags can reveal outcomes (a "date available" flag can identify an + event), so the protocol must test that no permitted output reconstructs + a target truth cell. B0.1's death-disposition counts were structural + for B2 but would reveal part of B4's mortality surface. +- **J:** six candidate joint estimands (J1-J6): + - J1, a common-date earnings, claiming and DI table; + - J2, the earnings-claiming association by DI history; + - J3, joint transition pathways; + - J4, recipients and payment within joint states; + - J5, survivor-path cumulative incidence; + - J6, post-bereavement payment persistence. + + Each has limit options drawn from the design's styles (2-3 percentage + points for probabilities, 5-10% for amounts). a5 also states what + blinded ratification must certify. +- **B2's medians:** ask Max whether to assign a separately named + cohort-median component to B3, with matching regression cells in B3L, + or to leave medians unadmitted. The current p50 is a weighted + positive-earner median. No gate holds ages 25-61 earnings to a fixed + limit (`b2.py:215-216`). +- **The single §5.3 feasibility review for B3 and B3L:** its inputs and + procedure are set out (a5, final section). It uses B0.1 audit section + 11.3's figures only through a cleared review process. + +**H's start roster** (a1, section 4). No H specification exists. Three +rosters are candidates: +- B2's 2010 head-and-spouse domain, defined in code; +- the full 2011 anchor roster; +- Track A's 2011 closed cohort (`cohorts/psid2010.py`). + +Each changes B4's coverage declaration. The choice is a question for Max +or for H's design, and the count runs under B0.2's protocol. + +## 4. Pricing of the design's still-unpriced items + +These are estimates in engineer-days (d) of reviewed output, or calendar +time, with their basis in the cited lane: + +| Item (design §3.3) | Access | Loader and gate work | Source | +|---|---|---|---| +| HRS loader and access, public | About one week of registration; agent use of a person's registration UNVERIFIED | Loader 4-8 d; B0.2 adds 1-2 d to freeze HRS elements per gate | a1 4-7 d, a2 5-8 d | +| HRS linked to SSA records | 2-4 months, about 6 if a sponsor is needed; 1-3 weeks of disclosure review per output | 6-10 d by a person in the enclave | a2 | +| Linked withholding data (B5-empirical) | Public: none, if BEPUF 2020 carries a deduction field (UNVERIFIED). Otherwise HRS-SSA (2-4 months) or an FSRDC (6-12 months, $15,000-20,000 a year) | 1 d documentation check; then 8-12 d to rebuild annual deductions; 4-6 d to freeze and run | a2 | +| I's timing gate: data, loaders, gate | Public HRS and SIPP: about one week; restricted fallback as above | SIPP extension 3-5 d; BEPUF 3-4 d; freezing estimands, limits and power 3-5 d; running the gate 4-6 d. About 10-20 d beyond B0.2's own 5-7 d | a1, a2 | +| Entrant initialization | Public ACS, CPS and EPUF | **Cited from the open-population lane: OP8, 25-39 d; 34-52 d with the demographic packets** (draft, PR #557 at `2acd0bed`, pending its review) | `inputs/entrant-initialization.md` | +| B6b access | Public HRS about one week; restricted 2-4 months; FSRDC 6-12 months | Shared with the HRS rows | a2 | +| 2032 and 2050 horizons | — | Not re-priced here. Design B priced its B8 at 40-70 d, for scale only (design line 203). The open-population lane prices its economic horizon separately (OP10b, OP11b; draft) | design; #557 | + +The insured-status rule that B6a, entrant initialization and OP12 need is +on the rules lane under d1802. a1's 2-3 d estimate for it is for scale +only. + +## 5. What must be verified before the protocol relies on it + +These are facts the lanes could not open. Verify them in a browser, +which works where automated fetches are refused, and open no numeric +table: +1. **BEPUF 2020's field list** (its user guide): entitlement dates, + payment status, work-deduction or withholding fields, deaths, spouse + links. This decides whether a public source can serve B5-empirical, I + and J. +2. **HRS terms:** whether an agent lane may analyse public files that a + person registered, and the restricted-data access route, eligibility + and lead time. +3. **The Disability Analysis File's access route.** +4. **Two staged files that may be restricted** (a1, section 7). Do not + open them until someone rules: + - `psid-data/validation-sources/urban-favreault2007-311436.pdf`; + - the `ssa-mint-*` captures. + +## 6. What the protocol can count, and what waits for Max + +PSID is the only staged source whose structure B0.2 can count now. So +protocol version 1 covers structural counts on the staged PSID files for: +- B3: support at ages 62-69 by wave, households and design units; +- B3L: the 2006-origin roster and later availability; +- B4: the opening roster, linkage coverage and design fields; +- H's candidate rosters; +- J: PSID-side linkage coverage outside B2's evaluation households. + +It applies a5's forbidden-disclosure lists. Gates with no staged source +(I, B5-empirical, B6a, B6b) get a source decision first. Their protocol +sections follow once Max chooses sources and the access questions above +are answered. + +Questions for Max, gathered from a2 and a5, go to him through +`cos add-decision` when the records are written. They include: +- the common power procedure; +- each gate's source; +- whether to pursue restricted HRS-SSA or FSRDC access; +- HRS terms for agent work; +- H's roster; +- B2's medians; +- J's estimands and limits; +- the percentage-error conventions. From b6a403123f5af5b81c4166bb258d0a340e8d56cf Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 06:49:54 -0400 Subject: [PATCH 06/26] B0.2 landscape: BEPUF 2020 is fully synthetic, so it cannot be an observed holdout Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_landscape.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/design/track_b_b0_2_landscape.md b/docs/design/track_b_b0_2_landscape.md index 51aa1758a..8cd10c952 100644 --- a/docs/design/track_b_b0_2_landscape.md +++ b/docs/design/track_b_b0_2_landscape.md @@ -55,12 +55,15 @@ SSA-linked file. | SIPP public-use panels | Monthly receipt, earnings and marital status | Public download | Variables verified through the Census API | | SIPP Synthetic Beta, validated on the Gold Standard File | Insured status and claim timing for older cohorts, to 2014 | Application approved in about five business days; real results only through Census validation | Verified on census.gov | | FSRDC files linking SIPP, CPS or ACS to SSA records | Everything, including possible deduction fields | Person with Special Sworn Status on site; 6-12 months; $15,000-20,000 a year per project | Verified on the SAP catalog and RDC pages | -| BEPUF 2020 | Linked earnings and benefit records, for J and recipient-side checks | Public CC0 download | Field list UNVERIFIED: the user guide was not opened | +| BEPUF 2020 | Structural and recipient-side sanity checks only | Public CC0 download | **Fully synthetic** (section 5): not observed data, so not a holdout for J, B5-empirical or I | | Disability Analysis File | Monthly disability participation, with birth and death dates | "Restricted public" | Access UNVERIFIED | | SSA Supplement, DI ASR, OACT series, Monthly Snapshot | Aggregate stocks, flows, awards, withholding (6.E) | Public | Section and table indexes verified; several sections already exposed (section 2) | | NCHS life tables, mortality files and HMD | B4 mortality | Public; HMD needs registration | HMD and NCHS share vital-registration inputs, so they are not independent of each other | | ACS and CPS marital and earnings tables | B4 state shares; B3 employment margins | Public | Metadata verified | +BEPUF 2020 was the strongest public candidate for J. It is fully +synthetic (section 5), which leaves these gaps. + **Gaps no verified public source closes:** - the insured, unclaimed person-month risk set (B6a); - observed annual deductions and withholding (B5-empirical); @@ -242,10 +245,19 @@ only. These are facts the lanes could not open. Verify them in a browser, which works where automated fetches are refused, and open no numeric table: -1. **BEPUF 2020's field list** (its user guide): entitlement dates, - payment status, work-deduction or withholding fields, deaths, spouse - links. This decides whether a public source can serve B5-empirical, I - and J. +1. **BEPUF 2020: settled, 2026-10-23.** SSA's landing page + (https://www.ssa.gov/policy/docs/microdata/bepuf-2020/index.html, opened + in a browser; it showed no outcome figures) describes the file as "fully + synthetic microdata records statistically representing a self-weighted + 10 percent sample of the current adult OASDI beneficiary population as + of December 2020". It says each record's earnings history and benefit + credited are related "by statistical modeling rather than by applying + Social Security program rules", and cautions against using them + together for individual-level analysis. A synthetic file is not + observed data, so it cannot serve as an observed holdout for J, + B5-empirical or I's timing gate, whatever its fields. The user guide + PDF did not render as text in the browser, and its field list is + unread; nothing above depends on it. 2. **HRS terms:** whether an agent lane may analyse public files that a person registered, and the restricted-data access route, eligibility and lead time. From 8225ae817d6018dbb5d9994c1713a605c17f4edf Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 06:52:43 -0400 Subject: [PATCH 07/26] B0.2: structural-audit protocol, version 1 (draft, not frozen) Structural counts on the staged PSID for H's candidate rosters, B3's support at ages 62-69 with lag pairs, B3L's 2006-origin roster and later support, and the persons and households outside B2's evaluation units. Reads roster, age, relationship, weight, design and labor-income validity only; counts no dispositions, transitions or outcomes. Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_protocol.md | 210 +++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/design/track_b_b0_2_protocol.md diff --git a/docs/design/track_b_b0_2_protocol.md b/docs/design/track_b_b0_2_protocol.md new file mode 100644 index 000000000..d4bb7f683 --- /dev/null +++ b/docs/design/track_b_b0_2_protocol.md @@ -0,0 +1,210 @@ +# Track B B0.2: structural-audit protocol + +**Status: version 1, DRAFT, 2026-10-23. Not yet frozen. No count has run.** + +Design section 5.2 (line 286) says B0's structural-audit protocol is +frozen before any count or power summary is computed. This file is that +protocol for B0.2. Version 1 covers only what the staged PSID can +answer. It is the first count step the landscape synthesis identifies +(`docs/design/track_b_b0_2_landscape.md`, section 6). + +A later version, recorded as such, will cover the other sources (HRS, +SIPP, SSA tables) once Max chooses them. + +The protocol is frozen by the commit that adds the counting script +`scripts/track_b_b0_2_counts.py`, its tests, and this file with the +block below unchanged. The unit test `tests/test_track_b_b0_2_counts.py` +pins the block's SHA-256, taken over the text from the begin marker +through the end marker. The script then runs once, from a clean worktree +at that commit. + + +## Structural-audit protocol, version 1 (frozen before any count) + +### Scope + +Version 1 counts the structure of the staged PSID for four purposes: + +1. **H's candidate start rosters.** B4's coverage declaration cites H's + roster (B0.1 addendum, "To B0.2"). +2. **B3's support at ages 62-69.** These are B3's bands, design line 127. +3. **B3L's 2006-origin roster and its later support.** B3L is the + 2006→2022 backtest, design line 128. +4. **The PSID persons and households outside B2's evaluation units.** + J needs a holdout independent of B2's evaluation data (design lines + 365-368). + +It computes no power quantity. The power procedure waits for Max's +ruling on it. + +### What this audit may and may not compute + +**It may compute:** +- counts of persons, of households (interview numbers) and of survey + design strata and clusters; +- counts of valid head and spouse labor-income rows, and of same-person + valid-row pairs at fixed lags; +- Kish and cluster-worst effective sizes from weights; +- SHA-256 hashes of the files it opens. + +**It may not:** +- read or compute any labor-income level, sign, zero or positive + indicator, quantile or change; +- read any marital, marriage-history, death, disability, + employment-status, Social Security, wealth or other income variable; +- count dispositions that separate death from attrition, or any + transition; +- compute any planning value, power probability, model fit, projection, + floor, gap or candidate outcome. + +These rules apply the forbidden-disclosure list in landscape lane a5 +("What the structural-audit protocol should allow and forbid") to the +PSID. In particular, for B3 and B3L no earnings sign, positive-pair +count, participation, transition, crossing, quantile, cumulative-earnings +or persistence value is computed. For B4, no marital state, transition, +DI status, death, survival or widowhood frequency is computed. For J, no +joint state, recipient or payment value is computed. + +### Inputs + +1. **The staged PSID directory,** resolved by the repository's own rule + (`populace_dynamics.data.psid`). +2. **The cross-year individual file `IND2023ER`,** read through + `panels.ind_person_period` with exactly the concept labels B0.1's + counting script uses for age, sequence number, relationship to the + reference person, individual weight and interview number + (`scripts/track_b_b0_1_counts.py`, `_individual_variables`). It covers + collection waves 1999 to 2023, every second year, plus 2007, 2009 and + 2011 where those are not already included. +3. **The 2011 anchor-wave variables,** read through + `cohorts.psid2010.read_anchor_wave(anchor_wave=2011)`: interview, + sequence, relationship, reported age and the 2011 cross-sectional + weight. Its `reported_birth_year` column is dropped as soon as it is + read. +4. **The survey design variables,** read through + `cohorts.age67.read_design_variables`: sampling error stratum + (ER31996) and cluster (ER31997). +5. **The family files for collection waves 1999 to 2023,** read through + `family.read_family_labor` with its label verification, then reduced + at once by B0.1's `validity_frame` + (`scripts/track_b_b0_1_counts.py`), which is imported unchanged. + +No other file or variable is opened. The script records the label of +every variable it requests, and a test pins that list as an allowlist. + +### Reduction at read time + +Each family file's labor-income levels are replaced by one flag per +role, "below the PSID missing sentinel", using B0.1's `validity_frame`. +The accuracy codes are replaced by "positive". No later step sees a +level. + +A unit test replaces every valid level with an arbitrary valid level and +requires every output to stay byte-identical. A second unit test adds +arbitrary extra columns of a forbidden kind to the input frames (marital +status, death year, Social Security amount) and requires every output to +stay byte-identical. + +### Definitions + +- **Collection wave and reference year.** Collection wave *w* collects + reference year *w* − 1. Ages are collection-wave ages, as in B2 + (`src/populace_dynamics/track_b/b2.py`, `cohort`). The B3 and B3L + records may choose another age convention; version 1 counts at + collection-wave age and says so. +- **Valid row.** At wave *w*, a person present (sequence 1-20) who is the + head or reference person, or the wife, spouse or partner, under that + wave's relationship codes, with a positive weight at *w*, and whose + role's labor income is below the sentinel. This is B0.1's + `earnings_rows` rule, imported unchanged. +- **Age bands.** + - Older bands: 62-64, 65-66, 67-69 (B3's bands). + - Context bands: 25-44, 45-61. + - Roster bands: 0-24, 25-44, 45-61, 62-64, 65-66, 67-69, 70-79, 80 and + over. +- **Rosters at the 2011 wave** (H's candidates): + - `R_FULL`: persons present (sequence 1-20) with a positive individual + weight at 2011. This is B0.1's full anchor. + - `R_B2`: B2's domain, `R_FULL` persons with a valid 2010-reference row + (`b2.domain_person_ids`). The script asserts that its size equals + B0.1's committed count (`docs/design/track_b_b0_1_counts.json`); + this is a consistency check only. + - `R_TA_PROXY`: persons present at 2011 with a positive 2011 + cross-sectional weight and a reported 2011 age of 30 or over. + - It is a proxy for Track A's closed cohort (persons born in 1980 or + earlier, `cohorts/psid2010.py`). + - Track A's own birth-year law reads marriage histories and earnings, + so version 1 does not reproduce it. The H record cites Track A's + own record for the exact cohort. +- **Roster counts.** For each roster, overall and by roster band at 2011: + - persons; + - households (2011 interview numbers); + - design strata and clusters; + - Kish effective size; + - cluster-worst effective size, with households as clusters, and again + with design clusters as clusters. + + Weights are the roster's own weight: individual weight for `R_FULL` + and `R_B2`, the cross-sectional weight for `R_TA_PROXY`. +- **B3 support.** For each wave *w* from 1999 to 2023 and each older + band: + - valid rows, persons and households; + - design strata and clusters; + - Kish and cluster-worst effective sizes, using the wave's individual + weight; + - the number of those rows whose accuracy code is positive (a + data-quality count). + + Across waves: distinct persons per older band. +- **B3 lag pairs.** For lags of 2 and 4 years, and for each pair of waves + (*w*, *w* + lag) inside 1999-2023, count persons with a valid row at + both waves, by older band at the later wave. These are structural + support for transition and persistence estimands; no earnings value + enters. +- **B3L roster and later support.** + - `R2006_FULL`: persons present with a positive individual weight at + the 2007 wave. + - `D2006`: those `R2006_FULL` persons with a valid row at 2007 (income + year 2006). + - Roster counts for both, as above, by roster band at 2007. + - Then, for each later wave *w* from 2009 to 2023: the number of + `D2006` persons with a valid row at *w*, by band at collection-wave + age (context and older bands), and the weighted Kish size of those + rows using `D2006`'s 2007 weight. + - No disposition (death, move-out, nonresponse) is counted. +- **Outside B2's evaluation units.** + - B2's evaluation units are the `R_B2` persons and every 2011 household + containing one. + - Count the `R_FULL` persons outside those households, overall and by + roster band: persons, households, design strata and clusters, and + Kish and cluster-worst effective sizes. + - Separately, at waves 2013 and 2015, count the households (wave + interview numbers) containing at least one `R_B2` person. These + count households, not outcomes. +- **Effective sizes.** Kish size (Σw)² / Σw². Cluster-worst size + (Σw)² / Σ_c W_c², where W_c is cluster *c*'s total weight. These are + B0.1's `kish_n_eff` and `cluster_worst_n_eff`, imported unchanged. A + set with no positive weight reports 0. + +### Output and stop rules + +- **The output.** The script writes one exclusive file, + `docs/design/track_b_b0_2_counts.json`, with schema + `track_b_b0_2_counts.v1`. It refuses to overwrite a file, or to write + under `runs/`. The record carries: + - the repository head; + - the script's SHA-256; + - every opened PSID file's SHA-256; + - the requested-label allowlist. +- **One run.** It runs once, from a clean worktree at the commit that + freezes this protocol, with at most four BLAS and OpenMP threads and + `OMP_WAIT_POLICY=PASSIVE`. If it fails, the failure is published, and + the protocol is revised in a new, dated version before any rerun. +- **No changes after counts.** No definition above changes once counts + are seen. A change is a new protocol version, recorded as such, with + the reason. +- **Separation.** Before the counts file reaches master, an exclusion + inventory naming it, in `scripts/track_b_b0_1_review_root.py`'s + `q6_exclusions` format, is committed, so that successor-design review + roots can exclude it (B0.2 plan, section 2). + From b491f4b6bca5c93dffc21efcb22182939f1724e1 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 06:54:22 -0400 Subject: [PATCH 08/26] B0.2 landscape: the restricted linked-data decision, which cannot wait for counts Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_landscape.md | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/design/track_b_b0_2_landscape.md b/docs/design/track_b_b0_2_landscape.md index 8cd10c952..fa76c36a2 100644 --- a/docs/design/track_b_b0_2_landscape.md +++ b/docs/design/track_b_b0_2_landscape.md @@ -292,3 +292,52 @@ Questions for Max, gathered from a2 and a5, go to him through - B2's medians; - J's estimands and limits; - the percentage-error conventions. + +## 7. A decision needed now: restricted linked data + +Design Q4 (line 606, approved in d515) sets the order: evaluate public +HRS and SSA sources first, and request restricted linked data "only if +B0 shows that public sources cannot supply the risk denominator, the +timing observations or the withholding observations". + +The landscape shows that condition is met for three of those needs: +- **No verified public source supplies B6a's risk denominator:** an + insured, unclaimed person-month risk set with entitlement dates. + Public HRS does not observe insured status (a2, section 2.1). +- **No verified public source supplies observed withholding or annual + deductions for B5-empirical.** Supplement table 6.E4 is a December + status diagnostic (design line 224), and BEPUF 2020 is synthetic + (section 5). +- **No verified public source supplies J's joint holdout,** with + observed payment and recipient denominators. + +Restricted access is slow: +- HRS linked to SSA records: about 2-4 months, through a named + person's virtual desktop in an enclave, probably with an institutional + sponsor (UNVERIFIED); +- an FSRDC: 6-12 months and $15,000-20,000 a year per project, with a + person holding Special Sworn Status working on site (verified on the + RDC pages). + +These lead times would fall on the path to B6a and J, and so to B7. +Neither route can be worked by an agent lane: a person has to do the +analysis inside the enclave or data center. + +The options: +1. **Apply for HRS-SSA access now,** in parallel with B0.2. It supplies + insured status, entitlement dates and administrative earnings. It + leaves B5's withholding UNVERIFIED (it depends on the benefit file's + detail). +2. **Apply for an FSRDC project now.** This is the only verified route to + Master Beneficiary Record payment data. Whether that data carries + work-deduction fields is UNVERIFIED. +3. **Both.** +4. **Neither, and accept the design's fallbacks:** + - B5 publishes statutory conformance and sensitivities only; + - B6a keeps a marginal claiming distribution (`NO_RISK_DENOMINATOR`); + - J is narrowed to what PSID or public HRS can observe, or stays + blocked, and B7 is limited accordingly. + +Public HRS registration is cheap, takes about a week, and is needed +under every option. Whether an agent lane may work on files a person +registers is UNVERIFIED and should be confirmed with HRS. From 6e6deb9356e5e346ef87b52ece9089ee9e8e9713 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 06:58:11 -0400 Subject: [PATCH 09/26] B0.2 protocol v1 draft: add the small-cell disclosure rule (d1899) Committed counts from 1 to 9 become "<10", with complementary suppression; unsuppressed counts stay outside the repository. Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_protocol.md | 29 +++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/docs/design/track_b_b0_2_protocol.md b/docs/design/track_b_b0_2_protocol.md index d4bb7f683..7fed90e3b 100644 --- a/docs/design/track_b_b0_2_protocol.md +++ b/docs/design/track_b_b0_2_protocol.md @@ -186,11 +186,38 @@ stay byte-identical. B0.1's `kish_n_eff` and `cluster_worst_n_eff`, imported unchanged. A set with no positive weight reports 0. +### Disclosure rule + +PSID's Conditions of Use, as agreed at the July 2026 download, allow +"aggregate statistical reporting" only. Max's ruling d1899 adds "no +small-cell counts on GitHub". So the committed output follows these +rules: + +- **Suppression.** Any count from 1 to 9 is written as the string `"<10"`. + So is every effective size computed on a set whose row or person count + is below 10. A count of 0 is written as 0. +- **Complementary suppression.** Where a total and all but one of its + parts are published, a suppressed part could be recovered by + subtraction. In that case the smallest remaining published part of + the same total is also suppressed, and this is repeated until no + suppressed cell can be recovered from published cells of the same + table. +- **Local copy.** The unsuppressed counts are written to a second file, + outside the repository, in B0.2's evidence folder. That file is never + committed. +- **Tests.** Unit tests check both rules on invented tables, including a + table where subtraction would reveal a suppressed cell. + +The threshold of 10 is version 1's. If Max or the coordinator sets +another before the freeze, this section changes before it freezes; after +the freeze, a change is a new version. + ### Output and stop rules - **The output.** The script writes one exclusive file, `docs/design/track_b_b0_2_counts.json`, with schema - `track_b_b0_2_counts.v1`. It refuses to overwrite a file, or to write + `track_b_b0_2_counts.v1`. It holds the suppressed counts, under the + disclosure rule. The unsuppressed copy goes outside the repository. It refuses to overwrite a file, or to write under `runs/`. The record carries: - the repository head; - the script's SHA-256; From be7d4f42db2f3dc1057a6ed20494bde483689213 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 08:22:02 -0400 Subject: [PATCH 10/26] B0.2: the structural count script, with the small-cell disclosure rule scripts/track_b_b0_2_counts.py implements protocol v1 (draft) as pinned: the block's SHA-256 is 94e22e3fff7b8ec541f87abe9890270a9893e0ec1653ce7b91fb0170b2732cb9. - Counts: the H rosters (R_FULL, R_B2, R_TA_PROXY), B3 support at ages 62-69 with lag-2 and lag-4 pairs, B3L's 2006 origin (R2006_FULL, D2006) and the units outside B2, by age band. - Disclosure (d1899): counts 1-9, and effective sizes on sets below ten, are written "<10"; complementary suppression repeats to a fixed point over every encoded total-and-parts relation, checked by exact rational elimination. The committed file holds suppressed values and a disclosure block only. The unsuppressed copy goes to a required --local-copy path outside the repository; both files are created exclusively. Built blind on invented frames by a Subfleet lane; no data was read and no count has been run. The protocol is not frozen: the Sol review and the freeze come before the one count run. Tests: 59 unit (81 passed with the 22 B0.1 count tests). Mutations: 10 of 10 killed. Black 26.5.1 (-l 79) and ruff clean. Tier counts: unit 9,958 -> 10,017. Co-Authored-By: Claude Opus 5.5 --- .../design/track_b_b0_2_counts_mutations.json | 199 +++ docs/design/track_b_b0_2_counts_report.txt | 122 ++ scripts/track_b_b0_2_counts.py | 882 ++++++++++ tests/test_track_b_b0_2_counts.py | 1565 +++++++++++++++++ tests/tier_counts.json | 2 +- 5 files changed, 2769 insertions(+), 1 deletion(-) create mode 100644 docs/design/track_b_b0_2_counts_mutations.json create mode 100644 docs/design/track_b_b0_2_counts_report.txt create mode 100644 scripts/track_b_b0_2_counts.py create mode 100644 tests/test_track_b_b0_2_counts.py diff --git a/docs/design/track_b_b0_2_counts_mutations.json b/docs/design/track_b_b0_2_counts_mutations.json new file mode 100644 index 000000000..28a01c993 --- /dev/null +++ b/docs/design/track_b_b0_2_counts_mutations.json @@ -0,0 +1,199 @@ +{ + "schema_version": 1, + "verification_class": "structural-count protocol conformance", + "admitted_scope": "Invented frames only; no real-data run.", + "mutation_method": "One deliberate production-source change per check; targeted pytest run serially; original source bytes restored and SHA-256 verified after every check, including failures. No mutated implementation is committed.", + "protocol_sha256": "94e22e3fff7b8ec541f87abe9890270a9893e0ec1653ce7b91fb0170b2732cb9", + "restored_source_sha256": { + "scripts/track_b_b0_2_counts.py": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb" + }, + "result": "10/10 mutations killed", + "checks": [ + { + "mutation": "band edges", + "old": "OLDER_BANDS = ((\"62_64\", 62, 64), (\"65_66\", 65, 66), (\"67_69\", 67, 69))", + "new": "OLDER_BANDS = ((\"62_64\", 63, 64), (\"65_66\", 65, 66), (\"67_69\", 67, 69))", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_band_edges_and_b3_support_are_hand_counted", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_band_edges_and_b3_support_are_hand_counted" + ], + "summary": "1 failed in 26.99s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-01.log", + "elapsed_seconds": 41.794 + }, + { + "mutation": "lag set", + "old": "LAGS = (2, 4)", + "new": "LAGS = (2, 6)", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_b3_lag_pairs_use_exact_lags_and_later_wave_band", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_b3_lag_pairs_use_exact_lags_and_later_wave_band" + ], + "summary": "1 failed in 8.08s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-02.log", + "elapsed_seconds": 11.5 + }, + { + "mutation": "R_TA_PROXY age threshold", + "old": " proxy = proxy[proxy[\"age\"] >= 30]", + "new": " proxy = proxy[proxy[\"age\"] > 30]", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_rosters_and_age_30_proxy_boundary_use_their_own_weights_and_ages", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_rosters_and_age_30_proxy_boundary_use_their_own_weights_and_ages" + ], + "summary": "1 failed in 7.17s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-03.log", + "elapsed_seconds": 9.096 + }, + { + "mutation": "outside-B2 household exclusion", + "old": " outside = full[~full[\"interview\"].isin(domain[\"interview\"])]", + "new": " outside = full[~full[\"person_id\"].isin(domain[\"person_id\"])]", + "anchor_occurrences": 2, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_outside_b2_excludes_entire_anchor_households_and_counts_later_units", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_outside_b2_excludes_entire_anchor_households_and_counts_later_units" + ], + "summary": "1 failed in 15.74s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-04.log", + "elapsed_seconds": 19.557 + }, + { + "mutation": "valid-row rule", + "old": " rows[\"valid\"], [\"person_id\", \"reference_year\", \"acc_flag\"]", + "new": " rows[\"weight_positive\"], [\"person_id\", \"reference_year\", \"acc_flag\"]", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_valid_rows_require_present_role_positive_weight_and_role_validity", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_valid_rows_require_present_role_positive_weight_and_role_validity" + ], + "summary": "1 failed in 8.79s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-05.log", + "elapsed_seconds": 10.805 + }, + { + "mutation": "B3L origin wave", + "old": "B3L_ORIGIN_WAVE = 2007", + "new": "B3L_ORIGIN_WAVE = 2009", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_b3l_uses_2007_origin_and_fixed_origin_weights_for_later_support", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_b3l_uses_2007_origin_and_fixed_origin_weights_for_later_support" + ], + "summary": "1 failed in 14.51s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-06.log", + "elapsed_seconds": 20.153 + }, + { + "mutation": "disclosure threshold changed from 10 to 9", + "old": "DISCLOSURE_THRESHOLD = 10", + "new": "DISCLOSURE_THRESHOLD = 9", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q 'tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_counts_one_through_nine[9]'", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_counts_one_through_nine[9]" + ], + "summary": "1 failed in 7.53s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-07.log", + "elapsed_seconds": 8.845 + }, + { + "mutation": "complementary suppression skipped", + "old": " equations = [_equation(relation) for relation in relations]", + "new": " equations = [_equation(relation) for relation in relations]\n equations = [] # Deliberately skip complements.", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_disclosure_complements_the_smallest_remaining_published_part", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_disclosure_complements_the_smallest_remaining_published_part" + ], + "summary": "1 failed in 26.43s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-08.log", + "elapsed_seconds": 42.835 + }, + { + "mutation": "effective sizes on small sets left unsuppressed", + "old": "elif key != \"origin_wave\": # geometry, not a count", + "new": "elif key != \"origin_wave\" and not key.endswith(\"_n_eff\"): # MUTANT", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[0.0-persons]", + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[0.0-valid_rows]", + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[8.125-persons]", + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[8.125-valid_rows]", + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[12.0-persons]", + "tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets[12.0-valid_rows]" + ], + "summary": "6 failed in 32.28s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-09.log", + "elapsed_seconds": 35.882 + }, + { + "mutation": "local copy allowed inside repository", + "old": "raise SystemExit(\"local copy must be outside the repository\")", + "new": "pass # Deliberately permit repository-local copies.", + "anchor_occurrences": 1, + "replacement_occurrence": 1, + "command": "OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_main_refuses_local_copy_inside_repository_before_read", + "exit_code": 1, + "failed_tests": [ + "tests/test_track_b_b0_2_counts.py::test_main_refuses_local_copy_inside_repository_before_read[False]", + "tests/test_track_b_b0_2_counts.py::test_main_refuses_local_copy_inside_repository_before_read[True]" + ], + "summary": "2 failed in 13.59s", + "status": "killed", + "restored": true, + "restored_source_sha256": "15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb", + "log_path": "/private/tmp/b02-disclosure-revision/mutation-10.log", + "elapsed_seconds": 16.765 + } + ] +} diff --git a/docs/design/track_b_b0_2_counts_report.txt b/docs/design/track_b_b0_2_counts_report.txt new file mode 100644 index 000000000..2c885d03f --- /dev/null +++ b/docs/design/track_b_b0_2_counts_report.txt @@ -0,0 +1,122 @@ +Track B B0.2 counts: disclosure revision, protocol version 1 + +Changes + Updated the four pre-existing uncommitted files: count script, tests, + mutations JSON, and this report. Raw structural definitions/readers and + existing tests are retained. No commit or push was made. + After counting, counts 1-9 and effective sizes on nonempty sets below + ten are replaced by "<10". Positive effective sizes below ten are also + suppressed conservatively even on larger sets. Zero counts and empty-set + effective sizes remain zero. Complementary person suppression also hides + that set's effective sizes, which could otherwise reveal its count. + Complementary suppression repeats to a fixed point. Exact rational + elimination also checks recovery from combined equations; it chooses + one smallest relevant published cell before recomputing that closure. + The public payload has only suppressed counts, plus a disclosure block + with threshold, rule, primary cell count, each complementary pass's cell + count, and the complementary total. It contains no private unit sets, + unsuppressed copy, or local path. + Default public destination: docs/design/track_b_b0_2_counts.json. + Required --local-copy supplies an external evidence-folder destination + for unsuppressed JSON. Repository-local paths (including symlink aliases) + are refused before input resolution. Relative external paths work. + Both files use exclusive creation, refusing existing files and dangling + symlinks. If the local path appears during reading, its bytes survive + and the newly reserved empty public file is removed. + +Encoded total/parts and duplicate-cell relations + Every output count is registered as a private set of its counting units. + Exact incidence identities encode all additive relations among these + cells; unit sets and coefficients are never written to either output. + Explicit table registrations cover: + - Overall versus eight age bands for R_FULL, R_B2, R_TA_PROXY, + R2006_FULL, D2006, and outside-B2. A sum partition is recorded only + when parts cover the total without overlap. This applies to persons, + households, strata and composite design clusters; missing ages do not + create an assumed exhaustive partition. + - R_FULL overall households = R_B2 overall households + outside-B2 + overall households, at the 2011 anchor. Persons are not assumed to + have that partition: non-B2 co-residents can occupy B2 households. + - B3 pooled distinct persons versus the union of wave counts for each + older band, together with lag intersections. Overlaps use exact + inclusion/exclusion identities, never an assumed sum across waves. + - Roster/subset overlaps, accuracy-positive subsets of valid rows, + both lag sets, origin/later D2006 support, and later B2-household + counts. No unreported remainder count is added to the output. + - B3 valid_rows = persons; R_B2 older-band unit counts = B3 2011; + D2006 older-band unit counts = B3 2007; D2006 support at 2009 and + 2011 = the respective 2007-origin lag-2 and lag-4 person counts. + Household identities include collection wave, and design clusters + include stratum. Distinct unit unions may overlap. Effective sizes + are not added across cells, and differing weight definitions remain + separate. + +Protocol pin + 94e22e3fff7b8ec541f87abe9890270a9893e0ec1653ce7b91fb0170b2732cb9 + SHA-256 covers exact bytes from through + inclusive, excluding the following newline. + The protocol itself is unchanged. No freeze or real-data run occurred. + +Validation (serial; commands omit log-only tee wrappers) + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py tests/test_track_b_b0_1_counts.py + 81 passed in 200.55s (0:03:20) + 59 B0.2 cases and 22 unchanged B0.1 cases, exit 0. + Includes threshold boundaries, small/empty effective sizes, smallest + complement, second-pass chain, combined-equation recovery, overlapping + pools, exact identities, no positive published statistic below ten, + path guards, exclusive-file races, and the revised protocol pin. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m black --version + python -m black, 26.5.1 (compiled: yes) + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m black --check -l 79 scripts/track_b_b0_2_counts.py tests/test_track_b_b0_2_counts.py + 2 files would be left unchanged. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m ruff check scripts/track_b_b0_2_counts.py tests/test_track_b_b0_2_counts.py + All checks passed! + +Mutation checks (six existing plus four disclosure mutations) + 1. band edges: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_band_edges_and_b3_support_are_hand_counted + 1 failed in 26.99s + 2. lag set: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_b3_lag_pairs_use_exact_lags_and_later_wave_band + 1 failed in 8.08s + 3. R_TA_PROXY age threshold: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_rosters_and_age_30_proxy_boundary_use_their_own_weights_and_ages + 1 failed in 7.17s + 4. outside-B2 household exclusion: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_outside_b2_excludes_entire_anchor_households_and_counts_later_units + 1 failed in 15.74s + 5. valid-row rule: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_valid_rows_require_present_role_positive_weight_and_role_validity + 1 failed in 8.79s + 6. B3L origin wave: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_b3l_uses_2007_origin_and_fixed_origin_weights_for_later_support + 1 failed in 14.51s + 7. disclosure threshold changed from 10 to 9: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q 'tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_counts_one_through_nine[9]' + 1 failed in 7.53s + 8. complementary suppression skipped: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_disclosure_complements_the_smallest_remaining_published_part + 1 failed in 26.43s + 9. effective sizes on small sets left unsuppressed: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets + 6 failed in 32.28s + 10. local copy allowed inside repository: detected, exit 1. + OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE PYTHONPATH=$PWD/src:$PWD PYTHONDONTWRITEBYTECODE=1 /Users/maxghenis/PolicyEngine/microcosm-dynamics/.venv/bin/python -m pytest -p no:xdist -p no:cacheprovider -q tests/test_track_b_b0_2_counts.py::test_main_refuses_local_copy_inside_repository_before_read + 2 failed in 13.59s + Final runner line: 10/10 mutations killed + Every mutation restored the exact production bytes, SHA-256 verified: + 15faf990bc1645c3a48968b111f2a46a0bedec3eb671a79d55fe6764695794eb + The effective-size mutation leaves effective sizes raw in the final + published payload. The local-copy mutation reaches only mocked refusal + tests. Old/new text, failed nodes, exit codes, commands, final lines and + restoration digests are recorded in the mutations JSON. + +No-data and integrity declaration + No data file or codebook was opened. No real-data count, power quantity, + or default counts JSON was generated. All numerical tests used invented + frames; reader IO and B0.1 consistency counts were mocked. + No restricted evidence directory, restricted file, or other Track B + workspace was opened. No package was installed. No protocol, gates.yaml, + runs/, tests/tier_counts.json, existing src/ file, or other tracked file + was edited. The assigned branch remains trackb-b0-2-20261021 at + 6e6deb9356e5e346ef87b52ece9089ee9e8e9713. All four files remain uncommitted. diff --git a/scripts/track_b_b0_2_counts.py b/scripts/track_b_b0_2_counts.py new file mode 100644 index 000000000..40ceb3efc --- /dev/null +++ b/scripts/track_b_b0_2_counts.py @@ -0,0 +1,882 @@ +#!/usr/bin/env python3 +"""Track B B0.2: structural counts under protocol version 1. + +Run once from the clean protocol-freezing commit. Family values are +immediately reduced by B0.1's validity_frame; all counting functions +accept frames and inspect only structural columns. Collection-wave ages +are used throughout; reference year is collection wave minus one. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import subprocess +from collections.abc import Mapping +from copy import deepcopy +from fractions import Fraction +from pathlib import Path +from typing import Any + +import pandas as pd + +from populace_dynamics.cohorts import age67, psid2010 +from populace_dynamics.data import family, panels, psid +from scripts.track_b_b0_1_counts import ( + _family_variables, + _individual_variables, + cluster_worst_n_eff, + earnings_rows, + kish_n_eff, + validity_frame, +) + +ROOT = Path(__file__).resolve().parents[1] +RUNS_DIR = ROOT / "runs" +DEFAULT_OUT = ROOT / "docs/design/track_b_b0_2_counts.json" +B0_1_COUNTS = ROOT / "docs/design/track_b_b0_1_counts.json" +SCHEMA = "track_b_b0_2_counts.v1" +DISCLOSURE_THRESHOLD = 10 +SUPPRESSED = "<10" +CellPath = tuple[str, ...] +Relation = tuple[CellPath, tuple[CellPath, ...]] | Mapping[CellPath, int] +WAVES = tuple(range(1999, 2024, 2)) +LAGS = (2, 4) +ANCHOR_WAVE = 2011 +B3L_ORIGIN_WAVE = 2007 +OLDER_BANDS = (("62_64", 62, 64), ("65_66", 65, 66), ("67_69", 67, 69)) +CONTEXT_BANDS = (("25_44", 25, 44), ("45_61", 45, 61)) +ROSTER_BANDS = ( + ("0_24", 0, 24), + *CONTEXT_BANDS, + *OLDER_BANDS, + ("70_79", 70, 79), + ("80_plus", 80, float("inf")), +) +PERSON_COLUMNS = ( + "person_id", + "period", + "age", + "sequence", + "relationship", + "weight", + "interview", +) +INDIVIDUAL_CONCEPTS = ( + "age", + "sequence", + "relationship", + "weight", + "interview", +) + + +# -------------------------------------------------------------------------- +# Pure structural counting +# -------------------------------------------------------------------------- +def valid_rows( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], +) -> pd.DataFrame: + """Apply B0.1's valid-row law, then attach wave weights/households.""" + people = person_wave[list(PERSON_COLUMNS)] + rows = earnings_rows( + people, validity_by_wave, (wave - 1 for wave in WAVES) + ) + selected = rows.loc[ + rows["valid"], ["person_id", "reference_year", "acc_flag"] + ].copy() + selected["period"] = selected.pop("reference_year") + 1 + return people.merge( + selected, on=["person_id", "period"], validate="one_to_one" + ) + + +def _present(frame: pd.DataFrame) -> pd.DataFrame: + return frame.loc[ + frame["sequence"].between(1, 20) & (frame["weight"] > 0) + ].copy() + + +def _full(person_wave: pd.DataFrame, wave: int) -> pd.DataFrame: + return _present( + person_wave.loc[person_wave["period"] == wave, list(PERSON_COLUMNS)] + ) + + +def _with_design(frame: pd.DataFrame, design: pd.DataFrame) -> pd.DataFrame: + return frame.merge( + design[["person_id", "stratum", "cluster"]], + on="person_id", + how="left", + validate="many_to_one", + ) + + +def _summary(frame: pd.DataFrame) -> dict[str, Any]: + """Count persons and units; effective sizes use positive weights only. + + Sampling-error clusters are identified within stratum, as in age67. + An empty positive-weight subset has effective sizes zero, without + changing B0.1's helpers (which reject supplied nonpositive weights). + """ + positive = frame[frame["weight"] > 0] + weights = positive["weight"].to_numpy(dtype=float) + design_clusters = pd.MultiIndex.from_frame( + positive[["stratum", "cluster"]] + ).to_numpy() + return { + "persons": int(frame["person_id"].nunique()), + "households": int(frame["interview"].nunique()), + "design_strata": int(frame["stratum"].nunique()), + "design_clusters": int( + len(frame[["stratum", "cluster"]].drop_duplicates()) + ), + "kish_n_eff": round(kish_n_eff(weights), 3), + "household_cluster_worst_n_eff": round( + cluster_worst_n_eff(weights, positive["interview"].to_numpy()), + 3, + ), + "design_cluster_worst_n_eff": round( + cluster_worst_n_eff(weights, design_clusters), 3 + ), + } + + +def _roster_summary(frame: pd.DataFrame) -> dict[str, Any]: + return { + "overall": _summary(frame), + "by_age_band": { + name: _summary(frame[frame["age"].between(low, high)]) + for name, low, high in ROSTER_BANDS + }, + } + + +def _b2_roster( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], +) -> pd.DataFrame: + full = _full(person_wave, ANCHOR_WAVE) + rows = valid_rows(person_wave, validity_by_wave) + ids = rows.loc[rows["period"] == ANCHOR_WAVE, "person_id"] + return full[full["person_id"].isin(ids)].copy() + + +def roster_counts( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + anchor: pd.DataFrame, + design: pd.DataFrame, +) -> dict[str, Any]: + """H's three candidate rosters, with each roster's own 2011 weight.""" + proxy = _present( + anchor[[column for column in PERSON_COLUMNS if column != "period"]] + ) + proxy = proxy[proxy["age"] >= 30] + frames = { + "R_FULL": _full(person_wave, ANCHOR_WAVE), + "R_B2": _b2_roster(person_wave, validity_by_wave), + "R_TA_PROXY": proxy, + } + return { + name: _roster_summary(_with_design(frame, design)) + for name, frame in frames.items() + } + + +def b3_counts( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + design: pd.DataFrame, +) -> dict[str, Any]: + """B3's valid-row support and same-person lag pairs at later-wave age.""" + rows = _with_design(valid_rows(person_wave, validity_by_wave), design) + by_wave = {} + for wave in WAVES: + by_band = {} + for name, low, high in OLDER_BANDS: + cell = rows[ + (rows["period"] == wave) & rows["age"].between(low, high) + ] + by_band[name] = { + **_summary(cell), + "valid_rows": int(len(cell)), + "accuracy_positive_rows": int(cell["acc_flag"].sum()), + } + by_wave[str(wave)] = by_band + pairs = {} + for lag in LAGS: + by_pair = {} + for wave in WAVES: + later = wave + lag + if later not in WAVES: + continue + earlier_ids = rows.loc[rows["period"] == wave, "person_id"] + both = rows[ + (rows["period"] == later) & rows["person_id"].isin(earlier_ids) + ] + by_pair[f"{wave}_{later}"] = { + name: int( + both.loc[ + both["age"].between(low, high), "person_id" + ].nunique() + ) + for name, low, high in OLDER_BANDS + } + pairs[str(lag)] = by_pair + return { + "by_wave": by_wave, + "distinct_persons_by_band": { + name: int( + rows.loc[rows["age"].between(low, high), "person_id"].nunique() + ) + for name, low, high in OLDER_BANDS + }, + "lag_pairs": pairs, + } + + +def b3l_counts( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + design: pd.DataFrame, +) -> dict[str, Any]: + """B3L's 2006-reference roster and later support with fixed weights.""" + full = _full(person_wave, B3L_ORIGIN_WAVE) + rows = valid_rows(person_wave, validity_by_wave) + origin_ids = rows.loc[rows["period"] == B3L_ORIGIN_WAVE, "person_id"] + domain = full[full["person_id"].isin(origin_ids)] + later_rows = rows[rows["person_id"].isin(domain["person_id"])].copy() + later_rows["weight"] = later_rows["person_id"].map( + domain.set_index("person_id")["weight"] + ) + support = {} + for wave in WAVES: + if wave < 2009: + continue + support[str(wave)] = {} + for name, low, high in (*CONTEXT_BANDS, *OLDER_BANDS): + cell = later_rows[ + (later_rows["period"] == wave) + & later_rows["age"].between(low, high) + ] + support[str(wave)][name] = { + "persons": int(cell["person_id"].nunique()), + "kish_n_eff": round( + kish_n_eff(cell["weight"].to_numpy(dtype=float)), 3 + ), + } + return { + "origin_wave": B3L_ORIGIN_WAVE, + "rosters": { + "R2006_FULL": _roster_summary(_with_design(full, design)), + "D2006": _roster_summary(_with_design(domain, design)), + }, + "later_support": support, + } + + +def outside_b2_counts( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + design: pd.DataFrame, +) -> dict[str, Any]: + """Exclude every 2011 household containing an R_B2 person.""" + full = _full(person_wave, ANCHOR_WAVE) + domain = _b2_roster(person_wave, validity_by_wave) + outside = full[~full["interview"].isin(domain["interview"])] + later_households = {} + for wave in (2013, 2015): + containing = person_wave[ + (person_wave["period"] == wave) + & person_wave["sequence"].between(1, 20) + & person_wave["person_id"].isin(domain["person_id"]) + ] + later_households[str(wave)] = int(containing["interview"].nunique()) + return { + **_roster_summary(_with_design(outside, design)), + "b2_households_by_wave": later_households, + } + + +def structural_counts( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + anchor: pd.DataFrame, + design: pd.DataFrame, +) -> dict[str, Any]: + """Return only protocol-v1 counts; never return person identifiers.""" + return { + "rosters": roster_counts( + person_wave, validity_by_wave, anchor, design + ), + "b3": b3_counts(person_wave, validity_by_wave, design), + "b3l": b3l_counts(person_wave, validity_by_wave, design), + "outside_b2": outside_b2_counts(person_wave, validity_by_wave, design), + } + + +# -------------------------------------------------------------------------- +# Disclosure, applied only after all raw counts have been computed +# -------------------------------------------------------------------------- +def _cell(record: dict, path: CellPath) -> Any: + for key in path: + record = record[key] + return record + + +def _put(record: dict, path: CellPath, value: Any) -> None: + for key in path[:-1]: + record = record[key] + record[path[-1]] = value + + +def _leaves(record: dict, prefix: CellPath = ()): + for key, value in record.items(): + path = (*prefix, key) + if isinstance(value, dict): + yield from _leaves(value, path) + elif key != "origin_wave": # geometry, not a count + yield path, value + + +def _subtract(row: dict, other: dict, scale: Fraction) -> None: + for key, value in other.items(): + result = row.get(key, 0) - scale * value + if result: + row[key] = result + else: + row.pop(key, None) + + +def _equation(relation: Relation) -> dict[CellPath, Fraction]: + if isinstance(relation, Mapping): + return {path: Fraction(c) for path, c in relation.items() if c} + total, parts = relation + result = {total: Fraction(1)} + for part in parts: + result[part] = result.get(part, 0) - 1 + return {path: c for path, c in result.items() if c} + + +def _recoverable(equations: list[dict], hidden: set) -> dict[CellPath, dict]: + """Find hidden cells determined even by combinations of equations. + + Exact rational elimination avoids both floating-point rank decisions + and the assumption that two hidden cells in every equation suffice. + """ + basis = {} + for equation in equations: + row = {path: c for path, c in equation.items() if path in hidden} + witness = equation.copy() + while row: + pivot = min(row) + if pivot not in basis: + scale = row[pivot] + basis[pivot] = ( + {key: c / scale for key, c in row.items()}, + {key: c / scale for key, c in witness.items()}, + ) + break + other_row, other_witness = basis[pivot] + scale = row[pivot] + _subtract(row, other_row, scale) + _subtract(witness, other_witness, scale) + for pivot in sorted(basis, reverse=True): + row, witness = basis[pivot] + for key in sorted(set(row) - {pivot}): + if key in basis and key in row: + other_row, other_witness = basis[key] + scale = row[key] + _subtract(row, other_row, scale) + _subtract(witness, other_witness, scale) + return { + pivot: witness + for pivot, (row, witness) in basis.items() + if len(row) == 1 + } + + +def suppress_counts( + counts: dict[str, Any], relations: tuple[Relation, ...] = () +) -> tuple[dict[str, Any], dict[str, Any]]: + """Return publishable counts and an audit of each suppression pass. + + Relations are exact total/parts identities or signed count equations. + They stay private: neither unit sets nor residual counts are emitted. + Effective sizes below ten are conservatively suppressed even when + their supporting set is larger. Empty-set zeros remain zero. + """ + published = deepcopy(counts) + values = dict(_leaves(counts)) + hidden = set() + + def suppress(path): + hidden.add(path) + _put(published, path, SUPPRESSED) + + for path, value in values.items(): + parent = _cell(counts, path[:-1]) + support = parent.get("valid_rows", parent.get("persons", 0)) + small_support = ( + path[-1].endswith("_n_eff") and 0 < support < DISCLOSURE_THRESHOLD + ) + if 0 < value < DISCLOSURE_THRESHOLD or small_support: + suppress(path) + primary = len(hidden) + equations = [_equation(relation) for relation in relations] + + def smallest(equation): + available = [ + path + for path in equation + if path not in hidden and values[path] > 0 + ] + parts = [path for path in available if equation[path] < 0] + candidates = parts or available + return min( + candidates, key=lambda path: (values[path], path), default=None + ) + + passes = [] + while True: + extra = set() + for equation in equations: + if len(hidden.intersection(equation)) == 1: + candidate = smallest(equation) + if candidate is not None: + extra.add(candidate) + # Coupled tables can disclose a cell without any single equation + # having just one hidden term. Close those paths as well. + if not extra: + proposals = [] + for witness in _recoverable(equations, hidden).values(): + candidate = smallest(witness) + if candidate is not None: + proposals.append(candidate) + if proposals: + # One complement can close several recovery equations; + # recompute before choosing another cell. + extra.add( + min(proposals, key=lambda path: (values[path], path)) + ) + # A complementary person-count suppression must not be undone by + # a published Kish size (equal to n for constant weights). + for path in values: + if path not in hidden and path[-1].endswith("_n_eff"): + parent = _cell(published, path[:-1]) + if parent.get("persons") == SUPPRESSED: + extra.add(path) + if not extra: + break + for path in extra: + suppress(path) + passes.append(len(extra)) + return published, { + "threshold": DISCLOSURE_THRESHOLD, + "rule": ( + "1-9 and statistics on nonempty sets below 10: <10; empty " + "zeros retained; positive effective sizes below 10 also hidden. " + "Complementary: hide the smallest remaining positive part, " + "or its total if no positive part remains; repeat to a fixed " + "point, including aliases and coupled subtraction identities." + ), + "primary_suppressed": primary, + "complementary_suppressed": sum(passes), + "complementary_passes": passes, + } + + +def _incidence_relations(cells: dict[CellPath, frozenset]) -> list[dict]: + """All exact additive identities among these private unit sets. + + Units with identical cell membership share one incidence column. + Reducing its rows yields a basis of every cardinality identity, + including disjoint partitions, aliases and inclusion/exclusion when + intersections are present. Overlapping unions are never just summed. + Only coefficients on existing output cells leave this function. + """ + memberships = {} + paths = list(cells) + for index, units in enumerate(cells.values()): + for unit in units: + memberships.setdefault(unit, []).append(index) + columns = sorted({tuple(indices) for indices in memberships.values()}) + vectors = {path: {} for path in paths} + for column in columns: + for index in column: + vectors[paths[index]][column] = Fraction(1) + basis = {} + relations = [] + for path, row in vectors.items(): + if not row: # zero counts are public; no useful suppression alias + continue + terms = {path: Fraction(1)} + while row: + pivot = min(row) + if pivot not in basis: + scale = row[pivot] + basis[pivot] = ( + {key: c / scale for key, c in row.items()}, + {key: c / scale for key, c in terms.items()}, + ) + break + other_row, other_terms = basis[pivot] + scale = row[pivot] + _subtract(row, other_row, scale) + _subtract(terms, other_terms, scale) + if not row and terms: + relations.append(terms) + return relations + + +def disclosure_relations( + person_wave: pd.DataFrame, + validity_by_wave: Mapping[int, pd.DataFrame], + anchor: pd.DataFrame, + design: pd.DataFrame, +) -> tuple[Relation, ...]: + """Explicitly register every output table and its unit-set relations. + + Person IDs link waves, rosters, lag intersections and pooled unions. + Household IDs are wave-specific; design clusters include stratum. + No earnings value, new statistic or unreported remainder is computed. + """ + cells = {} + partitions = [] + + def register(prefix, frame, wave): + def clean(value): + return None if pd.isna(value) else value + + units = { + "persons": frozenset( + ("person", value) for value in frame["person_id"].dropna() + ), + "households": frozenset( + ("household", wave, value) + for value in frame["interview"].dropna() + ), + "design_strata": frozenset( + ("stratum", value) for value in frame["stratum"].dropna() + ), + "design_clusters": frozenset( + ("cluster", clean(stratum), clean(cluster)) + for stratum, cluster in frame[["stratum", "cluster"]] + .drop_duplicates() + .itertuples(index=False, name=None) + ), + } + for metric, members in units.items(): + cells[(*prefix, metric)] = members + + def partition(total, parts): + # A union is additive only with exhaustive, disjoint parts. + members = set() + for part in parts: + if members.intersection(cells[part]): + return + members.update(cells[part]) + if members == cells[total]: + partitions.append((total, tuple(parts))) + + def roster(prefix, frame, wave): + register((*prefix, "overall"), frame, wave) + for name, low, high in ROSTER_BANDS: + register( + (*prefix, "by_age_band", name), + frame[frame["age"].between(low, high)], + wave, + ) + for metric in ( + "persons", + "households", + "design_strata", + "design_clusters", + ): + partition( + (*prefix, "overall", metric), + [ + (*prefix, "by_age_band", name, metric) + for name, _, _ in ROSTER_BANDS + ], + ) + + full = _with_design(_full(person_wave, ANCHOR_WAVE), design) + domain = _with_design(_b2_roster(person_wave, validity_by_wave), design) + proxy = _present( + anchor[[column for column in PERSON_COLUMNS if column != "period"]] + ) + proxy = _with_design(proxy[proxy["age"] >= 30], design) + for name, frame in ( + ("R_FULL", full), + ("R_B2", domain), + ("R_TA_PROXY", proxy), + ): + roster(("rosters", name), frame, ANCHOR_WAVE) + rows = _with_design(valid_rows(person_wave, validity_by_wave), design) + for wave in WAVES: + for name, low, high in OLDER_BANDS: + frame = rows[ + (rows["period"] == wave) & rows["age"].between(low, high) + ] + prefix = ("b3", "by_wave", str(wave), name) + register(prefix, frame, wave) + cells[(*prefix, "valid_rows")] = cells[(*prefix, "persons")] + cells[(*prefix, "accuracy_positive_rows")] = frozenset( + ("person", value) + for value in frame.loc[frame["acc_flag"], "person_id"] + ) + for name, low, high in OLDER_BANDS: + total = ("b3", "distinct_persons_by_band", name) + cells[total] = frozenset( + ("person", value) + for value in rows.loc[rows["age"].between(low, high), "person_id"] + ) + partition( + total, + [("b3", "by_wave", str(wave), name, "persons") for wave in WAVES], + ) + for lag in LAGS: + for wave in WAVES: + later = wave + lag + if later not in WAVES: + continue + earlier = rows.loc[rows["period"] == wave, "person_id"] + both = rows[ + (rows["period"] == later) & rows["person_id"].isin(earlier) + ] + for name, low, high in OLDER_BANDS: + cells[ + ("b3", "lag_pairs", str(lag), f"{wave}_{later}", name) + ] = frozenset( + ("person", value) + for value in both.loc[ + both["age"].between(low, high), "person_id" + ] + ) + origin = _with_design(_full(person_wave, B3L_ORIGIN_WAVE), design) + origin_ids = rows.loc[rows["period"] == B3L_ORIGIN_WAVE, "person_id"] + origin_domain = origin[origin["person_id"].isin(origin_ids)] + for name, frame in (("R2006_FULL", origin), ("D2006", origin_domain)): + roster(("b3l", "rosters", name), frame, B3L_ORIGIN_WAVE) + for wave in WAVES: + if wave < 2009: + continue + for name, low, high in (*CONTEXT_BANDS, *OLDER_BANDS): + frame = rows[ + (rows["period"] == wave) + & rows["person_id"].isin(origin_domain["person_id"]) + & rows["age"].between(low, high) + ] + cells[("b3l", "later_support", str(wave), name, "persons")] = ( + frozenset(("person", value) for value in frame["person_id"]) + ) + outside = full[~full["interview"].isin(domain["interview"])] + roster(("outside_b2",), outside, ANCHOR_WAVE) + for wave in (2013, 2015): + frame = person_wave[ + (person_wave["period"] == wave) + & person_wave["sequence"].between(1, 20) + & person_wave["person_id"].isin(domain["person_id"]) + ] + cells[("outside_b2", "b2_households_by_wave", str(wave))] = frozenset( + ("household", wave, value) for value in frame["interview"].dropna() + ) + partition( + ("rosters", "R_FULL", "overall", "households"), + [ + ("rosters", "R_B2", "overall", "households"), + ("outside_b2", "overall", "households"), + ], + ) + return tuple([*partitions, *_incidence_relations(cells)]) + + +# -------------------------------------------------------------------------- +# Thin readers and requested-label provenance +# -------------------------------------------------------------------------- +def read_inputs( + data_dir: Path, +) -> tuple[ + pd.DataFrame, dict[int, pd.DataFrame], pd.DataFrame, pd.DataFrame, dict +]: + """Read only the named protocol inputs; immediately reduce raw fields.""" + concepts = { + name: panels.DEMOGRAPHIC_CONCEPTS[name] for name in INDIVIDUAL_CONCEPTS + } + person_wave = panels.ind_person_period( + concepts, data_dir=data_dir, waves=list(WAVES) + )[list(PERSON_COLUMNS)] + anchor = psid2010.read_anchor_wave( + data_dir=data_dir, anchor_wave=ANCHOR_WAVE + ).drop(columns="reported_birth_year") + design = age67.read_design_variables(data_dir=data_dir) + validity = { + wave: validity_frame(family.read_family_labor(wave, data_dir=data_dir)) + for wave in WAVES + } + identifiers = { + "interview_1968": { + "variable": "ER30001", + "label": "1968 INTERVIEW NUMBER", + }, + "person_number": {"variable": "ER30002", "label": "PERSON NUMBER 68"}, + } + layout = psid2010.ANCHOR_LAYOUTS[ANCHOR_WAVE] + allowlist = { + "individual": { + "identifiers": identifiers, + "concepts": { + name: list(patterns) for name, patterns in concepts.items() + }, + "by_wave": _individual_variables(data_dir, WAVES), + }, + "anchor_2011": { + **identifiers, + **{ + role: {"variable": var, "label": label} + for role, (var, label) in layout.variables.items() + }, + }, + "design": { + **identifiers, + "stratum": { + "variable": "ER31996", + "label": "SAMPLING ERROR STRATUM", + }, + "cluster": { + "variable": "ER31997", + "label": "SAMPLING ERROR CLUSTER", + }, + }, + "family": { + str(wave): _family_variables(data_dir, wave) for wave in WAVES + }, + } + return person_wave, validity, anchor, design, allowlist + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +def _git(*arguments: str) -> str: + return subprocess.run( + ["git", *arguments], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def _check_runtime() -> None: + if _git("status", "--porcelain"): + raise SystemExit("B0.2 requires a clean protocol-freezing worktree") + for name in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS"): + try: + threads = int(os.environ.get(name, "")) + except ValueError: + threads = 0 + if not 1 <= threads <= 4: + raise SystemExit(f"set {name} to an integer from 1 to 4") + if os.environ.get("OMP_WAIT_POLICY") != "PASSIVE": + raise SystemExit("set OMP_WAIT_POLICY=PASSIVE") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--out", type=Path, default=DEFAULT_OUT) + parser.add_argument( + "--local-copy", + type=Path, + required=True, + help="exclusive unsuppressed JSON in B0.2's evidence folder, outside the repository", + ) + args = parser.parse_args(argv) + out = args.out if args.out.is_absolute() else ROOT / args.out + local = args.local_copy + local = local if local.is_absolute() else ROOT / local + repository = ROOT.resolve() + if Path(os.path.abspath(local)).is_relative_to( + repository + ) or local.resolve().is_relative_to(repository): + raise SystemExit("local copy must be outside the repository") + if out.resolve() == local.resolve(): + raise SystemExit( + "published output and local copy must be separate files" + ) + for destination in (out, local): + if destination.exists() or destination.is_symlink(): + raise SystemExit(f"refusing to overwrite {destination}") + if ( + RUNS_DIR.resolve() in destination.resolve().parents + or "runs" in destination.resolve().parts + ): + raise SystemExit("B0.2 writes no file under runs/") + _check_runtime() + repository_head = _git("rev-parse", "HEAD") + data_dir = psid._resolve_data_dir(None).resolve() + with psid2010.record_files_read(data_dir) as opened: + person_wave, validity, anchor, design, allowlist = read_inputs( + data_dir + ) + counts = structural_counts(person_wave, validity, anchor, design) + committed = json.loads(B0_1_COUNTS.read_text(encoding="utf-8")) + domain_size = counts["rosters"]["R_B2"]["overall"]["persons"] + if domain_size != committed["counts"]["domain_persons"]: + raise AssertionError( + "R_B2 size differs from B0.1 counts.domain_persons" + ) + payload = { + "schema": SCHEMA, + "repository_head": repository_head, + "worktree_clean": True, + "script_sha256": _sha256_file(Path(__file__)), + "psid_files_opened_sha256": dict(sorted(opened.items())), + "b0_1_counts_sha256": _sha256_file(B0_1_COUNTS), + "requested_label_allowlist": allowlist, + "geometry": { + "collection_waves": list(WAVES), + "reference_years": [wave - 1 for wave in WAVES], + "age_convention": "collection_wave", + "lag_years": list(LAGS), + }, + "counts": counts, + } + relations = disclosure_relations(person_wave, validity, anchor, design) + suppressed, disclosure = suppress_counts(counts, relations) + public_payload = { + **payload, + "counts": suppressed, + "disclosure": disclosure, + } + # Reserve both paths exclusively before writing either payload. If the + # second reservation fails, remove only the first file we just created. + with out.open("x", encoding="utf-8") as stream: + try: + local_stream = local.open("x", encoding="utf-8") + except OSError: + out.unlink() + raise + with local_stream: + local_stream.write( + json.dumps(payload, indent=2, sort_keys=True) + "\n" + ) + stream.write( + json.dumps(public_payload, indent=2, sort_keys=True) + "\n" + ) + print(out) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_track_b_b0_2_counts.py b/tests/test_track_b_b0_2_counts.py new file mode 100644 index 000000000..003edac74 --- /dev/null +++ b/tests/test_track_b_b0_2_counts.py @@ -0,0 +1,1565 @@ +"""B0.2's version-1 protocol, tested solely on invented frames.""" + +from __future__ import annotations + +import hashlib +import json +import os +import sys +from contextlib import contextmanager +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) + +import track_b_b0_2_counts as script # noqa: E402 + +from scripts import track_b_b0_1_counts as b01 # noqa: E402 + +PROTOCOL_SHA256 = ( + "94e22e3fff7b8ec541f87abe9890270a9893e0ec1653ce7b91fb0170b2732cb9" +) +WAVES = tuple(range(1999, 2024, 2)) +SENTINEL = 9_999_998 +PERSON_COLUMNS = [ + "person_id", + "period", + "age", + "sequence", + "relationship", + "weight", + "interview", +] +FORBIDDEN_COLUMNS = ( + "marital_status", + "death_year", + "social_security_amount", + "disability_status", +) + + +def _row( + person, + wave=2011, + age=62, + *, + sequence=1, + relationship=10, + weight=1.0, + interview=None, +): + return dict( + zip( + PERSON_COLUMNS, + ( + person, + wave, + age, + sequence, + relationship, + weight, + person if interview is None else interview, + ), + strict=True, + ) + ) + + +def _inputs(rows, *, invalid=(), missing=(), accuracy=()): + """Make complete wave mappings; no person or income is observed.""" + people = pd.DataFrame(rows, columns=PERSON_COLUMNS) + labor = {} + invalid = set(invalid) + missing = set(missing) + accuracy = set(accuracy) + for wave in WAVES: + interviews = set(people.loc[people["period"] == wave, "interview"]) + records = [] + for interview in sorted(interviews | {999_999}): + if (wave, interview) in missing: + continue + records.append( + { + "interview": interview, + "head_labor": ( + float(SENTINEL) + if (wave, interview, "head") in invalid + else 123.0 + ), + "spouse_labor": ( + float(SENTINEL) + if (wave, interview, "spouse") in invalid + else 0.0 + ), + "head_acc": int((wave, interview, "head") in accuracy), + "spouse_acc": int((wave, interview, "spouse") in accuracy), + } + ) + labor[wave] = pd.DataFrame(records) + anchor = people.loc[people["period"] == 2011].drop(columns="period") + ids = sorted(people["person_id"].unique()) + design = pd.DataFrame({"person_id": ids, "stratum": 1, "cluster": ids}) + return people, labor, anchor, design + + +def _validity(labor): + return { + wave: script.validity_frame(frame) for wave, frame in labor.items() + } + + +def _counts(inputs): + people, labor, anchor, design = inputs + return script.structural_counts(people, _validity(labor), anchor, design) + + +def _bytes(record): + return json.dumps(record, sort_keys=True, allow_nan=False).encode() + + +def _worked_rosters(): + inputs = _inputs( + [ + _row(1, age=29, weight=1.0, interview=10), + _row(2, age=30, relationship=20, weight=2.0, interview=10), + _row(3, age=64, relationship=30, weight=3.0, interview=10), + _row(4, age=65, relationship=30, weight=4.0, interview=20), + _row(5, age=80, weight=5.0, interview=30), + _row(6, age=62, sequence=21, weight=6.0, interview=40), + _row(7, age=62, weight=0.0, interview=50), + _row(1, 2013, age=31, interview=101), + _row(2, 2013, age=32, interview=102), + _row(3, 2013, age=66, interview=103), + _row(1, 2015, age=33, sequence=21, interview=201), + _row(2, 2015, age=34, relationship=30, weight=0, interview=202), + ], + invalid={(2011, 30, "head")}, + ) + people, labor, anchor, _ = inputs + anchor = anchor.copy() + anchor.loc[anchor["person_id"] == 1, "age"] = 30 + anchor.loc[anchor["person_id"] == 2, "age"] = 29 + anchor.loc[anchor["person_id"] == 3, "age"] = 65 + anchor["weight"] = [7.0, 1.0, 3.0, 0.0, 5.0, 6.0, 0.0] + design = pd.DataFrame( + { + "person_id": list(range(1, 8)), + "stratum": [1, 1, 2, 2, 2, 2, 2], + "cluster": [10, 10, 20, 20, 10, 30, 40], + } + ) + return people, labor, anchor, design + + +def _invariance_inputs(): + """Exercise nonempty rosters, older support, both lags and B3L.""" + return _inputs( + [ + _row( + person, + wave, + base_age + wave - 2007, + relationship={103: 20, 104: 30}.get(person, 10), + weight=float(person - 99 + (wave % 3)), + interview=101 if person == 103 else person, + ) + for wave in WAVES + for person, base_age in ( + (101, 60), + (102, 43), + (103, 62), + (104, 35), + ) + ], + invalid={(2013, 101, "spouse")}, + accuracy={(2009, 101, "head"), (2009, 101, "spouse")}, + ) + + +# -------------------------------------------------------------------------- +# Protocol and outcome blindness +# -------------------------------------------------------------------------- +def test_protocol_block_is_pinned(): + text = (ROOT / "docs/design/track_b_b0_2_protocol.md").read_bytes() + begin = b"" + end = b"" + block = text[text.index(begin) : text.index(end) + len(end)] + assert hashlib.sha256(block).hexdigest() == PROTOCOL_SHA256 + + +def test_b01_helpers_are_imported_unchanged(): + for name in ( + "validity_frame", + "earnings_rows", + "kish_n_eff", + "cluster_worst_n_eff", + "_individual_variables", + ): + assert getattr(script, name) is getattr(b01, name) + + +@settings(max_examples=25, deadline=None) +@given( + st.lists( + st.floats( + min_value=-1e9, + max_value=SENTINEL - 1, + allow_nan=False, + allow_infinity=False, + ), + min_size=8 * len(WAVES), + max_size=8 * len(WAVES), + ) +) +def test_counts_are_byte_invariant_to_valid_labor_values(values): + inputs = _invariance_inputs() + people, labor, anchor, design = inputs + changed = {} + replacements = iter(values) + for wave, frame in labor.items(): + frame = frame.copy() + for column in ("head_labor", "spouse_labor"): + frame[column] = [ + replacement if value < SENTINEL else value + for value, replacement in zip( + frame[column], replacements, strict=False + ) + ] + changed[wave] = frame + original = _counts(inputs) + assert original["b3"]["by_wave"]["2011"]["62_64"]["persons"] > 0 + assert original["b3"]["lag_pairs"]["4"]["2007_2011"]["62_64"] > 0 + assert original["b3l"]["rosters"]["D2006"]["overall"]["persons"] > 0 + assert original["b3l"]["later_support"]["2009"]["62_64"]["persons"] > 0 + assert original["outside_b2"]["overall"]["persons"] > 0 + assert _bytes(original) == _bytes( + _counts((people, changed, anchor, design)) + ) + + +@settings(max_examples=25, deadline=None) +@given( + st.lists( + st.one_of( + st.integers(), + st.text(max_size=20), + st.booleans(), + st.none(), + ), + min_size=len(FORBIDDEN_COLUMNS), + max_size=len(FORBIDDEN_COLUMNS), + ) +) +def test_counts_are_byte_invariant_to_forbidden_columns(values): + people, labor, anchor, design = _invariance_inputs() + additions = dict(zip(FORBIDDEN_COLUMNS, values, strict=True)) + baseline = script.structural_counts( + people, _validity(labor), anchor, design + ) + changed_labor = { + wave: frame.assign(**additions) for wave, frame in labor.items() + } + # Test both read-time reduction and extra columns on already-read frames. + changed_validity = { + wave: frame.assign(**additions) + for wave, frame in _validity(changed_labor).items() + } + changed = script.structural_counts( + people.assign(**additions), + changed_validity, + anchor.assign(**additions), + design.assign(**additions), + ) + assert _bytes(baseline) == _bytes(changed) + + +def _assert_no_forbidden_output(value): + forbidden = ( + "disposition", + "death", + "died", + "transition", + "earnings_level", + "earnings level", + "labor_level", + "labor level", + "income_level", + "income level", + "quantile", + "positive_pair", + "participation", + "marital", + "disability", + "payment", + "recipient", + "power", + ) + + def check(value): + if isinstance(value, dict): + for key, item in value.items(): + check(key) + check(item) + elif isinstance(value, (list, tuple)): + for item in value: + check(item) + elif isinstance(value, str): + assert not any(word in value.lower() for word in forbidden) + + check(value) + + +def test_no_forbidden_output_keys_or_values(): + _assert_no_forbidden_output(_counts(_worked_rosters())) + + +# -------------------------------------------------------------------------- +# Hand-derived rows, rosters and bands +# -------------------------------------------------------------------------- +def test_valid_rows_require_present_role_positive_weight_and_role_validity(): + people, labor, _, _ = _inputs( + [ + _row(1), + _row(2, sequence=20, relationship=20), + _row(3, relationship=22), + _row(4, relationship=30), + _row(5, sequence=0), + _row(6, sequence=21), + _row(7, weight=0), + _row(8, relationship=20), + _row(9), + _row(10, weight=-1), + ], + invalid={(2011, 8, "spouse")}, + missing={(2011, 9)}, + accuracy={(2011, 2, "spouse"), (2011, 3, "head")}, + ) + rows = script.valid_rows(people, _validity(labor)) + assert set(rows["person_id"]) == {1, 2, 3} + assert set(rows["period"]) == {2011} + assert rows.set_index("person_id")["acc_flag"].to_dict() == { + 1: False, + 2: True, + 3: False, + } + + +def test_rosters_and_age_30_proxy_boundary_use_their_own_weights_and_ages(): + people, labor, anchor, design = _worked_rosters() + counts = script.roster_counts(people, _validity(labor), anchor, design) + full = counts["R_FULL"] + assert full["overall"] == { + "persons": 5, + "households": 3, + "design_strata": 2, + "design_clusters": 3, + "kish_n_eff": 4.091, + "household_cluster_worst_n_eff": 2.922, + "design_cluster_worst_n_eff": 2.711, + } + assert { + band: count["persons"] for band, count in full["by_age_band"].items() + } == { + "0_24": 0, + "25_44": 2, + "45_61": 0, + "62_64": 1, + "65_66": 1, + "67_69": 0, + "70_79": 0, + "80_plus": 1, + } + b2 = counts["R_B2"]["overall"] + assert b2["persons"] == 2 + assert b2["households"] == 1 + assert b2["kish_n_eff"] == 1.8 + assert b2["household_cluster_worst_n_eff"] == 1.0 + proxy = counts["R_TA_PROXY"] + assert proxy["overall"]["persons"] == 3 + assert proxy["overall"]["kish_n_eff"] == 2.711 + assert proxy["overall"]["household_cluster_worst_n_eff"] == 1.8 + assert proxy["by_age_band"]["25_44"]["persons"] == 1 + assert proxy["by_age_band"]["62_64"]["persons"] == 0 + assert proxy["by_age_band"]["65_66"]["persons"] == 1 + + +def test_band_edges_and_b3_support_are_hand_counted(): + ages = [0, 24, 25, 44, 45, 61, 62, 64, 65, 66, 67, 69, 70, 79, 80] + people, labor, anchor, design = _inputs( + [_row(i, age=age) for i, age in enumerate(ages, 1)], + accuracy={(2011, 7, "head"), (2011, 9, "head")}, + ) + counts = script.structural_counts(people, _validity(labor), anchor, design) + roster_bands = counts["rosters"]["R_FULL"]["by_age_band"] + assert {key: value["persons"] for key, value in roster_bands.items()} == { + "0_24": 2, + "25_44": 2, + "45_61": 2, + "62_64": 2, + "65_66": 2, + "67_69": 2, + "70_79": 2, + "80_plus": 1, + } + support = counts["b3"]["by_wave"]["2011"] + for band in ("62_64", "65_66", "67_69"): + assert support[band]["valid_rows"] == 2 + assert support[band]["persons"] == 2 + assert support[band]["households"] == 2 + assert support[band]["design_strata"] == 1 + assert support[band]["design_clusters"] == 2 + assert support[band]["kish_n_eff"] == 2.0 + assert support[band]["household_cluster_worst_n_eff"] == 2.0 + assert support[band]["design_cluster_worst_n_eff"] == 2.0 + assert [support[band]["accuracy_positive_rows"] for band in support] == [ + 1, + 1, + 0, + ] + assert counts["b3"]["distinct_persons_by_band"] == { + "62_64": 2, + "65_66": 2, + "67_69": 2, + } + + +def test_b3_lag_pairs_use_exact_lags_and_later_wave_band(): + people, labor, _, design = _inputs( + [ + _row(1, 1999, 60), + _row(1, 2001, 62), + _row(1, 2003, 64), + _row(2, 1999, 61), + _row(2, 2001, 63), + _row(2, 2003, 65), + _row(3, 1999, 63), + _row(3, 2001, 65), + _row(3, 2003, 67), + _row(4, 2001, 67), + _row(4, 2003, 69), + _row(5, 1999, 69), + _row(5, 2001, 71), + ], + invalid={(2001, 3, "head")}, + ) + counts = script.b3_counts(people, _validity(labor), design) + pairs = counts["lag_pairs"] + assert set(pairs) == {"2", "4"} + assert set(pairs["2"]) == {f"{wave}_{wave + 2}" for wave in WAVES[:-1]} + assert set(pairs["4"]) == {f"{wave}_{wave + 4}" for wave in WAVES[:-2]} + assert pairs["2"]["1999_2001"] == { + "62_64": 2, + "65_66": 0, + "67_69": 0, + } + assert pairs["2"]["2001_2003"] == { + "62_64": 1, + "65_66": 1, + "67_69": 1, + } + assert pairs["4"]["1999_2003"] == { + "62_64": 1, + "65_66": 1, + "67_69": 1, + } + assert counts["distinct_persons_by_band"] == { + "62_64": 3, + "65_66": 1, + "67_69": 3, + } + + +def test_b3l_uses_2007_origin_and_fixed_origin_weights_for_later_support(): + people, labor, _, design = _inputs( + [ + _row(1, 2007, 60, weight=2), + _row(1, 2009, 62, weight=100), + _row(1, 2011, 64), + _row(1, 2023, 76), + _row(2, 2007, 25, relationship=30), + _row(2, 2009, 27), + _row(3, 2007, 43, weight=4), + _row(3, 2009, 45), + _row(3, 2011, 47, weight=8), + _row(4, 2007, 64), + _row(4, 2009, 66, weight=0), + _row(5, 2007, 60, sequence=21), + _row(5, 2011, 64), + _row(6, 2007, 59), + _row(6, 2009, 62, weight=999), + ] + ) + counts = script.b3l_counts(people, _validity(labor), design) + assert counts["origin_wave"] == 2007 + rosters = counts["rosters"] + assert rosters["R2006_FULL"]["overall"]["persons"] == 5 + assert rosters["D2006"]["overall"]["persons"] == 4 + assert rosters["D2006"]["by_age_band"]["25_44"]["persons"] == 1 + assert rosters["D2006"]["by_age_band"]["45_61"]["persons"] == 2 + assert rosters["D2006"]["by_age_band"]["62_64"]["persons"] == 1 + support = counts["later_support"] + assert set(support) == {str(wave) for wave in WAVES if wave >= 2009} + assert support["2009"] == { + "25_44": {"persons": 0, "kish_n_eff": 0.0}, + "45_61": {"persons": 1, "kish_n_eff": 1.0}, + "62_64": {"persons": 2, "kish_n_eff": 1.8}, + "65_66": {"persons": 0, "kish_n_eff": 0.0}, + "67_69": {"persons": 0, "kish_n_eff": 0.0}, + } + assert support["2011"]["62_64"] == { + "persons": 1, + "kish_n_eff": 1.0, + } + assert all(item["persons"] == 0 for item in support["2023"].values()) + + +def test_outside_b2_excludes_entire_anchor_households_and_counts_later_units(): + people, labor, _, design = _worked_rosters() + counts = script.outside_b2_counts(people, _validity(labor), design) + # Persons 1 and 2 form B2; their household also excludes person 3. + assert counts["overall"] == { + "persons": 2, + "households": 2, + "design_strata": 1, + "design_clusters": 2, + "kish_n_eff": 1.976, + "household_cluster_worst_n_eff": 1.976, + "design_cluster_worst_n_eff": 1.976, + } + assert counts["by_age_band"]["62_64"]["persons"] == 0 + assert counts["by_age_band"]["65_66"]["persons"] == 1 + assert counts["by_age_band"]["80_plus"]["persons"] == 1 + # The 2015 row for person 2 has zero weight and another relationship, + # but is present, and therefore still identifies a B2 household. + assert counts["b2_households_by_wave"] == {"2013": 2, "2015": 1} + + +def test_effective_sizes_match_hand_formulas_and_composite_design_clusters(): + weights = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + assert script.kish_n_eff(weights) == pytest.approx(225 / 55) + assert script.cluster_worst_n_eff( + weights, np.array([10, 10, 10, 20, 30]) + ) == pytest.approx(225 / 77) + counts = _counts(_worked_rosters()) + full = counts["rosters"]["R_FULL"]["overall"] + # Cluster label 10 occurs in two different strata: three design units. + assert full["design_clusters"] == 3 + assert full["design_cluster_worst_n_eff"] == round(225 / 83, 3) + + +def test_zero_weight_sets_have_zero_effective_size(): + inputs = _inputs([_row(1, 2007, weight=0), _row(1, weight=0)]) + counts = _counts(inputs) + for roster in counts["rosters"].values(): + assert roster["overall"]["persons"] == 0 + for key in ( + "kish_n_eff", + "household_cluster_worst_n_eff", + "design_cluster_worst_n_eff", + ): + assert roster["overall"][key] == 0.0 + assert counts["b3"]["by_wave"]["2011"]["62_64"]["kish_n_eff"] == 0 + assert counts["b3l"]["later_support"]["2011"]["62_64"] == { + "persons": 0, + "kish_n_eff": 0.0, + } + + +# -------------------------------------------------------------------------- +# Disclosure, using invented counts and structural frames only +# -------------------------------------------------------------------------- +@pytest.mark.parametrize("count", range(11)) +def test_disclosure_suppresses_counts_one_through_nine(count): + raw = {"table": {"persons": count}} + published, disclosure = script.suppress_counts(raw) + expected = "<10" if 1 <= count <= 9 else count + assert published == {"table": {"persons": expected}} + assert raw == {"table": {"persons": count}} + assert disclosure["threshold"] == 10 + assert disclosure["primary_suppressed"] == int(1 <= count <= 9) + assert disclosure["complementary_suppressed"] == 0 + assert not any(disclosure["complementary_passes"]) + + +@pytest.mark.parametrize("support", ["persons", "valid_rows"]) +@pytest.mark.parametrize("effective_size", [0.0, 8.125, 12.0]) +def test_disclosure_suppresses_effective_sizes_on_small_nonempty_sets( + support, effective_size +): + raw = { + "table": { + support: 9, + "kish_n_eff": effective_size, + "household_cluster_worst_n_eff": effective_size, + "design_cluster_worst_n_eff": effective_size, + } + } + published, disclosure = script.suppress_counts(raw) + assert all(value == "<10" for value in published["table"].values()) + assert disclosure["primary_suppressed"] == 4 + assert raw["table"][support] == 9 + assert raw["table"]["kish_n_eff"] == effective_size + + +def test_disclosure_keeps_empty_effective_sizes_and_suppresses_small_values(): + published, _ = script.suppress_counts( + { + "empty": {"persons": 0, "kish_n_eff": 0.0}, + "large": {"persons": 10, "kish_n_eff": 10.0}, + "unequal_weights": {"persons": 20, "kish_n_eff": 9.001}, + "origin_wave": 2007, + } + ) + assert published == { + "empty": {"persons": 0, "kish_n_eff": 0.0}, + "large": {"persons": 10, "kish_n_eff": 10.0}, + "unequal_weights": {"persons": 20, "kish_n_eff": "<10"}, + "origin_wave": 2007, + } + + +def test_disclosure_complements_the_smallest_remaining_published_part(): + raw = {"total": 40, "small": 5, "larger": 20, "smallest": 15} + relations = [(("total",), (("small",), ("larger",), ("smallest",)))] + published, disclosure = script.suppress_counts(raw, relations) + assert published == { + "total": 40, + "small": "<10", + "larger": 20, + "smallest": "<10", + } + assert disclosure["primary_suppressed"] == 1 + assert disclosure["complementary_suppressed"] == 1 + assert sum(disclosure["complementary_passes"]) == 1 + assert raw["smallest"] == 15 + + +def test_disclosure_repeats_complementary_passes_until_chain_is_safe(): + raw = { + "outer_total": 60, + "inner_total": 30, + "small": 5, + "shared": 25, + "other": 35, + } + # Outer is visited before the inner relation hides its shared part. + relations = [ + (("outer_total",), (("shared",), ("other",))), + (("inner_total",), (("small",), ("shared",))), + ] + published, disclosure = script.suppress_counts(raw, relations) + assert published == { + "outer_total": 60, + "inner_total": 30, + "small": "<10", + "shared": "<10", + "other": "<10", + } + assert disclosure["complementary_suppressed"] == 2 + assert len([n for n in disclosure["complementary_passes"] if n]) >= 2 + assert sum(disclosure["complementary_passes"]) == 2 + + +def test_disclosure_hides_effective_sizes_when_support_is_complemented(): + raw = { + "total": 25, + "small": {"persons": 5, "kish_n_eff": 5.0}, + "large": {"persons": 20, "kish_n_eff": 20.0}, + } + relations = [(("total",), (("small", "persons"), ("large", "persons")))] + published, _ = script.suppress_counts(raw, relations) + assert published["small"] == {"persons": "<10", "kish_n_eff": "<10"} + assert published["large"] == {"persons": "<10", "kish_n_eff": "<10"} + + +def _path_value(value, path): + for key in path: + value = value[key] + return value + + +def test_disclosure_relations_are_exact_structural_identities(): + from collections.abc import Mapping + + inputs = _worked_rosters() + people, labor, anchor, design = inputs + validity = _validity(labor) + counts = _counts(inputs) + relations = script.disclosure_relations(people, validity, anchor, design) + assert relations + for relation in relations: + if isinstance(relation, Mapping): + assert ( + sum( + coefficient * _path_value(counts, path) + for path, coefficient in relation.items() + ) + == 0 + ) + else: + total, parts = relation + assert _path_value(counts, total) == sum( + _path_value(counts, path) for path in parts + ) + expected = ( + ("rosters", "R_FULL", "overall", "persons"), + tuple( + ("rosters", "R_FULL", "by_age_band", band, "persons") + for band, _, _ in script.ROSTER_BANDS + ), + ) + assert expected in relations + + +def test_disclosure_published_counts_have_no_positive_value_below_ten(): + people, labor, anchor, design = _invariance_inputs() + validity = _validity(labor) + raw = script.structural_counts(people, validity, anchor, design) + relations = script.disclosure_relations(people, validity, anchor, design) + published, _ = script.suppress_counts(raw, relations) + + def check(value, path=()): + if isinstance(value, dict): + for key, item in value.items(): + check(item, (*path, key)) + elif isinstance(value, (int, float)) and path[-1] != "origin_wave": + assert value == 0 or value >= 10, (path, value) + + check(published) + + +def test_disclosure_closes_recovery_from_coupled_equations(): + raw = {"a": 5, "b": 5, "c": 6, "ab": 10, "bc": 11, "ac": 11} + relations = [ + (("ab",), (("a",), ("b",))), + (("bc",), (("b",), ("c",))), + (("ac",), (("a",), ("c",))), + ] + primary, _ = script.suppress_counts(raw) + primary_hidden = { + (name,) for name, value in primary.items() if value == "<10" + } + equations = [script._equation(relation) for relation in relations] + assert all(len(primary_hidden.intersection(eq)) == 2 for eq in equations) + # Each pair hides two cells, but together the three equations solve a,b,c. + assert set(script._recoverable(equations, primary_hidden)) == { + ("a",), + ("b",), + ("c",), + } + published, disclosure = script.suppress_counts(raw, relations) + assert published["ab"] == "<10" + assert published["bc"] == published["ac"] == 11 + assert disclosure["complementary_suppressed"] == 1 + hidden = {(name,) for name, value in published.items() if value == "<10"} + assert not script._recoverable(equations, hidden) + # Independently check every hidden coordinate remains underdetermined. + columns = sorted(hidden) + matrix = np.array( + [[float(eq.get(path, 0)) for path in columns] for eq in equations] + ) + rank = np.linalg.matrix_rank(matrix) + for column in range(len(columns)): + assert np.linalg.matrix_rank(np.delete(matrix, column, axis=1)) == rank + + +def test_disclosure_handles_overlapping_wave_pool_and_lag_intersection(): + # Five persons are in both waves; ten more occur only in each wave. + inputs = _inputs( + [ + *[_row(person, 2011, age=62) for person in range(1, 16)], + *[ + _row(person, 2013, age=64) + for person in [*range(1, 6), *range(16, 26)] + ], + ] + ) + people, labor, anchor, design = inputs + validity = _validity(labor) + raw = _counts(inputs) + relations = script.disclosure_relations(people, validity, anchor, design) + first = ("b3", "by_wave", "2011", "62_64", "persons") + second = ("b3", "by_wave", "2013", "62_64", "persons") + pooled = ("b3", "distinct_persons_by_band", "62_64") + overlap = ("b3", "lag_pairs", "2", "2011_2013", "62_64") + assert [ + _path_value(raw, path) for path in (first, second, pooled, overlap) + ] == [15, 15, 25, 5] + # A distinct-person union must not be treated as a sum of overlapping waves. + incorrect_addition = ( + pooled, + tuple( + ("b3", "by_wave", str(wave), "62_64", "persons") for wave in WAVES + ), + ) + assert incorrect_addition not in relations + equations = [script._equation(relation) for relation in relations] + + def hidden_paths(value, prefix=()): + if isinstance(value, dict): + return set().union( + *( + hidden_paths(item, (*prefix, key)) + for key, item in value.items() + ) + ) + return {prefix} if value == "<10" else set() + + primary, _ = script.suppress_counts(raw) + # Wave1 + wave2 - pooled would otherwise recover the hidden overlap. + assert overlap in script._recoverable(equations, hidden_paths(primary)) + published, _ = script.suppress_counts(raw, relations) + assert _path_value(published, overlap) == "<10" + assert any( + _path_value(published, path) == "<10" + for path in (first, second, pooled) + ) + assert not script._recoverable(equations, hidden_paths(published)) + + +# -------------------------------------------------------------------------- +# Output guards, checked before a reader can be reached +# -------------------------------------------------------------------------- +def test_main_refuses_to_overwrite(tmp_path, monkeypatch): + existing = tmp_path / "counts.json" + existing.write_text("{}") + + def no_read(*args, **kwargs): + pytest.fail("output refusal must happen before any input is read") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + with pytest.raises(SystemExit, match="refusing to overwrite"): + script.main( + [ + "--out", + str(existing), + "--local-copy", + str(tmp_path / "local.json"), + ] + ) + assert existing.read_text() == "{}" + + +def test_main_refuses_committed_evidence_directory(tmp_path, monkeypatch): + evidence = tmp_path / "".join(("ru", "ns")) + monkeypatch.setattr(script, "RUNS_DIR", evidence) + + def no_read(*args, **kwargs): + pytest.fail("output refusal must happen before any input is read") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + target = evidence / "nested" / "counts.json" + with pytest.raises(SystemExit, match="writes no file"): + script.main( + [ + "--out", + str(target), + "--local-copy", + str(tmp_path / "local.json"), + ] + ) + assert not target.exists() + + +def test_main_requires_local_copy_before_any_input_read(tmp_path, monkeypatch): + def no_read(*args, **kwargs): + pytest.fail( + "missing local-copy refusal must happen before input reads" + ) + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + with pytest.raises(SystemExit) as error: + script.main(["--out", str(tmp_path / "counts.json")]) + assert error.value.code == 2 + + +@pytest.mark.parametrize("relative", [False, True]) +def test_main_refuses_local_copy_inside_repository_before_read( + relative, tmp_path, monkeypatch +): + def no_read(*args, **kwargs): + pytest.fail("local-copy refusal must happen before input reads") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + local = Path("invented_private_counts.json") + if not relative: + local = ROOT / local + with pytest.raises(SystemExit, match="outside.*repository"): + script.main( + [ + "--out", + str(tmp_path / "counts.json"), + "--local-copy", + str(local), + ] + ) + assert not (ROOT / "invented_private_counts.json").exists() + + +def test_main_refuses_local_copy_through_repository_symlink_before_read( + tmp_path, monkeypatch +): + alias = tmp_path / "repository_alias" + alias.symlink_to(ROOT, target_is_directory=True) + + def no_read(*args, **kwargs): + pytest.fail("local-copy refusal must happen before input reads") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + with pytest.raises(SystemExit, match="outside.*repository"): + script.main( + [ + "--out", + str(tmp_path / "counts.json"), + "--local-copy", + str(alias / "invented_private_counts.json"), + ] + ) + assert not (ROOT / "invented_private_counts.json").exists() + + +def test_main_refuses_to_overwrite_local_copy_before_read( + tmp_path, monkeypatch +): + local = tmp_path / "local.json" + local.write_text("preserve local bytes") + out = tmp_path / "counts.json" + + def no_read(*args, **kwargs): + pytest.fail("local-copy refusal must happen before input reads") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + with pytest.raises(SystemExit, match="refusing to overwrite"): + script.main(["--out", str(out), "--local-copy", str(local)]) + assert local.read_text() == "preserve local bytes" + assert not out.exists() + + +@pytest.mark.parametrize("target", ["out", "local"]) +def test_main_refuses_dangling_output_symlinks_before_read( + target, tmp_path, monkeypatch +): + out = tmp_path / "counts.json" + local = tmp_path / "local.json" + guarded = out if target == "out" else local + guarded.symlink_to(tmp_path / "absent.json") + + def no_read(*args, **kwargs): + pytest.fail("symlink refusal must happen before input reads") + + monkeypatch.setattr(script.psid, "_resolve_data_dir", no_read) + with pytest.raises(SystemExit, match="refusing to overwrite"): + script.main(["--out", str(out), "--local-copy", str(local)]) + assert guarded.is_symlink() + assert not (tmp_path / "absent.json").exists() + + +@pytest.fixture +def invented_main(tmp_path, monkeypatch): + people, labor, anchor, design = _worked_rosters() + allowlist = {"invented": {"variable": "X1", "label": "AGE"}} + supplied = (people, _validity(labor), anchor, design, allowlist) + input_root = tmp_path / "synthetic_inputs" + calls = [] + + def read_inputs(path): + assert path == input_root + calls.append(path) + return supplied + + @contextmanager + def record_files_read(path): + assert path == input_root + # Only a mocked audit record is supplied; no input file is opened. + yield {"invented.txt": "a" * 64} + + monkeypatch.setattr(script, "read_inputs", read_inputs) + monkeypatch.setattr( + script, "_git", lambda *args: "" if args[0] == "status" else "b" * 40 + ) + monkeypatch.setattr( + script.psid, "_resolve_data_dir", lambda arg: input_root + ) + monkeypatch.setattr( + script.psid2010, "record_files_read", record_files_read + ) + baseline = tmp_path / "b01.json" + baseline.write_text(json.dumps({"counts": {"domain_persons": 2}})) + monkeypatch.setattr(script, "B0_1_COUNTS", baseline) + for name in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS"): + monkeypatch.setenv(name, "1") + monkeypatch.setenv("OMP_WAIT_POLICY", "PASSIVE") + return supplied, baseline, calls + + +def test_main_records_provenance_and_checks_b01_size(tmp_path, invented_main): + supplied, baseline, calls = invented_main + out = tmp_path / "counts.json" + assert ( + script.main( + ["--out", str(out), "--local-copy", str(tmp_path / "local.json")] + ) + == 0 + ) + record = json.loads(out.read_text()) + assert len(calls) == 1 + assert record["schema"] == "track_b_b0_2_counts.v1" + assert record["repository_head"] == "b" * 40 + assert record["worktree_clean"] is True + assert ( + record["script_sha256"] + == hashlib.sha256(Path(script.__file__).read_bytes()).hexdigest() + ) + assert ( + record["b0_1_counts_sha256"] + == hashlib.sha256(baseline.read_bytes()).hexdigest() + ) + assert record["psid_files_opened_sha256"] == {"invented.txt": "a" * 64} + assert record["requested_label_allowlist"] == supplied[-1] + raw_counts = script.structural_counts(*supplied[:-1]) + relations = script.disclosure_relations(*supplied[:-1]) + suppressed, disclosure = script.suppress_counts(raw_counts, relations) + assert record["counts"] == suppressed + assert record["disclosure"] == disclosure + assert disclosure["threshold"] == 10 + assert isinstance(disclosure["rule"], str) and disclosure["rule"] + assert disclosure["primary_suppressed"] > 0 + assert disclosure["complementary_suppressed"] >= 0 + assert ( + sum(disclosure["complementary_passes"]) + == disclosure["complementary_suppressed"] + ) + local_record = json.loads((tmp_path / "local.json").read_text()) + assert local_record["counts"] == raw_counts + assert local_record["repository_head"] == record["repository_head"] + assert record["geometry"]["collection_waves"] == list(WAVES) + assert record["geometry"]["reference_years"] == [ + wave - 1 for wave in WAVES + ] + assert record["geometry"]["age_convention"] == "collection_wave" + assert record["geometry"]["lag_years"] == [2, 4] + _assert_no_forbidden_output(record) + + +def test_main_allows_relative_local_copy_resolving_outside_repository( + tmp_path, invented_main +): + out = tmp_path / "counts.json" + local = tmp_path / "local.json" + relative_local = os.path.relpath(local, ROOT) + assert not (ROOT / relative_local).resolve().is_relative_to(ROOT) + assert ( + script.main(["--out", str(out), "--local-copy", relative_local]) == 0 + ) + assert out.exists() + assert local.exists() + assert json.loads(local.read_text())["counts"] == script.structural_counts( + *invented_main[0][:-1] + ) + + +def test_main_rejects_b01_size_mismatch(tmp_path, invented_main): + _, baseline, _ = invented_main + baseline.write_text(json.dumps({"counts": {"domain_persons": 99}})) + out = tmp_path / "counts.json" + with pytest.raises(AssertionError, match="R_B2 size differs"): + script.main( + ["--out", str(out), "--local-copy", str(tmp_path / "local.json")] + ) + assert not out.exists() + + +def test_main_uses_exclusive_open_if_output_appears_during_read( + tmp_path, monkeypatch, invented_main +): + supplied, _, _ = invented_main + out = tmp_path / "counts.json" + + def racing_read(path): + out.write_text("preserve these bytes") + return supplied + + monkeypatch.setattr(script, "read_inputs", racing_read) + with pytest.raises(FileExistsError): + script.main( + ["--out", str(out), "--local-copy", str(tmp_path / "local.json")] + ) + assert out.read_text() == "preserve these bytes" + + +def test_main_uses_exclusive_open_if_local_copy_appears_during_read( + tmp_path, monkeypatch, invented_main +): + supplied, _, _ = invented_main + out = tmp_path / "counts.json" + local = tmp_path / "local.json" + + def racing_read(path): + local.write_text("preserve local race bytes") + return supplied + + monkeypatch.setattr(script, "read_inputs", racing_read) + with pytest.raises(FileExistsError): + script.main(["--out", str(out), "--local-copy", str(local)]) + assert local.read_text() == "preserve local race bytes" + assert not out.exists() + + +@pytest.mark.parametrize( + ("name", "value"), + [ + ("OMP_NUM_THREADS", "0"), + ("OMP_NUM_THREADS", "5"), + ("OPENBLAS_NUM_THREADS", "invalid"), + ("OMP_WAIT_POLICY", "ACTIVE"), + ], +) +def test_runtime_rejects_thread_settings_before_read( + name, value, tmp_path, monkeypatch, invented_main +): + _, _, calls = invented_main + monkeypatch.setenv(name, value) + with pytest.raises(SystemExit, match=f"set {name}"): + script.main( + [ + "--out", + str(tmp_path / "counts.json"), + "--local-copy", + str(tmp_path / "local.json"), + ] + ) + assert not calls + + +def test_runtime_rejects_dirty_worktree_before_read( + tmp_path, monkeypatch, invented_main +): + _, _, calls = invented_main + monkeypatch.setattr(script, "_git", lambda *args: " M invented.py") + with pytest.raises(SystemExit, match="clean protocol-freezing worktree"): + script.main( + [ + "--out", + str(tmp_path / "counts.json"), + "--local-copy", + str(tmp_path / "local.json"), + ] + ) + assert not calls + + +def test_module_stays_in_unit_tier(tier_policy_collection): + own = [ + markers + for node, markers in tier_policy_collection["items"] + if node.startswith("tests/test_track_b_b0_2_counts.py::") + ] + assert own and all(markers == ("unit",) for markers in own) + + +_READER_CONCEPTS = { + "age": (r"^AGE OF INDIVIDUAL",), + "sequence": (r"^SEQUENCE NUMBER",), + "relationship": (r"^RELATION(SHIP)? TO (HEAD|REFERENCE PERSON)",), + "weight": ( + r"^INDIVIDUAL WEIGHT|CROSS-SECTION WT", + r"^CORE IND WEIGHT", + r"LONGITUDINAL W", + ), + "interview": (r"INTERVIEW NUMBER",), +} + +_READER_ACCURACY = { + 1999: { + "head": (("ER16464", "ACC LABOR INCOME-HD"),), + "spouse": (("ER16466", "ACC LABOR INCOME-WF"),), + }, + 2001: { + "head": ( + ("ER20426", "ACC WAGES AND SALARIES OF HEAD-2000"), + ("ER20440", "ACC MISC LABOR INCOME OF HEAD-2000"), + ), + "spouse": (("ER20448", "ACC LABOR INCOME OF WIFE-2000"),), + }, + 2003: { + "head": ( + ("ER24118", "ACC WAGES AND SALARIES OF HEAD LAST YEAR"), + ("ER24134", "ACC MISC LABOR INCOME OF HEAD LAST YEAR"), + ), + "spouse": (("ER24136", "ACC LABOR INCOME OF WIFE LAST YEAR"),), + }, + 2005: { + "head": ( + ("ER27914", "ACC WAGES AND SALARIES OF HEAD-2004"), + ("ER27930", "ACC MISC LABOR INCOME OF HEAD-2004"), + ), + "spouse": (("ER27944", "ACC LABOR INCOME OF WIFE-2004"),), + }, + 2007: { + "head": ( + ("ER40904", "ACC WAGES AND SALARIES OF HEAD-2006"), + ("ER40920", "ACC MISC LABOR INCOME OF HEAD-2006"), + ), + "spouse": (("ER40934", "ACC LABOR INCOME OF WIFE-2006"),), + }, + 2009: { + "head": ( + ("ER46812", "ACC WAGES AND SALARIES OF HEAD-2008"), + ("ER46828", "ACC MISC LABOR INCOME OF HEAD-2008"), + ), + "spouse": (("ER46842", "ACC LABOR INCOME OF WIFE-2008"),), + }, + 2011: { + "head": ( + ("ER52220", "ACC WAGES AND SALARIES OF HEAD-2010"), + ("ER52236", "ACC MISC LABOR INCOME OF HEAD-2010"), + ), + "spouse": (("ER52250", "ACC LABOR INCOME OF WIFE-2010"),), + }, + 2013: { + "head": ( + ("ER58021", "ACC WAGES AND SALARIES OF HEAD-2012"), + ("ER58037", "ACC MISC LABOR INCOME OF HEAD-2012"), + ), + "spouse": (("ER58051", "ACC LABOR INCOME OF WIFE-2012"),), + }, + 2015: { + "head": ( + ("ER65201", "ACC WAGES AND SALARIES OF HEAD-2014"), + ("ER65215", "ACC MISC LABOR INCOME OF HEAD-2014"), + ), + "spouse": ( + ("ER65229", "ACC WAGES AND SALARIES OF SPOUSE-2014"), + ("ER65243", "ACC MISC LABOR INCOME OF SPOUSE-2014"), + ), + }, + 2017: { + "head": ( + ("ER71278", "ACC WAGES AND SALARIES OF RP-2016"), + ("ER71292", "ACC MISC LABOR INCOME OF RP-2016"), + ), + "spouse": ( + ("ER71306", "ACC WAGES AND SALARIES OF SPOUSE-2016"), + ("ER71320", "ACC MISC LABOR INCOME OF SPOUSE-2016"), + ), + }, + 2019: { + "head": ( + ("ER77300", "ACC WAGES AND SALARIES OF RP-2018"), + ("ER77314", "ACC MISC LABOR INCOME OF RP-2018"), + ), + "spouse": ( + ("ER77328", "ACC WAGES AND SALARIES OF SPOUSE-2018"), + ("ER77342", "ACC MISC LABOR INCOME OF SPOUSE-2018"), + ), + }, + 2021: { + "head": ( + ("ER81627", "ACC WAGES AND SALARIES OF RP-2020"), + ("ER81641", "ACC MISC LABOR INCOME OF RP-2020"), + ), + "spouse": ( + ("ER81655", "ACC WAGES AND SALARIES OF SPOUSE-2020"), + ("ER81669", "ACC MISC LABOR INCOME OF SPOUSE-2020"), + ), + }, + 2023: { + "head": ( + ("ER85481", "ACC WAGES AND SALARIES OF RP-2022"), + ("ER85495", "ACC MISC LABOR INCOME OF RP-2022"), + ), + "spouse": ( + ("ER85509", "ACC WAGES AND SALARIES OF SPOUSE-2022"), + ("ER85523", "ACC MISC LABOR INCOME OF SPOUSE-2022"), + ), + }, +} + + +def _reader_metadata(): + """Invent totals/demographics; pin source-defined identities/accuracy.""" + identifiers = { + "interview_1968": { + "variable": "ER30001", + "label": "1968 INTERVIEW NUMBER", + }, + "person_number": {"variable": "ER30002", "label": "PERSON NUMBER 68"}, + } + anchor = { + **identifiers, + "interview": {"variable": "ER34101", "label": "2011 INTERVIEW NUMBER"}, + "sequence": {"variable": "ER34102", "label": "SEQUENCE NUMBER 11"}, + "relationship": { + "variable": "ER34103", + "label": "RELATION TO HEAD 11", + }, + "age": {"variable": "ER34104", "label": "AGE OF INDIVIDUAL 11"}, + "reported_birth_year": { + "variable": "ER34106", + "label": "YEAR INDIVIDUAL BORN 11", + }, + "weight": { + "variable": "ER34155", + "label": "CORE/IMM INDIVIDUAL CROSS-SECTION WT 11", + }, + } + design = { + **identifiers, + "stratum": {"variable": "ER31996", "label": "SAMPLING ERROR STRATUM"}, + "cluster": {"variable": "ER31997", "label": "SAMPLING ERROR CLUSTER"}, + } + labels = { + entry["variable"]: entry["label"] + for entry in [*anchor.values(), *design.values()] + } + by_wave = {} + stems = { + "age": "AGE OF INDIVIDUAL", + "sequence": "SEQUENCE NUMBER", + "relationship": "RELATION TO HEAD", + "weight": "INDIVIDUAL WEIGHT", + "interview": "INTERVIEW NUMBER", + } + for wave in range(1999, 2024, 2): + by_wave[str(wave)] = {} + for concept, stem in stems.items(): + if wave == 2011: + entry = anchor[concept] + else: + entry = { + "variable": f"INVENTED_{concept}_{wave}", + "label": ( + f"{wave} {stem}" + if concept == "interview" + else f"{stem} {wave % 100:02d}" + ), + } + by_wave[str(wave)][concept] = entry + labels[entry["variable"]] = entry["label"] + labels["FORBIDDEN_INDIVIDUAL"] = "MARITAL STATUS 11" + family_labels = {} + families = {} + for wave in range(1999, 2024, 2): + entries = { + "interview": { + "variable": f"INVENTED_INTERVIEW_{wave}", + "label": f"{wave} FAMILY INTERVIEW (ID) NUMBER", + }, + "head_labor": { + "variable": f"INVENTED_HEAD_{wave}", + "label": f"LABOR INCOME OF HEAD-{wave - 1}", + }, + "spouse_labor": { + "variable": f"INVENTED_SPOUSE_{wave}", + "label": f"LABOR INCOME OF WIFE-{wave - 1}", + }, + "head_accuracy": [ + {"variable": var, "label": label} + for var, label in _READER_ACCURACY[wave]["head"] + ], + "spouse_accuracy": [ + {"variable": var, "label": label} + for var, label in _READER_ACCURACY[wave]["spouse"] + ], + } + families[str(wave)] = entries + flat = [ + entries["interview"], + entries["head_labor"], + entries["spouse_labor"], + *entries["head_accuracy"], + *entries["spouse_accuracy"], + ] + family_labels[wave] = { + entry["variable"]: entry["label"] for entry in flat + } + family_labels[wave]["FORBIDDEN_FAMILY"] = "SOCIAL SECURITY AMOUNT" + expected = { + "individual": { + "identifiers": identifiers, + "concepts": { + name: list(patterns) + for name, patterns in _READER_CONCEPTS.items() + }, + "by_wave": by_wave, + }, + "anchor_2011": anchor, + "design": design, + "family": families, + } + return labels, family_labels, expected + + +def test_readers_request_the_pinned_allowlist(monkeypatch, tmp_path): + """Exercise the named readers with mocked metadata and fixed-width IO.""" + import hashlib + import json + + labels, family_labels, expected = _reader_metadata() + waves = list(range(1999, 2024, 2)) + source_dir = tmp_path / "invented_survey" + calls = [] + low_level = [] + family_reads = [] + by_wave = expected["individual"]["by_wave"] + + def resolve(data_dir): + assert data_dir == source_dir + return source_dir + + def parse_labels(path): + if path.name == "IND2023ER.sps": + return labels.copy() + return family_labels[int(path.parent.name)].copy() + + def family_paths(wave, data_dir): + assert data_dir == source_dir + base = source_dir / "family" / str(wave) + return base / "invented.sps", base / "invented.txt" + + def parse_layout(path): + names = list(family_labels[int(path.parent.name)]) + return pd.DataFrame( + { + "name": names, + "start": range(1, len(names) + 1), + "end": range(1, len(names) + 1), + } + ) + + individual_values = { + "ER30001": 1, + "ER30002": 1, + "ER34106": 1950, + "ER31996": 5, + "ER31997": 7, + } + values = { + "age": 62, + "sequence": 1, + "relationship": 10, + "weight": 2, + "interview": 1, + } + for entries in by_wave.values(): + for concept, entry in entries.items(): + individual_values[entry["variable"]] = values[concept] + + def read_psid(product, *, columns, data_dir, nrows): + assert product == "ind2023er" + assert data_dir == source_dir + assert nrows is None + low_level.append(columns) + return pd.DataFrame( + {name: [individual_values[name]] for name in columns} + ) + + def read_fwf(path, *, colspecs, names, header, nrows): + wave = int(path.parent.name) + assert header is None + assert nrows is None + assert colspecs == [(i, i + 1) for i in range(len(names))] + family_reads.append((wave, names)) + return pd.DataFrame({name: [1] for name in names}) + + original_panel = script.panels.ind_person_period + original_anchor = script.psid2010.read_anchor_wave + original_design = script.age67.read_design_variables + original_family = script.family.read_family_labor + + def read_panel(concepts, **kwargs): + calls.append(("panel", concepts, kwargs)) + return original_panel(concepts, **kwargs) + + def read_anchor(**kwargs): + calls.append(("anchor", kwargs)) + return original_anchor(**kwargs) + + def read_design(**kwargs): + calls.append(("design", kwargs)) + return original_design(**kwargs) + + def read_family(wave, **kwargs): + calls.append(("family", wave, kwargs)) + return original_family(wave, **kwargs) + + monkeypatch.setattr(script.psid, "_resolve_data_dir", resolve) + monkeypatch.setattr(script.psid, "parse_sps_labels", parse_labels) + monkeypatch.setattr(script.psid, "parse_sps_layout", parse_layout) + monkeypatch.setattr(script.psid, "read_psid", read_psid) + monkeypatch.setattr(script.family, "_family_paths", family_paths) + monkeypatch.setattr(pd, "read_fwf", read_fwf) + monkeypatch.setattr(script.panels, "ind_person_period", read_panel) + monkeypatch.setattr(script.psid2010, "read_anchor_wave", read_anchor) + monkeypatch.setattr(script.age67, "read_design_variables", read_design) + monkeypatch.setattr(script.family, "read_family_labor", read_family) + + person_wave, validity, anchor, design, allowlist = script.read_inputs( + source_dir + ) + assert calls == [ + ("panel", _READER_CONCEPTS, {"data_dir": source_dir, "waves": waves}), + ("anchor", {"data_dir": source_dir, "anchor_wave": 2011}), + ("design", {"data_dir": source_dir}), + *[("family", wave, {"data_dir": source_dir}) for wave in waves], + ] + assert low_level == [ + ["ER30001", "ER30002"] + + [ + by_wave[str(wave)][concept]["variable"] + for wave in waves + for concept in _READER_CONCEPTS + ], + [ + "ER30001", + "ER30002", + "ER34101", + "ER34102", + "ER34103", + "ER34104", + "ER34106", + "ER34155", + ], + ["ER30001", "ER30002", "ER31996", "ER31997"], + ] + assert family_reads == [ + ( + wave, + [ + name + for name in family_labels[wave] + if name != "FORBIDDEN_FAMILY" + ], + ) + for wave in waves + ] + assert allowlist == expected + _assert_no_forbidden_output(allowlist) + encoded = json.dumps( + allowlist, sort_keys=True, separators=(",", ":") + ).encode() + assert hashlib.sha256(encoded).hexdigest() == ( + "04b0d7683e935a8a841dea9e866a228ef643aa6e799bcff47f1ed16d20bf9425" + ) + assert person_wave["period"].tolist() == waves + assert set(validity) == set(waves) + assert "reported_birth_year" not in anchor.columns + assert anchor["person_id"].tolist() == [1001] + assert design.to_dict("records") == [ + {"person_id": 1001, "stratum": 5, "cluster": 7} + ] + for frame in validity.values(): + assert frame.columns.tolist() == [ + "interview", + "head_valid", + "spouse_valid", + "head_acc_flag", + "spouse_acc_flag", + ] + assert frame.iloc[0].tolist() == [1, True, True, True, True] diff --git a/tests/tier_counts.json b/tests/tier_counts.json index 3841240d4..bf5d3a222 100644 --- a/tests/tier_counts.json +++ b/tests/tier_counts.json @@ -1,7 +1,7 @@ { "schema_version": 1, "counts": { - "unit": 9958, + "unit": 10017, "artifact": 4253, "integration_psid": 1450, "reproduction_legacy": 520, From ec9eb4864bc67def4a7591ea4290fd68778fca38 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 08:22:24 -0400 Subject: [PATCH 11/26] B0.2 protocol: the freeze is an explicit status commit after review, not the script's arrival The preamble said the protocol froze with the commit adding the count script, which be7d4f42 now matches. The freeze is instead a later commit that changes only the status line to FROZEN, after an independent review with no blocking finding. The pinned block is unchanged (94e22e3f...). Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_protocol.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/design/track_b_b0_2_protocol.md b/docs/design/track_b_b0_2_protocol.md index 7fed90e3b..e16819449 100644 --- a/docs/design/track_b_b0_2_protocol.md +++ b/docs/design/track_b_b0_2_protocol.md @@ -11,12 +11,16 @@ answer. It is the first count step the landscape synthesis identifies A later version, recorded as such, will cover the other sources (HRS, SIPP, SSA tables) once Max chooses them. -The protocol is frozen by the commit that adds the counting script -`scripts/track_b_b0_2_counts.py`, its tests, and this file with the -block below unchanged. The unit test `tests/test_track_b_b0_2_counts.py` +The counting script `scripts/track_b_b0_2_counts.py` and its tests +arrived in `be7d4f42`. The unit test `tests/test_track_b_b0_2_counts.py` pins the block's SHA-256, taken over the text from the begin marker -through the end marker. The script then runs once, from a clean worktree -at that commit. +through the end marker. Adding the script did not freeze the protocol. +The freeze is a later commit that changes only the status line above +to FROZEN and states the pinned SHA-256. It follows an independent +review of the block and the script that leaves no blocking finding. The +script then runs once, from a clean worktree at that commit. If the +review changes the block, the pin and the script change with it before +the freeze. ## Structural-audit protocol, version 1 (frozen before any count) From 7165a801d3c410b9798d19c9da3e7ae3fb49b4c9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 08:26:43 -0400 Subject: [PATCH 12/26] B0.2: a standing exclusion record, so successor review roots never hold B0.2's work The coordinator's separation rule (2026-10-21) needs an exclusion inventory in the q6_exclusions form before any B0.2 count reaches master. docs/design/track_b_b0_2_exclusions.json lists every B0.2 file, whole: plan, landscape, protocol, count script and tests, the build record, and, ahead of their arrival, the counts file and the exposure inventory. scripts/track_b_b0_1_review_root.py now applies it as a second standing record, so every blinded review root excludes these paths whatever inventories a build names. tests/test_track_b_b0_2_exclusions.py pins the record's contents literally, requires every tracked file named track_b_b0_2_* to be listed (so a new one cannot reach master unlisted), and checks that a build removes them. The B2 outcome test's exact assertions now include the second record. Mutation checks, each restored: dropping the record from STANDING_EXCLUSIONS (2 failed); omitting the counts file (2 failed); a lines scope (1 failed); an unlisted tracked B0.2 file (1 failed). Tests: 10 passed with the B2 exclusion tests; 141 review-root tests passed. Black and ruff clean. Tier counts: unit 10,017 -> 10,021. Co-Authored-By: Claude Opus 5.5 --- docs/design/track_b_b0_2_exclusions.json | 52 +++++++++ scripts/track_b_b0_1_review_root.py | 10 +- tests/test_track_b_b0_2_exclusions.py | 116 ++++++++++++++++++++ tests/test_track_b_b2_outcome_exclusions.py | 19 +++- tests/tier_counts.json | 2 +- 5 files changed, 192 insertions(+), 7 deletions(-) create mode 100644 docs/design/track_b_b0_2_exclusions.json create mode 100644 tests/test_track_b_b0_2_exclusions.py diff --git a/docs/design/track_b_b0_2_exclusions.json b/docs/design/track_b_b0_2_exclusions.json new file mode 100644 index 000000000..ef415dce0 --- /dev/null +++ b/docs/design/track_b_b0_2_exclusions.json @@ -0,0 +1,52 @@ +{ + "added": "2026-10-23", + "q6_exclusions": [ + { + "lines": null, + "path": "docs/design/track_b_b0_2_counts.json", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_counts_mutations.json", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_counts_report.txt", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_exposure_inventory.json", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_landscape.md", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_plan.md", + "scope": "whole_file" + }, + { + "lines": null, + "path": "docs/design/track_b_b0_2_protocol.md", + "scope": "whole_file" + }, + { + "lines": null, + "path": "scripts/track_b_b0_2_counts.py", + "scope": "whole_file" + }, + { + "lines": null, + "path": "tests/test_track_b_b0_2_counts.py", + "scope": "whole_file" + } + ], + "reason": "The dynamics coordinator separated B0.2 from the B2 successor design on 2026-10-21: successor-design roles may not read B0.2's work (PR #554 beyond its title). These are B0.2's files, including the structural counts and exposure inventory it will add; the counts must be listed here before they reach master. No exposure-inventory scan lists them, so this standing record excludes them from every blinded review root that scripts/track_b_b0_1_review_root.py builds.", + "schema": "track_b_b0_2_exclusions.v1" +} diff --git a/scripts/track_b_b0_1_review_root.py b/scripts/track_b_b0_1_review_root.py index 376c1f9c9..8dcae6b1e 100644 --- a/scripts/track_b_b0_1_review_root.py +++ b/scripts/track_b_b0_1_review_root.py @@ -11,7 +11,7 @@ exposure inventory flags when rerun at that commit; - every file on a standing exclusion record (``STANDING_EXCLUSIONS``), whatever inventories the build names: B2's run record and attempt - ledger, which no inventory scan lists; + ledger, and B0.2's files, which no inventory scan lists; - ``data/external/psid_codebook_field_evidence/``, whose code maps carry frequency columns for B2's 2012 and 2014 labor-income variables; - any other file that holds both a code map and frequencies, unless it @@ -62,8 +62,12 @@ #: Standing exclusion records, in the inventories' ``q6_exclusions`` form. #: Every root excludes their paths, whatever inventories a build names: they #: hold values a blind reviewer must not see, and no exposure-inventory scan -#: lists them. -STANDING_EXCLUSIONS = ("docs/design/track_b_b2_outcome_exclusions.json",) +#: lists them. The second is B0.2's work, which successor-design roles may +#: not read. +STANDING_EXCLUSIONS = ( + "docs/design/track_b_b2_outcome_exclusions.json", + "docs/design/track_b_b0_2_exclusions.json", +) #: A file holding both of these is a codebook table and is removed. CODE_MAP_RX = re.compile(r"code_map", re.IGNORECASE) FREQUENCY_RX = re.compile(r"frequenc", re.IGNORECASE) diff --git a/tests/test_track_b_b0_2_exclusions.py b/tests/test_track_b_b0_2_exclusions.py new file mode 100644 index 000000000..4ce900d8d --- /dev/null +++ b/tests/test_track_b_b0_2_exclusions.py @@ -0,0 +1,116 @@ +"""B0.2's files stay out of every blinded review root. + +The dynamics coordinator separated B0.2 from the B2 successor design on +2026-10-21: successor-design roles may not read B0.2's work. B0.2's +structural counts must be on an exclusion inventory before they reach +master. ``docs/design/track_b_b0_2_exclusions.json`` is that inventory, a +standing record that ``scripts/track_b_b0_1_review_root.py`` applies to +every build, whatever inventories the build names. + +The record lists two files before they exist: the counts the frozen +protocol writes, and B0.2's exposure inventory. Every other B0.2 file in +the repository must be listed, so a new one cannot reach master unlisted. +""" + +from __future__ import annotations + +import io +import json +import subprocess +import sys +import tarfile +from pathlib import Path +from types import SimpleNamespace + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) + +import track_b_b0_1_review_root as review_root # noqa: E402 + +RECORD_PATH = "docs/design/track_b_b0_2_exclusions.json" +RECORD = ROOT / RECORD_PATH +#: Listed ahead of their arrival. +NOT_YET_COMMITTED = { + "docs/design/track_b_b0_2_counts.json", + "docs/design/track_b_b0_2_exposure_inventory.json", +} +B0_2_FILES = NOT_YET_COMMITTED | { + "docs/design/track_b_b0_2_counts_mutations.json", + "docs/design/track_b_b0_2_counts_report.txt", + "docs/design/track_b_b0_2_landscape.md", + "docs/design/track_b_b0_2_plan.md", + "docs/design/track_b_b0_2_protocol.md", + "scripts/track_b_b0_2_counts.py", + "tests/test_track_b_b0_2_counts.py", +} +#: Name B0.2's files without holding any of their content. +NOT_EXCLUDED = {RECORD_PATH, "tests/test_track_b_b0_2_exclusions.py"} + + +def b0_2_named(path: str) -> bool: + """Whether a repository path is named as one of B0.2's files.""" + name = path.rsplit("/", 1)[-1] + return name.startswith(("track_b_b0_2_", "test_track_b_b0_2_")) + + +def record(): + return json.loads(RECORD.read_text(encoding="utf-8")) + + +def test_the_standing_record_lists_b0_2s_files(): + entries = record()["q6_exclusions"] + assert record()["schema"] == "track_b_b0_2_exclusions.v1" + paths = [entry["path"] for entry in entries] + assert paths == sorted(B0_2_FILES) + assert all(entry["scope"] == "whole_file" for entry in entries) + assert all(entry["lines"] is None for entry in entries) + assert RECORD_PATH in review_root.STANDING_EXCLUSIONS + denied = set(review_root.denied_paths(review_root.standing_inventories())) + assert B0_2_FILES <= denied + + +def test_every_b0_2_file_in_the_repository_is_listed(): + tracked = subprocess.run( + ["git", "ls-files", "-z"], + cwd=ROOT, + capture_output=True, + check=True, + ).stdout.decode() + named = {path for path in tracked.split("\0") if b0_2_named(path)} + assert named - NOT_EXCLUDED <= B0_2_FILES + for relative in B0_2_FILES - NOT_YET_COMMITTED: + assert relative in named, relative + + +def test_the_name_rule_catches_b0_2s_files_only(): + assert b0_2_named("docs/design/track_b_b0_2_counts.json") + assert b0_2_named("tests/test_track_b_b0_2_new.py") + assert b0_2_named("track_b_b0_2_top.md") + assert not b0_2_named("docs/design/track_b_b0_1_counts.json") + assert not b0_2_named("docs/design/track_b_b0_2x/plain.md") + assert not b0_2_named("scripts/old_track_b_b0_2_counts.py") + + +def test_build_removes_b0_2s_files_without_any_inventory( + tmp_path, monkeypatch +): + files = {relative: "invented" for relative in B0_2_FILES} + files["docs/keep.md"] = "kept text" + files[RECORD_PATH] = "{}" + + def run(command, cwd=None, check=False): + archive = Path(command[command.index("-o") + 1]) + with tarfile.open(archive, "w") as tar: + for relative, content in files.items(): + data = content.encode() + info = tarfile.TarInfo(relative) + info.size = len(data) + tar.addfile(info, io.BytesIO(data)) + + monkeypatch.setattr(review_root, "subprocess", SimpleNamespace(run=run)) + out = tmp_path / "root" + built = review_root.build("h", out, [], {}) + assert B0_2_FILES <= set(built["removed"]) + for relative in B0_2_FILES: + assert not (out / relative).exists(), relative + assert set(review_root.manifest(out)) == {"docs/keep.md", RECORD_PATH} diff --git a/tests/test_track_b_b2_outcome_exclusions.py b/tests/test_track_b_b2_outcome_exclusions.py index 17ae59f8d..0f2f1ad0e 100644 --- a/tests/test_track_b_b2_outcome_exclusions.py +++ b/tests/test_track_b_b2_outcome_exclusions.py @@ -27,6 +27,9 @@ import track_b_b0_1_review_root as review_root # noqa: E402 RECORD = ROOT / "docs" / "design" / "track_b_b2_outcome_exclusions.json" +#: B0.2's standing record; tests/test_track_b_b0_2_exclusions.py pins its +#: contents. +B0_2_RECORD = ROOT / "docs" / "design" / "track_b_b0_2_exclusions.json" B2_OUTCOME_FILES = { "docs/design/track_b_attempt_ledger.md", "runs/track_b_b2_floor.json", @@ -46,9 +49,17 @@ def test_the_standing_record_lists_b2s_outcome_files(): assert (ROOT / relative).is_file(), relative assert review_root.STANDING_EXCLUSIONS == ( "docs/design/track_b_b2_outcome_exclusions.json", + "docs/design/track_b_b0_2_exclusions.json", ) standing = review_root.standing_inventories() - assert set(review_root.denied_paths(standing)) == B2_OUTCOME_FILES + assert set(review_root.denied_paths(standing)) == ( + B2_OUTCOME_FILES | b0_2_paths() + ) + + +def b0_2_paths(): + record = json.loads(B0_2_RECORD.read_text(encoding="utf-8")) + return {entry["path"] for entry in record["q6_exclusions"]} def test_a_missing_standing_record_refuses_rather_than_drops( @@ -118,7 +129,9 @@ def test_build_unions_named_and_standing_exclusions(tmp_path, monkeypatch): ) out = tmp_path / "root" record = review_root.build("h", out, [named], {}) - expected = sorted(B2_OUTCOME_FILES | {"docs/named.md", "docs/absent.md"}) + expected = sorted( + B2_OUTCOME_FILES | b0_2_paths() | {"docs/named.md", "docs/absent.md"} + ) assert record["denied_paths"] == expected assert record["removed"] == sorted(B2_OUTCOME_FILES | {"docs/named.md"}) assert set(review_root.manifest(out)) == {"docs/keep.md"} @@ -165,6 +178,6 @@ def test_standing_paths_are_denied_whatever_the_inventories(named): ) assert denied == sorted(set(denied)) assert B2_OUTCOME_FILES <= set(denied) - assert set(denied) == B2_OUTCOME_FILES | { + assert set(denied) == B2_OUTCOME_FILES | b0_2_paths() | { path for paths in named for path in paths } diff --git a/tests/tier_counts.json b/tests/tier_counts.json index bf5d3a222..4b2756d48 100644 --- a/tests/tier_counts.json +++ b/tests/tier_counts.json @@ -1,7 +1,7 @@ { "schema_version": 1, "counts": { - "unit": 10017, + "unit": 10021, "artifact": 4253, "integration_psid": 1450, "reproduction_legacy": 520, From f5d22c1d21f4ec92dbd6239672ce98e33fc06d4e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 23 Oct 2026 08:45:27 -0400 Subject: [PATCH 13/26] B0.2 protocol v1 (draft): answer review round 1 in the block Sol's round-1 review (ec9eb486) found two complete-release recoveries of small cells and a missing-age defect, all needing block changes: - Known age is 1-125, the range the repository's readers accept (cohorts.age67._MAX_AGE_CODE); 0 and 999 are unknown age, in no band and not in R_TA_PROXY. Roster tables gain an unknown-age cell, so the bands and unknown age partition each roster. - The disclosure rule now coarsens counts at base B (0 exact, 1..B-1 as " --- docs/design/track_b_b0_2_protocol.md | 145 +++++++++++++++++++++------ 1 file changed, 115 insertions(+), 30 deletions(-) diff --git a/docs/design/track_b_b0_2_protocol.md b/docs/design/track_b_b0_2_protocol.md index e16819449..ae2d5ee2d 100644 --- a/docs/design/track_b_b0_2_protocol.md +++ b/docs/design/track_b_b0_2_protocol.md @@ -22,6 +22,15 @@ script then runs once, from a clean worktree at that commit. If the review changes the block, the pin and the script change with it before the freeze. +Review round 1 (Sol, at `ec9eb486`) asked for changes, and the block was +revised before any freeze: +- **Ages:** a known-age definition, so a missing age (999) is in no band + and not in the Track A proxy. +- **Disclosure:** coarsening and an integer-programming audit, which + replace subtraction-based complementary suppression, after two + executed recoveries of small cells. +- **One run:** a frozen-status check and an attempt record. + ## Structural-audit protocol, version 1 (frozen before any count) @@ -121,10 +130,16 @@ stay byte-identical. wave's relationship codes, with a positive weight at *w*, and whose role's labor income is below the sentinel. This is B0.1's `earnings_rows` rule, imported unchanged. -- **Age bands.** +- **Known age.** An age is known when it is an integer from 1 to 125, + the range the repository's readers accept + (`cohorts.age67._MAX_AGE_CODE`, which treats larger values, such as + 999, as missing). Any other value, including 0 and 999, is an unknown + age. A person of unknown age is in no age band and in no age-defined + roster. +- **Age bands,** on known ages only. - Older bands: 62-64, 65-66, 67-69 (B3's bands). - Context bands: 25-44, 45-61. - - Roster bands: 0-24, 25-44, 45-61, 62-64, 65-66, 67-69, 70-79, 80 and + - Roster bands: 1-24, 25-44, 45-61, 62-64, 65-66, 67-69, 70-79, 80 and over. - **Rosters at the 2011 wave** (H's candidates): - `R_FULL`: persons present (sequence 1-20) with a positive individual @@ -134,13 +149,15 @@ stay byte-identical. B0.1's committed count (`docs/design/track_b_b0_1_counts.json`); this is a consistency check only. - `R_TA_PROXY`: persons present at 2011 with a positive 2011 - cross-sectional weight and a reported 2011 age of 30 or over. + cross-sectional weight and a known 2011 age of 30 or over. - It is a proxy for Track A's closed cohort (persons born in 1980 or earlier, `cohorts/psid2010.py`). - Track A's own birth-year law reads marriage histories and earnings, so version 1 does not reproduce it. The H record cites Track A's own record for the exact cohort. -- **Roster counts.** For each roster, overall and by roster band at 2011: +- **Roster counts.** For each roster, overall, by roster band at 2011, + and for unknown age at 2011, so that the bands and unknown age + partition the roster: - persons; - households (2011 interview numbers); - design strata and clusters; @@ -194,43 +211,111 @@ stay byte-identical. PSID's Conditions of Use, as agreed at the July 2026 download, allow "aggregate statistical reporting" only. Max's ruling d1899 adds "no -small-cell counts on GitHub". So the committed output follows these -rules: - -- **Suppression.** Any count from 1 to 9 is written as the string `"<10"`. - So is every effective size computed on a set whose row or person count - is below 10. A count of 0 is written as 0. -- **Complementary suppression.** Where a total and all but one of its - parts are published, a suppressed part could be recovered by - subtraction. In that case the smallest remaining published part of - the same total is also suppressed, and this is repeated until no - suppressed cell can be recovered from published cells of the same - table. -- **Local copy.** The unsuppressed counts are written to a second file, - outside the repository, in B0.2's evidence folder. That file is never - committed. -- **Tests.** Unit tests check both rules on invented tables, including a - table where subtraction would reveal a suppressed cell. - -The threshold of 10 is version 1's. If Max or the coordinator sets -another before the freeze, this section changes before it freezes; after -the freeze, a change is a new version. +small-cell counts on GitHub". Suppressing small cells, and their +complements by subtraction, is not enough: integer and nonnegativity +bounds, and remainders that are never published, can still pin a small +cell exactly (review round 1 of this protocol). So the committed output +is coarsened, then audited, by these rules. + +- **Coarsening at base B.** + - A count of 0 is written as 0. + - A count from 1 to B − 1 is written as the string `"