Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion data/external/track_u2/u1_identity.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
"draft_3_commit": "f7412e00cdcdba6ee1156b74fa93039f5329cb8f",
"draft_3_git_blob": "e07ee0d589d20f3b6d100e5d4ce173f9a653f5e9",
"version": "u2-draft-4",
"sha256": "77f30781d241fbc75c6ff056e0632632a8770b047c320db587dcda9a3af79d78",
"sha256": "c0c2685f8e23756895b66d09a6bbd618f930c488cc5ad5c52415f42b228d8f98",
"cited_lines_sha256": {
"74": "7b299c9e4473fd51d04bdfec51af83268e517181ab734d3e76ccd1c744f6daa9",
"75": "44454cc04e6a50863035d69ccb53edec175b4ad70a19d8877437901d2deb8428",
Expand Down
5 changes: 5 additions & 0 deletions docs/design/boomers2004_1946_55_comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -1884,6 +1884,11 @@ Revision verification comprised the two touched test files and `tests/track_u2/t
- The d1090 record test's follower lines for §17 item 9 and §20 step 2 move with their text.
- No count, statistic or outcome was computed, no model or estimator was run on real data, and no file on the restricted list was opened for reading; the U1 differential ran on invented data and its loader guard recorded no access. The revision's reading is recorded with the round-3 closeout lane in `docs/design/u2_m2_exposure.md`.

**`u2-draft-4`, Max's confirmation of B2's disclosed count recorded (2026-10-16, d1242):**

- Max confirmed §16c's proposed clarification (d1242): B2's disclosed count is the number of code-90 and code-92 persons in 2019, 2021 and 2023 family units that would otherwise supply any U2 observation. `sources.PERSON_COUNTED_EXCLUSIONS` now names B2 as well as B1, so the structural pass discloses both counts, and each decides nothing (§16c). Each is disclosed per row in two forms: per wave, whose sum counts a person who brings the exclusion in two waves twice, and as distinct persons across the waves, which is the number d1242 names. §16c's text, which records the proposal, is unchanged.
- No count, statistic or outcome was computed, no model or estimator was run on real data, and no file on the restricted list was opened for reading. The revision's reading is recorded with the round-3 closeout lane in `docs/design/u2_m2_exposure.md`.

## 20. Ratification and execution record

**Pending.**
Expand Down
3 changes: 2 additions & 1 deletion docs/design/u2_m2_exposure.json
Original file line number Diff line number Diff line change
Expand Up @@ -1114,6 +1114,7 @@
"guard_logs": "empty for mut-a, mut-c, b2, d2, both reviews and this lane; mut-b's and mut-d's were truncated unread by this lane when it reset those clones, and their stream logs show no refusal and no tool input naming a restricted directory",
"psid_documentation_opened_by_tests": "none: test_psid_research_sources.py's 244 documentation tests skipped because the PSID root was an empty directory; its other 147 tests open no PSID file",
"values_printed": "invented offsets, invented cells, invented-run counts and dispositions, hashes and timings; no 1946-55 value or direction",
"section_20_follow_up": "PR #544, cut from master f7a84d0b: the section 17, 19 and 20 record, the SSI changed-bytes test and the frame environment in the pre-registration evidence (schema v2); read only the specification ranges listed under 'specification' and the merged PR records #481, #486, #487, #500, #524, #525 and #540; ran the U1 differential at f7a84d0b and the U2 files under the hook (nothing refused); its reviews 20261016-140335-pr544-review and 20261016-152540-pr544-review-r2 (Claude Opus 5.5, standalone clone with the hook) read the specification and nothing else restricted; the second read its header, section headings, section 15's decisions and blocked_by, the section 17, 19 and 20 changes and section 20, and saw one grep line of the first appendix, which holds no value"
"section_20_follow_up": "PR #544, cut from master f7a84d0b: the section 17, 19 and 20 record, the SSI changed-bytes test and the frame environment in the pre-registration evidence (schema v2); read only the specification ranges listed under 'specification' and the merged PR records #481, #486, #487, #500, #524, #525 and #540; ran the U1 differential at f7a84d0b and the U2 files under the hook (nothing refused); its reviews 20261016-140335-pr544-review and 20261016-152540-pr544-review-r2 (Claude Opus 5.5, standalone clone with the hook) read the specification and nothing else restricted; the second read its header, section headings, section 15's decisions and blocked_by, the section 17, 19 and 20 changes and section 20, and saw one grep line of the first appendix, which holds no value",
"d1242_change": "after #544 merged (2ef10532): sources.PERSON_COUNTED_EXCLUSIONS gains B2 per Max's d1242, with its roles tests and a section 19 entry; read only the cos ledger's d1242, d1597 and d1649 entries, the code changed and the roles tests; ran the U2 files under the hook (nothing refused); answering #546's review, added an invented variant (a second, sex-unknown code-92 member in one excluded 2019 unit) and a distinct-person form of each count, reading the review, the invented-data generator, the runner and the scripts test; ran the reviewer's mutations and the U2 files under the hook (nothing refused)"
}
}
2 changes: 2 additions & 0 deletions docs/design/u2_m2_exposure.md
Original file line number Diff line number Diff line change
Expand Up @@ -350,4 +350,6 @@ The hook refused nothing at any of these runs. Its only refusals were the self-t

**The §20 follow-up (#544).** The same lane then cut #544 from master `f7a84d0b`: the §17, §19 and §20 record; the SSI changed-bytes test; and, after its review asked for it, the frame environment in the pre-registration evidence record. It read only the specification ranges listed above and the merged PR records (#481, #486, #487, #500, #524, #525, #540). It ran the U1 differential at `f7a84d0b` and the U2 files under the same hook, which refused nothing. Its independent reviews (`20261016-140335-pr544-review` and `20261016-152540-pr544-review-r2`, Claude Opus 5.5, in the standalone review clone with the hook) read the specification too, and nothing else restricted; the second read its header, section headings, §15's `decisions` and `blocked_by`, the §17, §19 and §20 changes and §20, and saw one grep line of the first appendix, which holds no value.

**The d1242 change.** After #544 merged (`2ef10532`), the same lane applied Max's d1242 ruling: B2 joins `sources.PERSON_COUNTED_EXCLUSIONS`, with its tests and a §19 entry. It read only the decision ledger's d1242, d1597 and d1649 entries, the code it changed and the roles tests. It ran the U2 files under the same hook, which refused nothing. Answering #546's review, it added an invented variant (a second, sex-unknown code-92 member in one excluded 2019 unit) and a distinct-person form of each count. For this it read the review, the invented-data generator, the runner and the scripts test. It ran the reviewer's mutations and the U2 files under the same hook, which refused nothing.

**Values:** this lane and its lanes stated and inferred no 1946–55 value or direction. Every number they printed is invented (offsets, invented cells, invented-run counts and dispositions), a count, a hash, a timing or a public parameter from the committed captures.
21 changes: 18 additions & 3 deletions src/populace_dynamics/uniform_cut_track_u2/cohort.py
Original file line number Diff line number Diff line change
Expand Up @@ -1540,9 +1540,9 @@ def build_u2_cohort(
"births_inputs_sha256": births.inputs_sha256,
# Section 16c: per wave, the members who brought each counted
# exclusion (sources.PERSON_COUNTED_EXCLUSIONS: B1's count, which
# option 1d-1 names and 1d-2 inherits) in family units that would
# otherwise have supplied this row an observation. Disclosed;
# decides nothing.
# option 1d-1 names and 1d-2 inherits, and B2's, which Max
# confirmed in d1242) in family units that would otherwise have
# supplied this row an observation. Disclosed; decides nothing.
"n_family_unit_exclusion_persons": {
name: {
str(wave): len(persons)
Expand All @@ -1551,6 +1551,21 @@ def build_u2_cohort(
}
for name in sources.PERSON_COUNTED_EXCLUSIONS
},
# The same members counted once across the waves: a person who
# brings an exclusion in two waves is one person, so the per-wave
# counts above sum to person-waves (d1242's count is persons).
"n_family_unit_exclusion_persons_distinct": {
name: len(
set().union(
*(
persons
for (named, _), persons in excluded_persons.items()
if named == name
)
)
)
for name in sources.PERSON_COUNTED_EXCLUSIONS
},
}
cohort = U2Cohort(obs, disp, spec, diagnostics)
object.__setattr__(
Expand Down
32 changes: 30 additions & 2 deletions src/populace_dynamics/uniform_cut_track_u2/invented.py
Original file line number Diff line number Diff line change
Expand Up @@ -1501,6 +1501,23 @@ def _code_88_in_excluded_unit_2019(base: cohort.U2Inputs) -> cohort.U2Inputs:
)


def _second_trigger_in_excluded_unit_2019(
base: cohort.U2Inputs,
) -> cohort.U2Inputs:
_, interview = _first_excluded_2019(base)
return _added_person(
base,
_PERSON_BASE - 4,
wave=2019,
interview=interview,
sequence=5,
relationship=92,
age=40,
weight=500.0,
sex="na",
)


def _zero_weight_target_in_excluded_unit_2015(
base: cohort.U2Inputs,
) -> cohort.U2Inputs:
Expand Down Expand Up @@ -1601,6 +1618,9 @@ def _code_88_target_with_ofum_income_2019(
"code_88_target_with_ofum_income_2019": (
_code_88_target_with_ofum_income_2019
),
"second_trigger_in_excluded_unit_2019": (
_second_trigger_in_excluded_unit_2019
),
}
_VARIANT_NOTES: dict[str, str] = {
"code_88_first_year_cohabitor": (
Expand Down Expand Up @@ -1661,8 +1681,9 @@ def _code_88_target_with_ofum_income_2019(
),
"sex_unknown_target_in_excluded_unit_2019": (
"the target of U0's first 2019 observation that the B2 exclusion "
"removes has unknown sex: it keeps its sex-unknown disposition "
"(section 16c precedence)"
"removes has unknown sex: it keeps its sex-unknown disposition, "
"and a unit that supplies no other observation adds nothing to "
"B2's person count (section 16c precedence; d1242)"
),
"spouse_slot_occupant_without_family_spouse_2019": (
"the family file of U0's first 2019 observation with a code-20 or "
Expand All @@ -1678,6 +1699,13 @@ def _code_88_target_with_ofum_income_2019(
"1c (the family income basis in U4 and the OFUM SSI cap), keeps "
"its recorded marital history and is no spouse-slot occupant"
),
"second_trigger_in_excluded_unit_2019": (
"a second code-92 member, aged 40 and of unknown recorded sex, "
"joins the 2019 unit of U0's first observation that the B2 "
"exclusion removes: the unit stays excluded and every observation "
"and disposition is unchanged, but B2's person count (d1242) rises "
"by one, since it counts persons, not family units, of any sex"
),
}
#: The generator's named variants (see :data:`_VARIANT_NOTES`).
INVENTED_VARIANTS: tuple[str, ...] = tuple(_VARIANT_BUILDERS)
Expand Down
16 changes: 10 additions & 6 deletions src/populace_dynamics/uniform_cut_track_u2/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,16 @@
UNCOOPERATIVE_SPOUSE_OR_PARTNER,
)
#: The exclusions whose person count the structural pass discloses
#: (section 16c): B1's, the count option 1d-1 names and 1d-2 inherits.
#: Option 1e-1 leaves B2's counting unit open; its person count is a
#: proposal pending Max's confirmation (decision d1242) and is not
#: computed until he confirms. B2's excluded observations appear in the
#: ordinary disposition counts either way.
PERSON_COUNTED_EXCLUSIONS: tuple[str, ...] = (MALE_CODE_20_2015,)
#: (section 16c): B1's, the count option 1d-1 names and 1d-2 inherits; and
#: B2's, the code-90 and code-92 persons in 2019-2023 family units that
#: would otherwise supply a U2 observation, as section 16c proposed and
#: Max confirmed on 2026-10-16 (decision d1242). Each count is disclosed
#: and decides nothing: per row, per wave (which sums to person-waves) and
#: as distinct persons across the waves.
PERSON_COUNTED_EXCLUSIONS: tuple[str, ...] = (
MALE_CODE_20_2015,
UNCOOPERATIVE_SPOUSE_OR_PARTNER,
)
#: The waves of each ratified exclusion (section 16c).
_B1_WAVES: tuple[int, ...] = (2015,)
_B2_WAVES: tuple[int, ...] = (2019, 2021, 2023)
Expand Down
117 changes: 86 additions & 31 deletions tests/track_u2/test_u2_roles.py
Original file line number Diff line number Diff line change
Expand Up @@ -732,11 +732,13 @@ def test_b2_excludes_whoever_the_target_is(u1_cohort, u2_inputs):
def test_exclusion_counts_are_the_ones_section_16c_names(
row, u2_inputs, declared, u2_births
):
"""Section 16c's disclosed person count, against an oracle from the
raw frames: per wave, the 2015 code-20 persons recorded male in family
units that would otherwise supply this row an observation (B1's count,
which option 1d-1 names and 1d-2 inherits). B2's person count is a
proposal pending Max (d1242) and is not computed."""
"""Section 16c's disclosed person counts, against an oracle from the
raw frames: per wave, the members who bring each exclusion in family
units that would otherwise supply this row an observation. B1's: the
2015 code-20 persons recorded male (the count option 1d-1 names and
1d-2 inherits). B2's: the code-90 and code-92 persons in 2019-2023
(section 16c's proposal, confirmed by Max in d1242). INVENTED DATA -
NOT A COMPARISON."""

built = cohort.build_u2_cohort(
u2_inputs,
Expand All @@ -745,18 +747,70 @@ def test_exclusion_counts_are_the_ones_section_16c_names(
births=u2_births,
)
stated = built.diagnostics["n_family_unit_exclusion_persons"]
assert sources.PERSON_COUNTED_EXCLUSIONS == (sources.MALE_CODE_20_2015,)
assert set(stated) == {sources.MALE_CODE_20_2015}
assert sources.PERSON_COUNTED_EXCLUSIONS == (
sources.MALE_CODE_20_2015,
sources.UNCOOPERATIVE_SPOUSE_OR_PARTNER,
)
assert set(stated) == set(sources.PERSON_COUNTED_EXCLUSIONS)
assert stated == _oracle_exclusion_persons(u2_inputs, built)
assert all(stated.values())
assert set(stated[sources.MALE_CODE_20_2015]) <= {
str(wave) for wave in B1_WAVES
}
assert set(stated[sources.UNCOOPERATIVE_SPOUSE_OR_PARTNER]) <= {
str(wave) for wave in B2_WAVES
}
distinct = built.diagnostics["n_family_unit_exclusion_persons_distinct"]
assert distinct == _oracle_distinct_exclusion_persons(u2_inputs, built)
for name in sources.PERSON_COUNTED_EXCLUSIONS:
assert 0 < distinct[name] <= sum(stated[name].values())
# d1242 counts persons, not family units, of any recorded sex: a
# second, sex-unknown code-92 member in one excluded 2019 unit leaves
# every observation and disposition as it was and adds exactly one.
changed = invented.invented_variant(
u2_inputs, "second_trigger_in_excluded_unit_2019"
)
b2 = sources.UNCOOPERATIVE_SPOUSE_OR_PARTNER
for context in (declared, sources.RoleContext.from_registry()):
base = cohort.build_u2_cohort(
u2_inputs, cohort.U2CohortSpec(row=row), role_context=context
)
more = cohort.build_u2_cohort(
changed, cohort.U2CohortSpec(row=row), role_context=context
)
pd.testing.assert_frame_equal(more.observations, base.observations)
pd.testing.assert_frame_equal(more.dispositions, base.dispositions)
before = base.diagnostics["n_family_unit_exclusion_persons"][b2]
after = more.diagnostics["n_family_unit_exclusion_persons"][b2]
assert after == {**before, "2019": before["2019"] + 1}, context.kind
assert after == _oracle_exclusion_persons(changed, more)[b2]
assert (
more.diagnostics["n_family_unit_exclusion_persons_distinct"][b2]
== base.diagnostics["n_family_unit_exclusion_persons_distinct"][b2]
+ 1
)
summary = cohort.structural_summary(built)
assert summary["n_family_unit_exclusion_persons"] == stated


def _oracle_exclusion_persons(inputs, built):
return {
name: {wave: len(persons) for wave, persons in waves.items()}
for name, waves in _oracle_exclusion_person_sets(inputs, built).items()
}


def _oracle_distinct_exclusion_persons(inputs, built):
return {
name: len(set().union(*waves.values()))
for name, waves in _oracle_exclusion_person_sets(inputs, built).items()
}


def _oracle_exclusion_person_sets(inputs, built):
sex = inputs.persons.set_index("person_id")["sex"]
disp = built.dispositions
expected: dict[str, dict[str, int]] = {
expected: dict[str, dict[str, set[int]]] = {
name: {} for name in sources.PERSON_COUNTED_EXCLUSIONS
}
for (name, wave), rows in disp[
Expand All @@ -770,16 +824,14 @@ def _oracle_exclusion_persons(inputs, built):
anchor["interview"].isin(interviews)
& anchor["sequence"].between(1, 20)
]
expected[name][str(wave)] = len(
{
int(m.person_id)
for m in members.itertuples(index=False)
if expected_exclusion(
wave, int(m.relationship), str(sex.get(m.person_id))
)
== name
}
)
expected[name][str(wave)] = {
int(m.person_id)
for m in members.itertuples(index=False)
if expected_exclusion(
wave, int(m.relationship), str(sex.get(m.person_id))
)
== name
}
return expected


Expand All @@ -805,8 +857,8 @@ def test_earlier_dispositions_keep_precedence(
):
"""Section 16c: an observation already disposed of keeps that
disposition, and a unit that supplies no other observation adds
nothing to B1's person count; under both role contexts (review of
07916bb, finding 2)."""
nothing to its exclusion's person count (B1's, and B2's since d1242);
under both role contexts (review of 07916bb, finding 2)."""

changed = invented.invented_variant(u2_inputs, variant)
for context in (
Expand All @@ -824,14 +876,13 @@ def test_earlier_dispositions_keep_precedence(
persons_before = base.diagnostics["n_family_unit_exclusion_persons"]
persons_after = built.diagnostics["n_family_unit_exclusion_persons"]
assert persons_after == _oracle_exclusion_persons(changed, built)
if rule == sources.MALE_CODE_20_2015:
# The unit's male code-20 member no longer counts.
assert (
persons_after[rule][str(wave)]
== persons_before[rule][str(wave)] - 1
)
else:
assert persons_after == persons_before
# The unit's member who brings the exclusion (B1's male code 20;
# B2's code 90 or 92 since d1242) no longer counts.
assert rule in sources.PERSON_COUNTED_EXCLUSIONS
assert (
persons_after[rule][str(wave)]
== persons_before[rule][str(wave)] - 1
)
pd.testing.assert_frame_equal(built.observations, base.observations)


Expand Down Expand Up @@ -874,9 +925,13 @@ def test_structure_and_runner_disclose_each_exclusion(u1_cohort, u2_run):
for row_id, entry in u2_run["rows"].items():
counts = entry["population"]["dispositions"]
assert set(sources.FAMILY_UNIT_EXCLUSIONS) <= set(counts), row_id
assert set(
entry["population"]["n_family_unit_exclusion_persons"]
) == set(sources.PERSON_COUNTED_EXCLUSIONS), row_id
for key in (
"n_family_unit_exclusion_persons",
"n_family_unit_exclusion_persons_distinct",
):
assert set(entry["population"][key]) == set(
sources.PERSON_COUNTED_EXCLUSIONS
), (row_id, key)


def test_cohabitor_22_occupies_the_slot_without_legal_status(u1_cohort):
Expand Down
1 change: 1 addition & 0 deletions tests/track_u2/test_u2_scripts.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ def test_dry_run_records_every_branch_and_refusal(dry_run):
"sex_unknown_target_in_excluded_unit_2019",
"spouse_slot_occupant_without_family_spouse_2019",
"code_88_target_with_ofum_income_2019",
"second_trigger_in_excluded_unit_2019",
]
assert all(entry["regenerated"] for entry in variants.values())
# Amendment 1c: the two code-88 variants build under the declared
Expand Down
Loading