Skip to content

SM1: differential mortality by lifetime earnings, education and race (normalized within age-sex-year cells) - #570

Merged
MaxGhenis merged 18 commits into
masterfrom
mgf-sm1-differential-mortality-20261024
Oct 25, 2026
Merged

MaxGhenis merged 18 commits into
masterfrom
mgf-sm1-differential-mortality-20261024

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

SM1 of the mortality, groups and financing lane: an opt-in mechanism for mortality that differs by lifetime earnings, education and race and Hispanic origin, normalized so each age × sex × year cell keeps the baseline's expected deaths. It fits nothing and is wired into nothing.

The lane design is docs/design/mortality_groups_financing.md (#558), which Max approved on 2026-10-23 (d1922) at 2c19f5b6. Max's rulings that apply here:

  • relative risks are normalized within each cell (Q4);
  • lifetime earnings is the primary dimension, with education and race and Hispanic origin as further dimensions (Q5);
  • the gradient holds at its last observed level, with widening as a registered alternative path (Q6).

The brief is EV/mortality-groups-financing-20261021/builds/sm1-brief-20261024.md, kept outside the repository.

What it adds

  • The package src/populace_dynamics/mortality_ses/, with four modules:
    • specification.py: log relative risks by dimension, category, sex and single age, combined log-additively. Coefficients are pluggable by year. The default path holds the last year's table. Reference categories must have zero log risk. Unclassified values refuse by default, or count as the reference under reference; unknown categories always refuse.
    • normalizer.py: a pure normalizer. Fixed rows keep their fixed probability. Free rows get min(1, q_base × RR × k), with iterative capping, so the cell's weighted deaths equal the baseline's. It writes a per-cell log: deaths, factor, clipped persons, residual, the infeasible flag, and the normalization basis with its persons target.
      • Weightless cells. A feasible cell whose free rows with positive baseline q all weigh zero sets its factor with unit weights, since every factor conserves its weighted deaths. The log marks it with normalization_basis = "persons".
      • Decimal path. A float64 path handles ordinary cells. A Decimal path handles cells with extreme magnitudes, and every comparison or cancellation the float path cannot decide reliably. That includes any row whose scaled probability lies within 1e-9 of one, since whether it caps decides every other row's factor. After the last capping round it also includes any uncapped zero-weight row that close to one: such a row doesn't set the factor, but whether it clips sets the log's clipped count.
    • model.py: DifferentialMortality wraps a year-aware baseline. It exposes bands = SINGLE_YEAR_AGE_BANDS and __call__(frame, context). It deliberately has no probabilities attribute, so the DI-aware adapter calls it with the year. It refuses an age that disagrees with year − 1 − birth_year. DI-origin persons, as the adapter defines them, keep baseline q, so the adapter still owns their rates. It consumes no RNG. An optional cell_log mapping keeps each call's cell log by year, because the adapter keeps only probabilities.
    • step.py: apply_differential_mortality draws the same uniforms as engine/steps.py's apply_mortality, person-keyed through the RNG registry when one is present.
  • The design note: docs/design/mortality_ses_sm1.md.
  • Housekeeping: the five sources join POST_REVIEW_SOURCE_EXCLUSIONS, its exact-tuple test, and a check that they stay unreachable from the birth-evidence reducer.

Nothing touches a registered run, runner, config, gates.yaml, engine/*, baselines/* or track_b/. SM mortality enters a projection only after SM3 passes and the change-admission protocol is frozen (lane design, section 7.1).

Invariants

Each holds for every valid input, within the stated tolerance. All are tested with Hypothesis properties on invented data, plus example tests.

  • Cell conservation: in every feasible cell, applied weighted deaths equal baseline weighted deaths within rtol = atol = 1e-12, and the logged residual is exactly zero. An infeasible cell logs its signed gap (baseline minus applied) as the residual. The float path checks conservation itself rather than assuming it, and recomputes on the Decimal path any cell that misses.
  • Float accuracy: outside the fallback margins, the float path's factor and every probability are within about 1.3e-12, relative, of the exact result. The design note derives the bound. Every cell inside a margin takes the Decimal path, so every cap decision and every clipped count is the exact one.
  • Feasibility: with fixed rows at baseline q, no cell is infeasible. When fixed probabilities exceed the target, the flag fires.
  • Identity: with every relative risk 1, probabilities equal the baseline's bit for bit. Deaths are identical with and without the RNG registry, both with and without the DI adapter.
  • Bounds: every probability lies in [0, 1].
  • DI composition:
    • DI-origin persons keep baseline q;
    • a cell's applied deaths equal its baseline deaths minus two separate gaps: SM1's residual, which is zero in every feasible cell, and the adapter's own clipping gap (its population-model deaths minus its applied deaths). So the composition conserves only where the adapter's one-pass clip does not bind;
    • the ratio of two non-DI-origin persons' probabilities is unchanged where neither is clipped.
  • Monotonicity (intended): probabilities follow the order of the relative risks by earnings group or education, within a cell, among non-fixed rows with a common baseline q.
  • Invariance:
    • permuting rows, or scaling every weight by a positive constant, changes no probability;
    • a cell whose free rows share one risk gets the baseline probability.
  • Determinism: repeated calls give identical output, and no generator state changes.
  • Weightless cells. Take a cell where every free row with positive baseline q has zero weight and the weighted target is zero.
    • The cell counts persons equally to set the factor, and logs basis persons and the unweighted target.
    • With fixed rows at baseline, its probabilities and factor equal exactly those it gets with every weight 1. Zero weight never makes such a person immortal.
    • If the weighted target is not zero (fixed probabilities off baseline), the cell is infeasible and follows the weighted rules: a negative target zeroes the free rows, and a positive one saturates the support. Those rules are the limit as the support's weight vanishes, and a property checks the limit.
  • A target that exactly fills the weighted support takes the smallest factor that caps every weighted row. That is the capping loop's limit as the target rises to capacity, so zero-weight rows are not pushed to one.
  • Exact oracle: the normalizer matches an independent exact implementation's probabilities to rtol = 1e-11, atol = 1e-15, about seven times the float error bound, and its clipped count, infeasible flag and normalization basis exactly. The oracle uses rational arithmetic and searches over how many rows cap rather than iterating. Four generators feed it:
    • cells across float magnitudes, with near-one q, cascading caps, ties and arbitrary fixed probabilities;
    • cells whose dominant row sits at the cap, a few units of rounding either way, beside rows of tiny weight;
    • cells of ordinary magnitude whose fixed deaths cancel all but a share near the margin;
    • cells whose zero-weight rows sit a few rounding steps either side of one, sometimes after a round that caps a dominant row.

Differential test: with unit risks for the free rows, the normalizer reproduces engine/di_entitlement.py's _net_of_di_origin on every cell where no clip binds, including cells with zero weights, infeasible cells and arbitrary fixed probabilities. Every such cell, apart from the weightless infeasible ones below, is also held to the adapter's formula evaluated in exact rational arithmetic, at the oracle's tolerance. They differ by design in three places, and the test labels each as intended:

  • where a clip binds, SM1 caps iteratively and conserves, while the adapter clips without redistributing;
  • in an infeasible cell whose free support weighs nothing, the adapter leaves the factor at one, while SM1 applies the infeasible-cell rules above;
  • where the adapter's max(P - D, 0) / E is unreliable in float64, its factor can cancel, absorb the free deaths, overflow or be off by percents. That is where P - D or the free deaths E lies within a hundredth of max(P, D), or where max(P, D) is below 2^-1000, so that float64 can round the cell's products absolutely (a product below 2^-1022 is subnormal). There the test compares SM1 with the exact formula only. Elsewhere a first-order bound, given in the design note, keeps the adapter within the test's tolerance for every probability up to one, and where P = D exactly its factor stays under the absolute tolerance.

Checks (run on the host at 8ca39c26, with an empty PSID root; e1155b4b and 5bda0fed change only the design note and a docstring, and tests/mortality_ses, Black and Ruff were rerun at each; 3078d18b and 3384364a change the differential test, and the checks marked † were rerun at each; cd12003a changes only the design note and a docstring, and tests/mortality_ses, Black and Ruff were rerun there; 75df561e merges master, and the checks under "On the merged tree" were run there)

  • tests/mortality_ses: 116 passed, all in the unit tier †
  • tests/test_m6_engine_steps.py and tests/test_m6_engine_di_entitlement.py: 83 passed †
  • tests/baselines -m unit: 29 passed
  • Birth-evidence exclusion and reachability (-k post_review): 2 passed
  • Full collection, -k tiers_match_policy_manifest: 1 passed, 2 skipped †
  • Black (79 columns) and Ruff over the repository: clean; git diff --check clean †
  • The adapter differential test, stressed outside the suite at 3384364a †, with a runner that counts every failure, Hypothesis's ExceptionGroup included:
    • 100,000 fresh draws of its generator pass through the test's own body;
    • in 1,000 fresh 35-draw runs, the size CI runs (seeds 1000–1999), 70 fail at 5bda0fed; at 3384364a, none of 3,000 (seeds 1000–3999);
    • judged by the test's own guard, every failing cell of 100,000 draws (512 of 257,220 compared cells) is guarded, 2.56% of cells are guarded in all, and the worst float64 comparison kept uses 0.3% of its tolerance;
    • the three new subnormal cells fail at 3078d18b's guard (its float64 adapter misses the tolerance by a factor of 6e9 to 3e11) and are guarded now, and SM1 equals the exact formula in each;
    • multiplying SM1's free-row probabilities by 1 + 5e-11 fails six of the seven examples through the exact check alone. The seventh, with equal subnormal P and D, gives its free rows probability zero, which a relative change can't move. 2e-12, below the exact check's tolerance, passes all seven.
  • Searches against the exact oracle, run outside the suite at rtol = 1e-11, comparing clipped counts exactly. They are derandomized: each draws the same cells on every run and for both heads, so these figures reproduce. Hypothesis also draws some values from the literals of loaded local modules that aren't tests, so a seeded figure reproduces only with the same source for the runner script and for src/.
    • At 8ca39c26, no disagreement in 8,000 cells drawn across magnitudes, 6,000 at cap boundaries, 10,000 near the margin and 4,000 with zero-weight rows at one.
    • At ad827f1e, the same 4,000 zero-weight cells give 1,174 miscounted clips. The other three generators' cells give identical results at both heads: the fix changes a cell only when an uncapped zero-weight row lies within 1e-9 of one, which the boundary and margin generators never draw and none of these 8,000 wide cells contains.
    • The largest float-path error was 1.8e-13 relative, near the margin, at both heads. Earlier unseeded searches of that generator gave 1.4e-13 to 2.4e-13, about a fifth of the bound at most.
    • In round 2's searches (unseeded), at fda767c2, the cap-boundary generator found 189 disagreements in 2,000 cells, the largest a relative error of nearly one.
    • 8edc9d90's message quotes unseeded counts (605 of 2,000 at ad827f1e); the derandomized figures above supersede them.
  • An earlier search, before fda767c2, found no disagreement in 7,000 cells.
  • An earlier ad hoc search, of 20,000 invented trials (53,481 unclipped cells) against _net_of_di_origin, found 1,207 mismatches before 8f341b0f, all in cells whose positive-q free rows weigh zero, and none after
  • Timing, on a million invented rows in 242 cells, on a loaded host, at 8ca39c26:
    • the normalizer takes 0.49 s with common risks, 0.63 s with heterogeneous risks and 0.73 s under heavy capping, with no Decimal fallback in any of them;
    • in a frame whose baseline q follows a Gompertz curve to one, the 42 oldest-age cells fill their capacity exactly and take the Decimal path; the normalizer takes 3.3 s;
    • fallback and clipped counts are the same as at fda767c2 in all four frames. These frames have no zero weights; the delta review's variant with 10% zero weights also left fallback and clipped counts unchanged;
    • the complete model's evaluate takes 1.53 s.
    • Round 3's reviewer measured 26–53 s for the same runs inside its build-lane sandbox, which runs much slower than the host; the fallback patterns matched.
  • The builder also ran:
    • the four brief-listed test files, tests/test_m6_engine_steps.py and tests/baselines/ -m unit (122 passed);
    • Black, Ruff and git diff --check.
  • On the merged tree at 75df561e, which merges master 1ec57a5d, with an empty PSID root:
    • full collection, -k tiers_match_policy_manifest: 1 passed, 4 skipped;
    • birth-evidence exclusion and reachability (-k post_review): 2 passed;
    • tests/mortality_ses, with RuntimeWarning as an error: 116 passed;
    • tests/test_m6_engine_steps.py and tests/test_m6_engine_di_entitlement.py: 83 passed;
    • tests/baselines -m unit: 29 passed in one run. Master's test_ungrouping_is_homogeneous_of_degree_one (tests/baselines/test_interpolation.py) fails in about a fifth of fresh Hypothesis runs, independent of this PR: 13 of 60 in the delta check, 7 of 30 in the lane's rerun. Its test file and module are byte-identical to master, and no mortality_ses module loads while it runs. A subnormal band total makes the negativity tolerance (1e-12 times the largest total) underflow to zero, so a rounding-level negative raises. If a CI shard fails on it, rerun that shard;
    • Black (79 columns), Ruff and git diff --check over the changes: clean.

Tiers (recount-tiers.py):

Tier Master 1ec57a5d This PR at 75df561e
unit 18,433 18,549
Total 25,003 25,119

The other tiers are unchanged. The +116 equals SM1's own delta on its original base (unit 18,088 to 18,204, total 24,619 to 24,735): tests/mortality_ses adds 116 unit tests. This PR edits tests/tier_counts.json, so its merge waits for the coordinator's tier-count slot.

The master merge (75df561e) resolves four conflicts:

  • scripts/first_estimates_birth_evidence.py and its exact-tuple test in tests/estimates/test_birth_evidence_artifact.py: both sides added entries to POST_REVIEW_SOURCE_EXCLUSIONS at the same place. The merge keeps both, SM1's five mortality_ses paths and then master's two D1a reader paths, in the same order in the script and the test;
  • tests/tier_counts.json and tests/README-tiers.md: master's versions, recounted on the merged tree.

Against master, the merged tree differs only in SM1's 14 files.

Provenance

  • Built by a Subfleet build lane (GPT-6.1 Sol, job 20261024-073814-mgf-sm1-build). That lane's sandbox could not write .git, so the lane session reran the checks above and committed the work in three steps.
  • Written by the lane session (Claude Opus 5.5):
    • the weightless-cell fix (8f341b0f) and its tier counts (6801a686);
    • the round-1b2 fixes (b99300cc) and their tier counts (fda767c2);
    • the round-2 fixes (bf49d608), their tier counts (cee73cb7) and a design-note figure (ad827f1e);
    • the round-3 fix (8edc9d90) and its tier counts (8ca39c26);
    • the delta review's documentation follow-up (e1155b4b), the confirmation's wording follow-up (5bda0fed), the second confirmation's differential-test fix (3078d18b), the third confirmation's guard fix (3384364a), and the fourth confirmation's wording follow-up (cd12003a);
    • the merge of master 1ec57a5d (75df561e).
  • Data: no PSID or other real-data file was opened, and no restricted file was read.
  • Not in SM1:
    • empirical coefficients, category definitions and lifetime-earnings construction (SM2);
    • the registered gradient gate and death alignment (SM3).

Review

  • Round 1 (job 20261024-105442-mgf-sm1-review-r1, Opus 5.5) ran for 2 h 41 min and was lost to an API error before it reported. Its progress notes found the weightless-cell defect, now fixed in 8f341b0f; they also confirmed that tests/mortality_ses, the DI adapter tests, the baselines unit tests and the post_review checks passed at 0c226e3f.

  • Round 1b2 (job 20261024-142615-mgf-sm1-review-r1b2, GPT-6.1 Sol) reviewed 6801a686 and returned REVISE, with one high, three medium and two low findings:

    • high: float roundoff could zero a capped cell;
    • medium: a rounded target-capacity tie saturated rows;
    • medium: the unit-weight rule overrode a negative target;
    • medium: the log did not show the unit-weight basis;
    • low: an overclaimed equivalence;
    • low: per-row Python work.

    b99300cc addresses all six. The review judged the unit-weight convention right for feasible weightless cells, with explicit logging, and that the negative-target rule should be kept separately. Under the coordinator's ruling, that judgment makes the convention the lane's call.

  • Round 2 (job 20261024-170311-mgf-sm1-review-r2, GPT-6.1 Sol) reviewed fda767c2 and returned REVISE, with one medium and two low findings. It confirmed all six round-1b2 fixes, found that the fallback cannot recurse, and found that the exact oracle is independent and correct.

    • medium: a scaled probability rounded to exactly one capped nothing, so rows of tiny weight took a factor ten times too small while the totals conserved;
    • low: the stated error bound ignored fixed-death and capping cancellation compounding (up to 1.85e-10 in the review's search);
    • low: this body claimed the DI composition conserves after the adapter clips.

    bf49d608 addresses all three, and this body is corrected. On the conventions, the review's judgment was to accept both the weightless-cell rule split and the smallest-factor tie rule.

  • Round 3 (job 20261024-180937-mgf-sm1-review-r3, GPT-6.1 Sol) reviewed ad827f1e and returned APPROVE, with no high or medium finding. It confirmed all three round-2 fixes, the error bound to first order (including 80-bit longdouble), the 1e-9 cap margin and the oracle's coverage.

    • low: a zero-weight row whose exact scaled probability lay just above one could go uncounted in clipped_persons, so "every cap decision is exact" was too broad. Probabilities stayed within the bound.

    8edc9d90 fixes it as the review proposed and adds a regression test, an exact clipped-count check in every oracle property, and a fourth generator for the case.

  • Delta review of ad827f1e..8ca39c26, by an in-session Claude Opus 5.5 subagent (every Subfleet lane was closed), returned APPROVE, with no high or medium finding. It probed every branch of the new check and confirmed that the Decimal path counts clips as the oracle does. It also found the generator valid and reaching both sides of one, and reproduced the tests, tiers and lint. It had two low findings and a nit, all wording:

    • low: clipped_persons was compared exactly but never defined;
    • low: the search figures came from unseeded runs, set side by side as if one improved on the other;
    • nit: a stray line break in the design note.

    e1155b4b defines the count in the design note and the docstring, rewraps the paragraph, and restates the note's figure from a derandomized search. This body's figures come from derandomized searches too. Its commit message names the reviewed range as 8edc9d90..8ca39c26; the range was ad827f1e..8ca39c26.

  • Confirmation of e1155b4b by the same reviewer returned APPROVE. It found every seeded search figure reproduced exactly at both heads, and the definition matching the code in every branch. It had two low findings, both wording:

    • low: this body said the other generators agree at both heads "since the fix changes no float-path arithmetic"; the fix can change a wide-generator cell, and these 8,000 draws simply contain none. The 189 figure also came from an unseeded search;
    • low: the definition holds at the cell's exact final factor, not at the logged one, which approximates it in float64.

    5bda0fed qualifies the definition in the design note and the docstring, and this body uses the reviewer's wording for the first.

  • Confirmation of 5bda0fed by the same reviewer returned REVISE, with one medium and one low finding:

    • medium: the differential test against the adapter was flaky, and this body and the design note described the adapter as reproduced on every unclipped cell. The cause is the adapter's float64 arithmetic: in every failing cell SM1 equals the adapter's formula evaluated exactly. The reviewer measured 17 of 200 failing runs. The lane's figure, 4 of 1,000, was wrong (see the next confirmation);
    • low: the logged factor equals the exact factor's float64 rounding only on the Decimal path; on the float path it approximates it within the error bound.

    3078d18b holds every unclipped cell to the exact formula, compares the float64 adapter only where its arithmetic is well-conditioned, and names that as a third intended divergence, with four examples. It also corrects the wording. The reviewer proposed skipping those cells; the exact comparison keeps them tested.

  • Confirmation of 3078d18b by the same reviewer returned REVISE. It found the exact helper equal to the adapter's formula, the new check not vacuous, and every test and lint check passing. It had one medium and two low findings:

    • medium: the figure "4 of 1,000 at 5bda0fed", in this body and in 3078d18b's commit message, was a measurement error by the old runner, which caught only AssertionError and stopped at the first ExceptionGroup, after 46 runs. The reviewer measured 70 of 1,000, and the lane's corrected runner 70 of 1,000. The figures above are the corrected ones. 3384364a's commit message records the correction and supersedes 3078d18b's, which is not rewritten;
    • low: the guard missed cells whose sums are all below the normal float64 range, where products round absolutely, not relatively. The reviewer minimized three such cells;
    • low (optional): at a thousandth, the float64 comparison held only empirically, with a worst-case bound that could exceed the tolerance for probabilities over 0.41.

    3384364a raises the threshold to a hundredth, where the first-order bound covers every probability up to one. It also adds a clause for subnormal sums, the three cells as examples, and the silencing of the 0 * inf warning, and narrows two "every unclipped cell" claims.

  • Confirmation of 3384364a by the same reviewer returned APPROVE, with no high, medium or low finding:

    • it reran every figure in the commit message and this body, and each matched;
    • it checked the subnormal clause's sufficiency and the first-order bound's derivation;
    • it found bincount bit-identical to an index-order sum from zero in 20,000 invented trials (the bound doesn't depend on summation order).

    It had two wording nits: the subnormal clause overstated which products round absolutely, and the bound didn't cover cells with P = D exactly. It also traced the seed difference between the two margin scripts to Hypothesis's local-constant pool (internal/constants_ast.py, internal/conjecture/providers.py, 6.170.9), which the lane checked in that source. cd12003a fixes both nits, and this body notes the reproducibility condition.

  • Confirmation of cd12003a by the same reviewer returned APPROVE, with no high, medium or low finding.

    • Both nits are fixed, and the diff changes no code: the test file's syntax tree, without docstrings, is identical to 3384364a's.
    • The literal pool Hypothesis collects from normalizer.py is the same at ad827f1e, 8ca39c26 and cd12003a, so the seeded figures hold across those heads; it reran two of them at cd12003a to the same values.
    • Its two optional nits, on this body's wording, are applied.
  • Delta check of the master merge (75df561e) by the same reviewer returned APPROVE, with no high or medium finding.

    • It found the merge clean: the tree differs from master only in SM1's 14 files, and the 10 that didn't conflict are blob-identical to cd12003a.
    • It found the union right. It parsed POST_REVIEW_SOURCE_EXCLUSIONS at the base, master, SM1 and the merge: 185 entries plus master's 4 and SM1's 5 give the merge's 194, with no duplicates, both orders kept, and the script equal to the test's tuple.
    • The recount reproduces byte for byte, and it reproduced the merged-tree checks.
    • low: the baselines line above came from a single run, and a master property test is flaky (see above). This body now says so.

axiom: n/a: projection mechanism with invented-data tests only; no statutory rule is encoded

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits October 24, 2026 10:45
Adds the opt-in package populace_dynamics.mortality_ses.

- Relative risks are specified over three dimensions: lifetime-earnings
  group (primary), education, and race and Hispanic origin. They are
  log-additive and indexed by sex, age and year. Years past the table
  hold the last year's coefficients. A coefficient path is pluggable.
- The unclassified rule is `refuse` by default; `reference` contributes
  zero log risk.
- A pure normalizer rescales probabilities so that each single-age x
  sex x year cell keeps the baseline's expected deaths. Fixed rows keep
  their fixed probability, and the others get min(1, q_base x RR x k),
  with iterative capping. It logs the residual and flags infeasible
  cells.
- DifferentialMortality wraps a year-aware baseline. Its fixed rows are
  DI-origin persons at baseline q, so the DI-aware adapter sees the
  baseline rate for them. It exposes bands and __call__(frame, context),
  and deliberately no probabilities attribute.
- apply_differential_mortality draws apply_mortality's uniforms.

Nothing is wired into a registered run, runner, config or gates.yaml.

Tests are on invented data: Hypothesis properties for cell
conservation, feasibility, bit-identity at unit risk, bounds, DI
composition, intended monotonicity, permutation and weight invariance,
and determinism; plus a differential test against the DI adapter's
_net_of_di_origin arithmetic.

Built by a Subfleet build lane (GPT-6.1 Sol) from the lane brief; that
lane's sandbox could not write .git, so the lane session commits it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documents the interfaces, the cells and fixed persons, the unclassified rule, normalization and its tolerance, traceability to the lane design (docs/design/mortality_groups_financing.md, approved at 2c19f5b under d1922), the invariant evidence, and what stays outside SM1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the five mortality_ses sources to POST_REVIEW_SOURCE_EXCLUSIONS and its exact-tuple test, and asserts that they stay unreachable from the birth-evidence reducer. It also recounts the tiers: unit 18,088 -> 18,178, total 24,619 -> 24,709, from recount-tiers.py with the empty PSID root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
social-security-model Ready Ready Preview Oct 25, 2026 3:09pm UTC

Request Review

MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…known gaps

- Tracking and next-step cells for the twelve rows the mortality,
  groups and financing (#558), income-and-resources (#559) and
  open-population (#557) designs cover, citing their merged files.
- The poverty row records U2's completed second structural pass and
  its ratification by #556's merge; next are the evidence file, a
  fresh forecast and the #42 registration.
- Known gaps keeps only what no issue, PR or approved design plans.
- Rulings d1922, d1923 and d1925; link refs for #556, #566–#568, #570.
- Scorecard: the wealth row names the approved design; progress page
  regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 24, 2026 14:17
When every free row with positive baseline q in a cell had zero weight,
the weighted target was zero and the normalizer set the factor to zero,
so those persons could never die. An invented cell with baseline 0.2 and
relative risks 1 and 2 returned q = [0, 0], logged factor 0 and raised no
flag; through the model, an all-zero-weight frame did the same.

No factor changes such a cell's weighted deaths, so the weighted target
says nothing about it. The cell now sets the factor as if each person had
weight one: its free rows keep their relative risks, and their unweighted
expected deaths equal their unweighted baseline total. Both arithmetic
paths (float and Decimal) apply the rule. If fixed probabilities off
baseline leave a nonzero weighted target, the cell stays infeasible and
the residual records the gap. With unit risks this reproduces the DI
adapter, which keeps the factor at one when the other rows carry no
expected deaths.

Tests:
- examples on both arithmetic paths, mixed cells, infeasible cells of
  both signs, and the model;
- a property that zero weights give exactly the unit-weight
  probabilities and factors;
- a differential property against _net_of_di_origin on every unclipped
  cell, over zero weights and arbitrary fixed probabilities;
- the normalizer properties now draw zero weights.

An ad hoc search of 20,000 invented trials (53,481 unclipped cells)
found 1,207 mismatches with the adapter before this change and none
after.

Found by the SM1 review r1, whose attempt was lost to an API error
before it reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/mortality_ses collects 96 unit tests, up from 90, and no others.
unit 18,178 -> 18,184; total 24,709 -> 24,715. The full-collection
tiers_match_policy_manifest check passes with the empty PSID root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 24, 2026 16:54
Review round 1b2 found two float-path faults. Roundoff in the capping
loop could zero a capped cell's probabilities and hide the loss, and a
rounded tie between target and capacity saturated rows it should not.
The float path now hands the Decimal path every call it cannot make
reliably: a target within 1e-3 of capacity or of zero, a capping round
that leaves less than 1e-3 of the target, and a feasible result that
misses conservation by more than 1e-12. Capped rows stay at one.

Infeasibility is now decided before the unit-weight rule. A negative
weighted target zeroes the free rows and a positive one saturates the
support, whatever their weights; these are the limits as the support's
weight vanishes. Unit weights apply only where the weighted target is
zero. A target that exactly fills the weighted support takes the
smallest capping factor, so zero-weight rows are not pushed to one.

The cell log gains normalization_basis and persons_target.
DifferentialMortality takes an optional cell_log mapping, so a run under
the DI adapter keeps that log. Grouping checks missing values once per
distinct cell, and the stable sum converts its terms in bulk.

The tests add an exact rational oracle, a vanishing-weight limit
property, a regression for each finding, and a composed DI run with
unequal weights and clipping in both steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/mortality_ses now collects 110 unit tests, up from 96.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 24, 2026 18:06
Round 2 of the review found a float-path cell whose largest row's scaled
probability rounded to exactly one. The loop capped nothing, so rows of
tiny weight took a factor ten times too small while the weighted totals
still conserved.

- A row within 1e-9 of one at any capping round now sends the cell to
  the Decimal path. The float path's error bound is about 1.3e-12, so
  every other cap decision is the exact one.
- The remaining-target guard is now measured against the totals the
  target is taken from, not the target. Fixed-death and capping
  cancellation can no longer compound; they once reached an error of
  1.8e-11.
- The comment and the design note state the error bound and its
  derivation. The separate 1e-12 conservation check covers weighted
  deaths only.
- Tests:
  - two new unit regressions;
  - two new oracle properties, one for cap boundaries beside tiny
    weights and one for cancellation near the margin;
  - the oracle tolerance tightens from 1e-9 to 1e-11.
- The composition test now checks that applied deaths equal baseline
  deaths minus SM1's residual and the adapter's own clipping gap. It
  also checks that the gap is positive where the adapter clips.

On invented million-row frames, the normalizer's time and fallback
counts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit 18,198 to 18,202; total 24,729 to 24,733. These counts are against
the PR's base and will be recounted on fresh master at the tier-count
slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The design note now gives the largest float-path error from 10,000
invented near-margin cells at cee73cb: 2.4e-13 relative, under the
1.3e-12 bound. The earlier figure came from a search run before the
cap-decision margin was set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 24, 2026 21:18
…he oracle

Round 3 found that the float path's cap-decision check examined only the
rows that set the factor. A zero-weight row takes the factor without
setting it, so one whose exact scaled probability lay a rounding step
above one could go uncounted in the log's clipped count, though its
probability stayed within the error bound.

After the last capping round, any uncapped free row within the 1e-9 cap
margin of one now sends the cell to the Decimal path. The exact oracle
now returns the clipped count, and the oracle check compares it exactly,
with the reviewer's cell as an example and a new generator that places
zero-weight rows a few rounding steps either side of one, sometimes
after a round that caps a dominant row. At ad827f1 that generator finds
605 miscounted cells in 2,000; here it finds none in 4,000, and the
other three generators find no disagreement in 24,000 cells. On a
million invented rows the fallback and clip counts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit 18,202 to 18,204; total 24,733 to 24,735. These counts are against
the PR's base and will be recounted on fresh master at the tier-count
slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review of 8edc9d9..8ca39c2 approved it, with two wording
findings and a nit. The design note and the normalizer's docstring now
define `clipped_persons`: free rows whose scaled probability exceeds
one at the cell's final factor, not counting a row exactly at one; in a
cell whose target exceeds its free capacity, every free row with
positive baseline q. The note's measured error now comes from a
derandomized search (1.8e-13 relative, near the margin), with the
unseeded range beside it, and a paragraph is rewrapped.

The search figures in 8edc9d9's message came from unseeded runs. In
derandomized runs, ad827f1 miscounts 1,174 of the same 4,000
zero-weight cells this head gets right, and the other three generators
give identical results at both heads. No code path changes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The confirmation of e1155b4 found that the definition holds only at the
cell's exact final factor. The logged factor is its float64 rounding, so
a row exactly at one, such as a q = 1 row at a terminal age, can appear
to exceed one when recomputed from the log. The design note and the
normalize_probabilities docstring now say so. No code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The confirmation of 5bda0fe found the differential test against the DI
adapter's _net_of_di_origin flaky. On 1,000 fresh CI-sized runs at
5bda0fe, 4 failed. In every failing cell SM1 equals the adapter's
formula evaluated exactly; the adapter forms max(P - D, 0) / E in
float64 with no guard, so its factor can cancel, absorb the free deaths
or overflow.

The test now holds every unclipped cell to the adapter's formula in
rational arithmetic (exact_adapter_cell), at the oracle's tolerance.
It compares the adapter's float64 result only where neither P - D nor
E lies within a thousandth of max(P, D), with the sums taken exactly so
a subnormal E is not lost. That is a third named intended divergence.
Four examples cover overflow, absorption, subnormal absorption beside a
weightless row, and cancellation. The adapter's intended overflow
warning is silenced around its call.

Over 140,000 fresh draws the test passes; at 1e-3 the worst float64
comparison it keeps uses under 3% of its tolerance. A relative error of
5e-11 in SM1 fails all four examples through the exact check alone.

The clipped-count definition now says the logged factor approximates
the exact one in float64, within the float path's error bound, as the
same review proposed. No change to SM1's code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The confirmation of 3078d18 found that the adapter guard missed cells
whose sums are all subnormal, where float64 rounds products absolutely,
and that at a thousandth the float64 comparison held only empirically.

- exact_adapter_cell flags a cell when P - D or E lies within a
  hundredth of max(P, D), or when max(P, D) is below 2**-1000. At a
  hundredth, a first-order bound keeps the adapter within RTOL and ATOL
  for every probability up to one; the design note states it.
- The reviewer's three minimized subnormal cells become examples.
- np.errstate also ignores invalid, for the 0 * inf the overflow forms.
- The design note no longer says every unclipped cell meets the exact
  formula: weightless infeasible cells are skipped.

Correction to 3078d18's message, which this one supersedes: its "4
failed" of 1,000 fresh CI-sized runs at 5bda0fe was a measurement
error by the old runner. That runner caught only AssertionError and
stopped at the first ExceptionGroup, after 46 runs. A runner that
counts every failure measures 70 of 1,000 at 5bda0fe (seeds
1000-1999), as the reviewer found, and 0 of 3,000 at this commit
(seeds 1000-3999).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wording only, from the confirmation of 3384364: below 2**-1000 float64
can round a cell's products absolutely, but only those below 2**-1022
(subnormal) do; and where P = D exactly the adapter's factor is at most
about 19u/0.01, under the test's absolute tolerance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four files conflicted. Each is resolved as follows.

- scripts/first_estimates_birth_evidence.py and
  tests/estimates/test_birth_evidence_artifact.py: both sides added
  entries at the same place in POST_REVIEW_SOURCE_EXCLUSIONS and in its
  exact-tuple test. The resolution keeps both, SM1's five mortality_ses
  paths first and then master's two D1a reader paths, in the same order
  in the script and the test.
- tests/tier_counts.json and tests/README-tiers.md: master's versions,
  regenerated with recount-tiers.py on the merged tree. unit goes from
  18,433 to 18,549 and the total from 25,003 to 25,119. The +116 equals
  SM1's own delta on its base (tests/mortality_ses, 116 unit tests).

Against master the merged tree differs only in SM1's 14 files.

Checks on the merged tree, with an empty PSID root:
- full collection, -k tiers_match_policy_manifest: 1 passed, 4 skipped;
- test_birth_evidence_artifact.py -k post_review: 2 passed;
- tests/mortality_ses with RuntimeWarning as an error: 116 passed;
- test_m6_engine_steps.py and test_m6_engine_di_entitlement.py: 83 passed;
- tests/baselines -m unit: 29 passed;
- black -l 79, ruff and git diff --check clean on the changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 25, 2026 16:13
@MaxGhenis
MaxGhenis merged commit d830223 into master Oct 25, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Landing audit (Microcosm Dynamics coordinator, 2026-10-25 16:15Z).

Merged at the reviewed head 75df561e24382bc2bae32bdecfcad78a502caa87 with --match-head-commit, as merge commit d8302233. This PR held the tier-count slot under the coordinator's 11:05Z ruling: the first slot PR with an approved head merged with master 1ec57a5d, recounted tiers, green CI and a delta approval at that head lands first.

Gates checked at 16:14Z:

  • gh pr checks 570 exited 0: 11 passed, 1 skipped (deploy). Tests run 40205527021 succeeded at 75df561e.
  • MERGEABLE, merge state CLEAN, not a draft, no changes-requested review.
  • 75df561e's parents are cd12003a (the approved SM1 head) and master 1ec57a5d, which was still master at merge time.
  • Tier counts at the head: unit 18,433 → 18,549, total 25,003 → 25,119. The +116 is SM1's own tests, and the per-tier sum matches the README table.

Reviews:

  • GPT-6.1 Sol r3 approved ad827f1e.
  • Delta checks and confirmations through cd12003a all approved; the last two commits changed wording only.
  • An independent Opus 5.5 delta review approved the master merge at 75df561e, with no high or medium finding. It confirmed that the merge is clean, that the exclusions union holds 194 entries with no duplicates, and that the recount reproduces byte for byte.

Other open tier-slot PRs must merge master at d8302233 or later and recount before they land.

@MaxGhenis
MaxGhenis deleted the mgf-sm1-differential-mortality-20261024 branch October 25, 2026 16:25
MaxGhenis added a commit that referenced this pull request Oct 25, 2026
Master brings #587 (ledger delta), #575 (D1a) and #570 (SM1). None of
them changes the registry module, its tests or its mutation specs, and
the registry module imports only the standard library and yaml.

The tier conflict resolves to master's counts plus this branch's nine
unit tests: unit 18,549 + 9 = 18,558, total 25,128. The README tier
table now matches the tier file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 25, 2026
Master brings SM1 (#570). The merge has no conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 26, 2026
Master brings #570 (SM1: the mortality_ses package, its design note and
tests), #566 (OP1: the open_population package, its evidence and tests),
changes to scripts/first_estimates_birth_evidence.py and its artifact
test, and the tier files. None of it is a B0.2 file, and none of it is
imported by the B0.2 script or by anything that script imports.

The tier conflict resolves to master's counts plus this branch's unit
delta: 18,780 + 233 = 19,013 unit, with the other tiers as on master,
for a total of 25,583. The README tier table matches the tier file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 75df561e Deployed Oct 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant