Skip to content

S0.1: source, exposure, access and power audit for SM2–SM4 and ED1–ED2 - #588

Merged
MaxGhenis merged 9 commits into
masterfrom
mgf-s0-1-source-audit-20261024
Nov 1, 2026
Merged

MaxGhenis merged 9 commits into
masterfrom
mgf-s0-1-source-audit-20261024

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

S0.1 of the mortality, groups and financing lane: the source, exposure, access and power audit for the gates SM2, SM3, SM4, ED1 and ED2. It is validation-only. It freezes what each gate's later registered test will use, and computes power on invented data. It estimates nothing from real data.

The coordinator lands this PR at its tier-count slot.

Sources:

What it adds

  • docs/design/s0_1_source_audit.md: the frozen record, one section per gate, in the order SM3, SM2, SM4, ED1, ED2. For each gate it records:

    • the sources, with their manifest hashes;
    • the population and estimands;
    • the exposure history;
    • the holdout partition and prediction origin;
    • the critical cells and tolerances;
    • the uncertainty treatment;
    • the pass conjunction and power procedure;
    • the suppression threshold;
    • the named conditions.

    It marks each item frozen, proposed or open. It also has a code-input metadata table (each repository input's path, commit, SHA-256, verification date and reliance). It ends with the input for Q7, what SM3's real-data step would need, and makes no recommendation on authorization.

  • scripts/s0_1_power.py and its committed results, docs/design/s0_1_power_results.json. The script simulates SM3's and ED2's registered tests on invented populations, over a declared grid of assumed counts, and reports power at the 0.90 criterion. Its threshold helpers return the minimal sufficient count or raise a named refusal.

  • tests/s0_1/: 160 example and Hypothesis property tests on invented data.

  • Housekeeping: the tier counts.

Invariants

Tested as properties (Hypothesis) and examples, on invented data:

  • simulated type-I error under the null is within Monte Carlo error of the nominal level;
  • power is nondecreasing in sample size and in effect size;
  • fixed seeds are deterministic;
  • Bonferroni bounds cover at least nominally across the registered family;
  • threshold helpers: for every accepted input, the returned count meets the helper's own floating-point predicate and, above the smallest supported count, one fewer does not. The supported counts are the integers from one through MAX_SUPPORTED_COUNT. Every other input ends in a named refusal (ThresholdInputError, UnreachableThreshold or UnrepresentableThreshold, all ThresholdRefusal subclasses of ValueError), never None or another exception;
  • failure preservation: an earlier FAIL survives any later inestimable cell, and inestimability never produces a PASS;
  • malformed inputs are refused, and insufficient precision is inconclusive, never a pass.

Review r3 found two places where earlier code broke an invariant it stated. Round 3 fixed both, and round 4 fixed the two numerical gaps review r4 then found in the threshold helpers. Round 4 deliberately refuses some inputs round 3 returned a result for; the note lists those kinds. The recorded planning and validation counts are unchanged.

Checks

  • On the host, at the committed tree, with an empty PSID root: tests/s0_1: 160 passed, under an audit hook that counted opens of the files the project bars from these runs. It counted none, and no populace_dynamics module was loaded. The full-collection tier check runs at the coordinator's slot.
  • Values scans (validation-only jobs) after each round: scans r5, r6, r8 and r9 covered rounds 3, 4, 5 and 6, and each returned CLEAR. Each audited the fix job's every read from its event stream, and checked every changed line against the restricted sources. They found no line in a restricted class and no carried-over value.
  • A first attempt at round 5 was set aside. Its values scan, r7, found no carried-over value but could not certify the job's reads, because the job had searched code directories more broadly than its brief allowed. The lane session restored the round-4 note and reran the round as fix r5b under stricter search rules. Scan r8 certified r5b's reads.
  • Review r5 20261026-190834-mgf-s01-review-r5 (GPT-6.1 Sol): APPROVE, with no high or medium finding.
    • It proved sufficiency and strict minimality for the helpers and tried to break them on plateaus, underflow, adjacent target floats and the domain's edges. It found no counterexample, and no caller that swallows a refusal.
    • It reran the script without its anchor inputs; every non-anchor field matched the committed results exactly.
    • It checked that review r4's two medium findings and its qualification are resolved, and that no earlier fix regressed.
  • Review r6 20261026-215157-mgf-s01-review-r6 (GPT-6.1 Sol), on round 5: APPROVE, with one low wording finding, fixed in round 6.
    • It checked the new support definition against the code, and probed the list of refused inputs on invented argument, type and boundary cases, and found no kind the note misses.
    • It confirmed that review r5's two findings are resolved and that no earlier fix regressed.
  • Review r7 20261026-223940-mgf-s01-review-r7 (GPT-6.1 Sol), on round 6: APPROVE, with no findings.
    • It confirmed from the code and invented cases that both helpers can return one and never evaluate zero, and that both statements of the one-fewer property now except a threshold of one.
    • It replayed review r6's finding, checked every minimality statement in all nine of the PR's files, reran tests/s0_1 (all passed), and found no earlier fix narrowed or regressed.
  • The committed blobs match the SHA-256 hashes scan r9 and review r7 covered.

Tiers (on master 6413d842):

Tier Before After
unit 19,504 19,664
Total 26,074 26,234

The other tiers keep master's counts. The 160 new unit tests are tests/s0_1: a named collection on the merged tree gives 160 unit tests and none in any other tier, and CI's full collection checks the total. The master merge took master's two tier files, and one later commit recounts them. This PR's seven other files keep the blobs review r7 covered at 781ba61f.

Provenance

  • Built by Subfleet job 20261024-110212-mgf-s0-1-build.
  • Revised in six fix rounds: 20261024-211800-mgf-s01-fix-r1, 20261025-015100-mgf-s01-fix-r2b, 20261025-114320-mgf-s01-fix-r3, 20261026-162925-mgf-s01-fix-r4, 20261026-203129-mgf-s01-fix-r5b and 20261026-220958-mgf-s01-fix-r6. The set-aside attempt was 20261026-194239-mgf-s01-fix-r5.
  • Values-scanned after the build and after each round: 20261024-171232-mgf-s01-values-scan, then delta scans r2 to r9 (the last 20261026-222719-mgf-s01-values-scan-r9).
  • Committed by the lane session (Claude Opus 5.5): the fix jobs left their trees uncommitted.
  • Data: invented data only. PSID use follows the project's data-access record.

Review

  • Rounds 1 and 2b returned REVISE; fix rounds r1 and r2b addressed them.
  • Round 3 (20261025-105651-mgf-s01-review-r3) returned APPROVE with five low findings, fixed in round 3.
  • Round 4 (20261026-131614-mgf-s01-review-r4) returned REVISE with two medium findings in the threshold helpers' numerical handling: a reachable finite target could return None, and some numerical endpoints raised raw exceptions. Round 4 fixed both.
  • Round 5 (20261026-190834-mgf-s01-review-r5) returned APPROVE with two low documentation findings: the support definition omitted the cap, and the preservation claim was unqualified. Round 5 fixed both.
  • Round 6 (20261026-215157-mgf-s01-review-r6) returned APPROVE with one low finding: the note's statement that each reported threshold passes and one fewer fails needed the lower-end exception, since the smallest supported count is one and zero is unsupported. Round 6 fixed it.
  • Round 7 (20261026-223940-mgf-s01-review-r7) returned APPROVE with no findings.

axiom: n/a: source audit and invented-data power calculations

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits October 25, 2026 11:37
docs/design/s0_1_source_audit.md is the frozen record that S0.1 owes
the lane's design (#558, approved under d1922). It covers the gates
SM3, SM2, SM4, ED1 and ED2, in that order. For each gate it records:
- the sources, with their manifest hashes;
- the population and estimands;
- the exposure history;
- the holdout partition and prediction origin;
- the critical cells and tolerances;
- the uncertainty treatment;
- the pass conjunction and power procedure;
- the suppression threshold;
- the named conditions.

Each item is marked frozen, proposed or open. The record ends with
the input for Q7, what SM3's real-data step would need, and makes no
recommendation on authorization. The group gates (GR1 to GR4) are out
of scope; their audit is briefed separately.

Provenance:
- built by Subfleet job 20261024-110212-mgf-s0-1-build;
- revised in fix rounds 20261024-211800-mgf-s01-fix-r1 and
  20261025-015100-mgf-s01-fix-r2b;
- values-scanned by 20261025-052455-mgf-s01-values-scan-r4, which
  returned CLEAR;
- reviewed by 20261025-105651-mgf-s01-review-r3, which returned
  APPROVE with five low findings. A follow-up round fixes them.

The build lanes' sandboxes could not write .git, so the lane session
commits the tree they left. It matches the reviewed SHA-256 hashes.

PSID use follows the project's data-access record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/s0_1_power.py simulates SM3's and ED2's registered tests on
invented populations. It reports power at the 0.90 criterion over a
declared grid of assumed counts. docs/design/s0_1_power_results.json
holds its committed results.

tests/s0_1/ holds 99 example and Hypothesis property tests on
invented data. They cover type-I error under the null, power that is
monotone in sample and effect size, seed determinism, Bonferroni
coverage and refusals.

Checks on the host, with an empty PSID root:
- tests/s0_1: 99 passed;
- Black (79 columns) and Ruff: clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/s0_1 adds 99 unit tests. On the base 2195258, unit goes from
18,088 to 18,187 and the total from 24,619 to 24,718. Full collection
with -k tiers_match_policy_manifest passes on the host: 1 passed,
2 skipped. The counts are recounted on fresh master at the
coordinator's tier-count slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
social-security-model Ready Ready Preview Nov 1, 2026 6:18pm UTC

Request Review

MaxGhenis and others added 2 commits October 26, 2026 19:40
Two fix rounds change scripts/s0_1_power.py, its tests and the audit
note. The results file is unchanged: every non-anchor field reproduces
exactly.

Round 3 (Subfleet job 20261025-114320-mgf-s01-fix-r3) fixed review
r3's five low findings:
- the threshold helpers refuse a threshold the floating-point model
  cannot represent, verify each returned count through
  family_power_lower_bound, and find the minimal count by an integer
  search;
- the SM3 boundary test reports pass counts and treats inconclusive
  runs explicitly, instead of a vacuous assertion;
- the note gains a code-input metadata table and two locators for the
  DI composition in mortality_ses/model.py;
- property tests check that an earlier FAIL survives any later
  inestimable cell, and that inestimability never yields a PASS.

Round 4 (job 20261026-162925-mgf-s01-fix-r4) fixed review r4's two
medium findings:
- a reachable finite target no longer returns None. Reachability is
  decided by the same predicate the search uses, at the largest
  supported count. A target no count can reach raises
  UnreachableThreshold; one the model cannot represent raises
  UnrepresentableThreshold. Both, with ThresholdInputError, are
  ThresholdRefusal subclasses of ValueError;
- every argument is validated on entry, one observation is tried
  first, a critical value must be finite and positive, very large
  family sizes are refused by name, and a closed-form union bound
  replaces the per-cell list for equal cells;
- the note and docstrings state that sufficiency is defined by the
  helper's own floating-point predicate.

Checks:
- values scans 20261026-100223-mgf-s01-values-scan-r5 and
  20261026-184550-mgf-s01-values-scan-r6: CLEAR;
- review 20261026-190834-mgf-s01-review-r5: APPROVE, with two low
  documentation findings that a follow-up round fixes;
- tests/s0_1 on the host, with an empty PSID root: 160 passed.

The fix jobs left their changes uncommitted, so the lane session
commits the tree. It matches the reviewed SHA-256 hashes.

PSID use follows the project's data-access record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/s0_1 now holds 160 unit tests, up from 99. On the base 2195258,
unit goes from 18,187 to 18,248 and the total from 24,718 to 24,779.
The counts are recounted on fresh master at the coordinator's
tier-count slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r5 approved the round-4 tree with two low findings in the audit
note's wording. This round changes only docs/design/s0_1_source_audit.md:

- It defines the supported counts as the code does: the integers from
  one through MAX_SUPPORTED_COUNT.
- It replaces the round-4 record's unqualified preservation claim. The
  recorded planning and validation counts are unchanged, but round 4
  refuses some inputs round 3 returned a result for. The note lists those
  kinds, with the exception each raises, and records that a non-double
  criterion can change a count.
- It adds the round-5 record.

The script, results, tests and tier files are unchanged.

Made by Subfleet job 20261026-203129-mgf-s01-fix-r5b. Values scan
20261026-213217-mgf-s01-values-scan-r8: CLEAR. Review
20261026-215157-mgf-s01-review-r6: APPROVE, with one low finding on
wording that predates this round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r6 approved the round-5b tree with one low finding. The audit
note said each reported threshold passes and one fewer fails, but a
reported threshold can be one, and zero is not a supported count. This
round changes only docs/design/s0_1_source_audit.md:

- Both statements of that property now except a threshold of one.
- It adds the round-6 record.

The script, results, tests and tier files are unchanged.

Made by Subfleet job 20261026-220958-mgf-s01-fix-r6. Values scan
20261026-222719-mgf-s01-values-scan-r9: CLEAR. Review
20261026-223940-mgf-s01-review-r7: APPROVE, no findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 1, 2026 13:14
Conflicts: tests/README-tiers.md and tests/tier_counts.json only. Both take
master's blobs here; the next commit recounts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
unit 19,504 -> 19,664, total 26,074 -> 26,234. The 160 new unit tests are
tests/s0_1, this PR's own modules; the other tiers keep master's counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 1, 2026 18:16
@MaxGhenis
MaxGhenis merged commit 1808285 into master Nov 1, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

The Microcosm Dynamics coordinator merged this PR at 502915ea54dfad3cb21837fcdcfb90f238845505 with --merge --match-head-commit, as 1808285925a069143047ae7e7bb2412012db0ad7 (19:31Z, 2026-11-01).

  • Review. Review r7 (GPT-6.1 Sol, job 20261026-223940-mgf-s01-review-r7) approved 781ba61f with no findings. The two later commits change no reviewed content:
    • 1f46d682 merges master 6413d842. The PR's seven non-tier files keep their 781ba61f blobs, and every other path equals master's.
    • 502915ea recounts: it touches only tests/README-tiers.md and tests/tier_counts.json, and adds the 160 unit tests in tests/s0_1 (unit 19,664, total 26,234).
  • CI. gh pr checks 588 exited 0 at the head, 11 checks passing and deploy skipped: run 41335801487, with its four pytest shards and graph-integration 3.13 and 3.14. The full collection in the tier-policy test confirms the total.
  • The slot now. Master is 18082859. Each remaining tier-slot PR merges it, or later, and recounts.

This branch was successfully deployed

1 active deployment
Preview — 502915ea Deployed Nov 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant