Repository navigation
Add donor transport operators: reader, stable seeds, currency bridge and quantile map (NZ v0, G3b) - #1130
Merged
Conversation
Authenticate local populace-US donor inputs before decoding concepts; recode household roster pointers, preserve donor weights and support provenance, and derive stable take-up seeds from exact scoped source identities. Add one-step currency conversion and signed weighted quantile maps with declared interpolation, component isolation, and pro-rata entity aggregation. Cover the invariants with synthetic round trips, independent differentials, Hypothesis properties, and explicit float64 precision refusals. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…1130) The donor reader refused the real file it is pinned to. Treat a line pointer that is missing or <= 0 as absent (the pinned donor writes -1 for an absent parent or cohabiting partner and 0 for an absent spouse), as the US build's _parent_person_ids does for PEPAR1/2, and use the -1 sentinel in the synthetic fixture. Take partners only from A_SPOUSE, or from PECOHAB on rows whose vintage carries it, and refuse the two naming different people. Drop the relationship-code-13 recovery: code 13 also covers roommates and housemates (docs/us-asec-census-person-columns.md, COVERAGE_PROGRESS.md). On the pinned donor only the 2024 rows carry PECOHAB. Take the reference person from A_EXPRRP 1 or 2, fall back to is_household_head where a household has none, and refuse the whole donor if any household's two disagree. On the pinned donor 56,943 households have an A_EXPRRP reference person and 297 take the fallback; 113 of the 56,943 disagree, so the reader refuses that file until the hub rules. All 113 are 2022/2023 rows, whose A_EXPRRP equals the build's line-1 derivation (asec_pool._with_relationship_recode); is_household_head is P_SEQ == 1 on every row. A new smoke test runs the full reader on the pinned blob, skips when it is absent, and asserts that exact refusal, so any earlier failure (the old -1 refusal, a pin mismatch) turns it red. The is_household_head override before decode is dropped: with the disagreement refused, it rewrote every flag to its own value. Fix aggregate quantile_map refusing ordinary members, which made the PR's own Hypothesis property fail (8 of 10 seeds; the engine-free CI job's one failure). Correcting only the largest member cannot reach a total that the other members leave exactly halfway between two of its sums (members [1, 2] mapped to 0.21), nor one the residual step overshoots by several units (13 members mapped to 8192, where the spacing halves). After the old correction fails, restart the largest member from the correctly rounded difference (fsum), and if needed move one other member by one unit in the last place: up to four members finer than the largest, largest first, then the smallest, each away from and then toward zero. Inputs the old correction accepted map bit-identically, same-sign members always reach the exact total, and the refusal path stays O(n). Pin the counterexamples and add a same-sign exact-total property. Add a fault-injected test for the pro rata correction bound, which the exact-total check otherwise always caught first. Document pro rata amplification of nearly cancelling mixed-sign members, the remaining mixed-sign refusals, and the hash-then-reopen gap. Update the ADR status and changelog to say what the reader does on the pinned file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Require exactly one is_household_head person per donor household and use that person as its reference. Remove A_EXPRRP selection, fallback and disagreement refusal; document the P_SEQ and Census-recode evidence. Repair reciprocal two-person parent cycles by dropping the edge naming a younger parent, or both edges for equal ages or an absent age column. Check the original graph for longer cycles before mutation, preserve all other concept refusals, compact surviving parents and expose the dropped directed-edge count as DonorBank.dropped_parent_cycle_edges. Update regression tests, the cached full-reader smoke, ADR and changelog. The pinned donor now returns 57,240 households, 166,321 persons and two dropped parent edges with no concept violations. Targeted donor tests: 84 passed, no skips. Targeted quantile tests: 60 passed. Ruff and CI test layout/graph acceptance verification pass. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…exact lines (#1130) The delta review at 9df282d found that the top band's truncated-Pareto inverse lost precision for small alpha: it formed 1 - (lower/upper)**alpha, which cancels, so alpha=1e-18 on [10, 100] mapped the median to 10 instead of 31.6227766 and alpha=1e-16 was 4% off. _pareto_inverse now writes the truncated mass as -expm1(-alpha * w) with w = log(upper) - log(lower) and the offset as -log1p(-fraction * mass) / alpha, uses the exact alpha -> 0 limit (log-uniform, lower * exp(fraction * w)) when alpha * w underflows, clamps the offset to w, and forms lower * exp(offset) through log space only where the plain product would overflow inside a finite band. The untruncated path is the same expression with mass 1. Donor line numbers and pointers are validated as float64 (pointer columns can hold NaN), which holds every integer exactly only below 2**53; 2**53 + 1 rounded onto line 2**53. _line_numbers now refuses any magnitude >= 2**53 in the column's own dtype before conversion (infinities still fall to the integer check). CPS rosters are far below that. Tests (written by the previous GPT-6.1 Sol lane attempt, which stopped at the disk floor before any fix; the fix and the line-number unit test are the hub's): small-alpha medians against a 100-digit Decimal reference, a Hypothesis property against that reference over alpha in [1e-18, 50] and bands up to 400 decades (rtol 1e-12), the alpha -> 0 log-uniform limit at 1e-18, 1e-24 and 1e-100, and _line_numbers refusing 2**53, 2**53 + 1 and -(2**53) - 1 while keeping 2**53 - 1 exact. On 9df282d's transport.py the six Pareto cases and both line-number cases fail; with the fix all pass. The lane's HDF-level dangling-pointer test passes on both versions (the reader already refuses that fixture for another reason) and is kept as coverage. test_transport_quantile.py 66 passed; test_transport_donor.py 90 passed, including the full read of the pinned real donor (no skips). ruff check and format clean; tools/ci_test_plan.py verify ok. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review at ce4fd0e (in-session, two lenses plus a verifier, against mpmath at 120 digits) found one blocking defect in the same small-alpha class: the log-uniform limit ran only when -expm1(-alpha * w) was exactly 0, so a subnormal alpha * w kept only a few bits of the mass and returned silently wrong values (alpha=1e-320 on [10, 100]: 3.8e-4 relative; a single record at alpha=5e-324: 27.18 instead of 31.62). The limit now applies once alpha * w < 2**-60, where the dropped term is below 2**-61. The same review's accuracy nits, all fixed here: - ranks near 1 with mass near 1: log1p(-fraction * mass) cancelled (1.4e-3 relative at fraction = 1 - 2**-53 on [1e-10, 1e20], alpha=0.5); above fraction * mass = 1/2 the inverse now computes log((1 - fraction) + fraction * exp(-alpha * w)), where 1 - fraction is exact; - the band width is log1p((upper - lower) / lower) for narrow bands (upper - lower is exact there) and log(upper / lower) otherwise, and truncated results are clipped to [lower, upper] (they could exceed upper by ~1e-13 relative); - the untruncated path also falls back to log space where lower * exp(offset) alone overflows (lower=1e-300, alpha=1e-3 was refused although the quantiles are finite); - _line_numbers guards only values >= 2**53: only positive values can round onto a roster line, abs() wrapped int64's minimum, and negative pointers are absent as before 9df282d (negative lines stay invalid); - the HDF-level dangling-pointer test now repoints only row 0's household, uses line 2**53 - 1 with a pointer of 2**53, and matches the pointer column's own message, so each of A_SPOUSE, PECOHAB, PEPAR1 and PEPAR2 is reached (all four fail on 9df282d); - the high-precision property test's tolerance is condition-aware (1e-12 + 4 * kappa * 2**-53, kappa = d log x / d log rank), because quantile_map rounds ranks to float64; the review's counterexample (1e-200, 1e200, alpha=10**-2.1, weights (99, 1)) is now an example. Accuracy on an 8,000-point grid against an mpmath reference (alpha down to 3e-324, bands up to 600 decades, ranks to 1 - 3e-16): no non-finite or out-of-band value, and every error within the condition-aware tolerance (worst 0.17 of it). On ce4fd0e's transport.py, 10 of the new or extended cases fail; the four HDF dangling-pointer cases fail on 9df282d. Refusals that depend on float64 last bits (pro-rata exact totals, distinct mapped ranks) can move in either direction for a small share of synthetic mixed-sign inputs; they still fail closed. test_transport_quantile.py 77 passed; test_transport_donor.py 90 passed, including the full read of the pinned real donor. ruff check and format clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review at 833999b (APPROVE_WITH_NITS) found that the narrow-band width fix and the final clip were each untested on their own. A narrow-band test ([1e300, 1e300 * (1 + 1e-14)], alpha 2, four records) now requires quantile_map to accept the band and match the high-precision reference; with the naive log(upper) - log(lower) width it refuses the distinct ranks. A clip test requires _pareto_inverse(1 - 2**-53, 1e-300, 1e-299, 0.1) <= 1e-299; without the clip it returns 1.0000000000000003e-299. The HDF pointer test is renamed to what it checks (pointer columns at or above 2**53, refused by their own name) and its comment says 9df282d refused the same input with a different message. test_transport_quantile.py 79 passed; test_transport_donor.py 90 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Merged by the NZ hub (session local_e407a527) at 2089808 with --merge. Gates: |
MaxGhenis
added a commit
that referenced
this pull request
Oct 9, 2026
Two conflicts: - docs/concept-schema-transport-adr.md: keep both sides. Main's status text (#1130) describes the transport reader; G4 adds that benefit-unit construction from the concept pointers is built for New Zealand. Main's closing bullet listed 13 of New Zealand's 80 module bindings as deferred until unit construction exists; G4 builds it, so the bullet now says New Zealand's group bindings execute on built units, Belgium's and the US/UK group bindings stay deferred, and a country pack is still needed. - packages/microcosm-build/tests/golden/nz_country_spec.json: regenerated from the loaded spec (canonical_json_bytes(_loaded_spec_summary("nz"))), not hand-merged. Resource hashes are the union of both sides (G4's axiom_input_closure.json and country_package.json, #1138's gates.json); fingerprint 09ed486f... Tests (one file at a time, after `uv sync --all-packages --locked --inexact` for main's new sqlalchemy dependency): country spec 135, NZ input closure 17 (1 skipped), NZ spec package 71, transport gate bindings 110, group encode 78, unit construction 38, input closure 52, concept mapping 168; all passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Authenticate local populace-US donor inputs before decoding concepts; recode
household roster pointers, preserve donor weights and support provenance, and
derive stable take-up seeds from exact scoped source identities.
Add one-step currency conversion and signed weighted quantile maps with
declared interpolation, component isolation, and pro-rata entity aggregation.
Cover the invariants with synthetic round trips, independent differentials,
Hypothesis properties, and explicit float64 precision refusals.
What this PR does now (head 2089808)
read_populace_us_donor): authenticates the pinned populace-US file (SHA-256 and size), reads CPS line pointers as the file writes them (any pointer<= 0is absent; values at or above 2**53, which float64 cannot hold exactly, are refused), uses the donor'sis_household_headas the reference person (exactly one per household, refused otherwise), sets partner pointers only fromA_SPOUSE > 0orPECOHAB > 0(CPS code 13 also covers roommates, so it is not used), and drops the impossible edge of a two-person parent cycle (the named parent younger than the child; equal or missing ages drop both; longer cycles refuse). On the pinned real donor: 57,240 households, 166,321 persons, 2 cycle edges dropped. The reference-person and parent-cycle rules are the NZ hub's defaults, recorded for the D1 method card (cos d1043).expm1/log1p), uses the exact log-uniform limit oncealpha * w < 2**-60, computes ranks near 1 aslog((1 - f) + f * exp(-alpha * w)), measures narrow bands withlog1pof the exact width, clips truncated results to the band, and reaches finite untruncated quantiles through log space. On a 170,860-point grid against mpmath (alpha down to 5e-324, bands to 600 decades, ranks tonextafter(1, 0)), the independent review found no non-finite, out-of-band or non-monotone value and a worst error of 3.5e-13.Review history
Tests (at 2089808)
test_transport_quantile.py79 passed;test_transport_donor.py90 passed, including the full read of the pinned real donor; earlier rounds also rantest_concept_mapping.py(146 passed). Every regression added in rounds 3 and 4 fails on the code before its fix.Built on Subfleet lanes (GPT-6.1 Sol and Claude Opus 5.5, named in each commit's trailers) and by the NZ hub; the original build-lane report is superseded by the sections above.
Shared files and owners
packages/microcosm-frame/src/microcosm/frame/transport.py, tests and a synthetic fixture. The only tracked file changed isdocs/concept-schema-transport-adr.md(status line).liquid_financial_assetsis quantile-mapped by these operators.Part of the Microcosm NZ v0 graph plan (Max ruling d962). Built by the NZ hub on a Subfleet build lane (model named in the commit trailer); the hub verified scope (no frozen graph files, uk_runtime/us_runtime or frame/kernels.py touched) before committing.
🤖 Generated with Claude Code