[DTrim99]

Summary

Part of the US 2036 extension, tracked in PolicyEngine/policyengine-api#3926. CBO's February 2026 baseline runs through 2036, and its 10-year budget window is now 2027–2036. The canonical forecast ended in 2035, so every 2036 SPM request downstream failed with SPM_YEAR_UNAVAILABLE.

This PR extends the forecast horizon to 2036 without changing any 2022–2035 value, and sets the version to 1.1.0. The publish workflow uploads on push to main with skip-existing, so merging this publishes 1.1.0.

@MaxGhenis rebuilt the CE and ACS components from the 9/9 raw caches (2d8acf0). The full horizon verifier accepts the rebuild:

• Every 2022–2035 field is unchanged.
• All 2,110,752 retained calculations are bit-for-bit equal: 2 scenarios × 14 years × 349 areas × 3 tenures × 72 compositions.
• The retained-grid digest (aaf2d3d4…) equals the one recorded by the 9/9 receipt.

New identities

Identity	1.0.x (2022–2035)	1.1.0 (2022–2036)
Content (expected_sha256)	3d86d5c4…	335e6755dc8e0bd883e09b76c2bd53a5315683773e44d50085c7d6b8832b6291
File bytes	cc06784f…	53d2b4b32251d3afbcf62a76c0f843c2a9369b3565acf088d6729b3bd8069d1b
Assumptions	d81cf3b1…	6ef10c97d39a95720cce7993036b3db702b9f7c8c71a1e1ec37c104d85f18dfd

Downstream bundles (PolicyEngine-US, policyengine.py) must adopt the new content digest explicitly.

2036 national renter thresholds:

Scenario	2035	2036	Growth
ce_trend	$58,520.42	$60,388.65	+3.19%
zero_real	$54,735.04	$55,970.71	+2.26%

zero_real grows at CBO's 2036 CPI-U rate (2.2575%), as expected.

What changes

Horizon (hashed scientific code):

• spm_calculator/forecast_inputs.py: projection_rates(end_year=2036).
• spm_calculator/ce_forecast.py: build_ce_forecast(end_year=2036).
• scripts/build_forecast_inputs.py: CBO price growth for 2026–2036, and county year maps for 2022–2036.
• scripts/build_acs_forecast.py: end_spm_year=2036, and PUMA vintages through 2036.
• scripts/build_rolling_forecast.py: assembly requires 2022–2036.

forecast_horizon_inputs.json, rebuilt offline. Exactly five values change:

• prices.annual_growth_by_year["2036"] is added. It is CBO's 416.542 / 407.346 − 1 = 2.2575%, and the pinned cbo_calendar_2026_02.csv already held 2036.
• county_assignments.year_maps["2036"] is added. It points to the 2023 map, like 2023–2035.
• The assignment, generator and content digests are recomputed.

Rebuilt components (Max, 2d8acf0): acs_2021_source_inputs.json, ce_rolling_forecast.json, acs_rolling_forecast.json and the reassembled rolling_forecast_2026_09_09.json.

scripts/verify_horizon_extension.py (new) compares the rebuilt forecast with the retained pre-extension download, rolling-forecast-cc06784f….json.

• It reverses only the declared edits, then requires byte-identical canonical JSON. Any other difference fails.
• The declared edits are:
  • the derived digests;
  • the five code identities above;
  • the four rebuilt components or inputs;
  • relative_index_stabilization.through_year (2035 → 2036);
  • year-keyed records that gain exactly "2036";
  • the 9/9 adaptation link, which a fresh ACS build does not repeat.
• It checks that the 2036 entries match 2035's structure and statuses and use fully projected windows.
• It runs the full household grid against the old artifact, and calculates 2036 once (150,768 calculations).
• Its receipt replaces data/current/scientific_refresh_receipt.json and links the archived 9/9 receipt as its predecessor.

Tests for the verifier: tests/test_horizon_extension.py builds a synthetic extension and checks that 12 resealed mutations are rejected. A new test binds the committed receipt to this artifact, its rebuilt components and its code. It also checks that the receipt's retained-grid digest matches the 9/9 receipt's.

The 9/9 evidence is retained, and the adaptation scripts don't change.

• acs_code_identity_adaptation.json pins the bytes of adapt_acs_code_identity.py, and the archived receipt pins verify_provenance_equivalence.py. Both scripts therefore stay byte-identical.
• data/provenance/horizon-2036-extension/ keeps the files the rebuild replaces: the adapted ACS component, the CE component (gzip, mtime=0) and the adaptation's verification receipt. The archive is in package-data.
• retained_bytes() maps each replaced path to those bytes and checks their pins. test_provenance_adaptation.py and test_provenance_equivalence.py read through it.
• adapt_acs_code_identity.py --check itself no longer reproduces current files, because the rebuild replaced the adapted component. The docs now list verify_horizon_extension.py instead.

Browser export:

• release_config.json carries the 2036 forecast, and canonical/rolling-forecast-53d2b4b3….json is added as its content-addressed download. The older downloads stay.
• The export is a local preview of 1.1.0 (status local_preview, no PyPI link), as Replace legacy SPM calculations with canonical rolling CE and ACS forecasts #36 shipped new content. It names no published package whose bundled forecast differs from the one it serves. A bare export_web_release.py --check passes, and test_web_config.py compares the committed export with a fresh one. test_release.py reproduces the export in either status. The preview link points at this PR.
• Promotion to the published 1.1.0 follows the PyPI upload (Release spm-calculator 1.1.0 with the 2022–2036 forecast #52). Vercel deploys about a minute after the merge, and PyPI about 20 minutes later, so a staged "published" label would have pointed at a package that cannot load this forecast.

Verifier bindings (round 1, SA3): tests/test_horizon_bindings.py adds six tests:

• the retained content of the artifact and of both rebuilt components equals the archived bytes, except at the declared paths;
• every 2036 value follows the documented rule: CPI-U from the pinned CBO levels, times the real trend for ce_trend, with 2035's housing shares, rent indices and geography;
• the receipt builder reproduces the committed receipt;
• the calculation grid rejects a retained amount change.

Each mutation case in test_horizon_extension.py now names its expected message.

Rebuild record (SA4): the receipt no longer carries a constant scientific_component_rebuild_performed. It records what the artifacts show: all 56 external source pins and the normalized ACS products (49b12d77…) are unchanged from 1.0.x. The commands, the ACS cache mode and each builder's --check result are left to the operator record here. docs/spm-releases.md gives the full rebuild order. The verifier gains --check, which reruns the full grid and compares with the committed receipt, and it always reads the committed artifacts. The receipt is regenerated with the full grid; only its audit_script hash and the rebuild block change, and 335e6755…, 53d2b4b3… and 6ef10c97… do not move.

Tests: coverage moves to 2022–2036 in these places:

• the CLI, artifact, Microcosm, Axiom and web config tests, plus the stabilization disclosure;
• the web tests that read the real export (compactParity, offlineRelease) and the e2e spec. Their horizon-end checks now use 2036, and the year chart has 14 segments.

The unit tests built on the synthetic 2022–2035 fixture are unchanged. test_full_scenarios_preserve_every_original_value still hashes 2022–2035 against the original digest.

Docs: these now run through 2036:

• coverage statements, the window table and the CPI path;
• the 2036 example thresholds;
• the identity table in docs/spm-releases.md, with the 1.0.x pins kept as history;
• the verification commands.

The README's install lines target 1.1.0, because the README becomes the 1.1.0 PyPI page. The site description and the BLS-correction note now say 2036. docs/index.md, quickstart.md and migration.md move with the promotion (#52).

Also in 1.1.0: the wheel is built from main, so it also ships #45. #45 changes how spm_unit_id attaches foster children and unrelated children under 15 with a linked parent. No digest moves, and policyengine-us does not call it.

Verification

CI on b145fcc is green, Vercel included:

• CI: tests on Python 3.9–3.14, the web unit tests and the docs build. The Python tests include the receipt-binding test and test_committed_forecast_rebuilds_offline.
• Calculator browser QA: the e2e spec against the real static export, both standalone and under /us/spm-calculator.

Local runs (Python 3.11, Windows):

• verify_horizon_extension.py with the full grid passes and writes the committed receipt.
• tests/test_horizon_extension.py and tests/test_web_config.py pass: 78 tests.
• After round 1 (ea3362c):
  • verify_horizon_extension.py --output (full grid, about 3 minutes here), then --check: matches.
  • export_web_release.py --check: passes.
  • 144 passed in test_horizon_bindings.py, test_horizon_extension.py, test_web_config.py, test_release.py, the provenance suites and test_scientific_refresh.py. One test_provenance_adaptation.py case passes only in UTF-8 mode on Windows: it reads the module with the locale encoding, cp1252 here. Linux CI is unaffected.
• Skipped in CI and here: test_microcosm_adapter.py and the Microcosm-backed Axiom cases need microcosm and pandas, which no extra installs. The Axiom acceptance cases need AXIOM_CORE_BIN. Their 2036 coverage is unverified until someone runs them with those installed.
• On Windows, build_rolling_forecast.py --check and build_release.py --check differ only by \ path separators in component paths. test_published_web_export_reproduces_exact_bundled_bytes needs symlink privileges. All three pass on Linux CI.

Merge order

This is the SPM rebuild that the 2036 merge hold in PolicyEngine/policyengine-api#3926 waits on.

1. Merging publishes 1.1.0.
2. Then the browser export is promoted to 1.1.0 (Release spm-calculator 1.1.0 with the 2022–2036 forecast #52).
3. Then the PolicyEngine-US pin moves to spm-calculator>=1.1.0,<1.2, and the policyengine.py bundle is certified (Certify a bundle with spm-calculator 1.1.0 and the 2022–2036 SPM forecast policyengine.py#618).

🤖 Generated with Claude Code

[vercel]

@DTrim99 is attempting to deploy a commit to the PolicyEngine Team on Vercel.

A member of the Team first needs to authorize it.

[DTrim99]

First CI run, as expected:

• Python 3.11, 3.12 and 3.14 each had 13 failed, 513 passed, 83 skipped, with the same 13 tests. Fail-fast cancelled 3.9, 3.10 and 3.13.
• Twelve of the 13 need the rebuilt 2022–2036 artifact:
  • test_axiom_adapter: the portable-export endpoint probe;
  • test_cli: the year list;
  • test_rolling_forecast_artifact: year coverage;
  • test_scientific_refresh: stabilization through 2036;
  • nine test_web_config cases, for 2036 and the web export.
• test_committed_forecast_rebuilds_offline fails with "Historical county assignment changed". That is the 2021 bundle's county digest, which build_acs_2021_inputs.py in the rebuild updates.
• The repointed provenance suites (test_provenance_adaptation, test_provenance_equivalence) and the new test_horizon_extension pass.
• Vercel fails only because a fork PR needs deploy authorization.

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[DTrim99]

@MaxGhenis this is ready for your review. Your rebuild passes the full horizon verifier: every 2022–2035 value and all 2,110,752 retained calculations are unchanged, and the retained-grid digest matches the 9/9 receipt. CI and browser QA are green on 836762d. The browser export keeps the 1.0.0.post1 package pin, as in #41 and #50; promotion to 1.1.0 follows the PyPI upload (#52). Per the hold in PolicyEngine/policyengine-api#3926, this is the rebuild the other 2036 PRs wait on.

[DTrim99]

@PavelMakarchuk you're now the primary reviewer for the 2036 work, so tagging you here. This PR is the critical path: merging it publishes 1.1.0, which the other 2036 PRs wait on (PolicyEngine/policyengine-api#3926). Summary and verification are in the description above. Max did the component rebuild, so he may also want to sign off before it merges.

[MaxGhenis]

Independent review (US + core hub, GPT-6.1 Sol, shell-backed): APPROVE at 836762d6dd, no findings.

• Full horizon verifier: scripts/verify_horizon_extension.py passed the whole grid. Every retained 2022–2035 field is unchanged, all 2,110,752 retained calculations are bit-for-bit equal, and all 150,768 calculations for 2036 pass. The regenerated receipt is byte-identical to the committed one, and the retained-grid digest matches the 9/9 receipt (aaf2d3d4…).
• 2036 national renter thresholds, recomputed independently: zero_real $55,970.71 (2035 × 416.542 / 407.346, CPI-U +2.2575%) and ce_trend $60,388.65. Both match the committed values to within $3×10⁻¹¹.
• Declared edits only: a recursive base/head comparison finds exactly the five horizon-input changes. 2036 uses the 2023 county map, and the content, file-byte and assumptions digests all match the body.
• Tests: test_horizon_extension 16, test_rolling_forecast_artifact 17, test_scientific_refresh 12, test_provenance_equivalence 12 and test_web_config 62, all passed.
• Release safety: 1.1.0 isn't on PyPI yet. PE-US pins >=1.0.0,<=1.0.0.post1 and policyengine.py pins ==1.0.0, so publishing changes neither until each adopts the new digest.
• One small note: a bare export_web_release.py --check fails because of the intentionally staged package pin. The exact export comparison passes once only the declared package fields are normalized, so the body's wording about the bare command is slightly imprecise.

Still open before merge: the Vercel check fails with "Authorization required to deploy". It fails only on this author's commits, so it needs a Vercel team owner, and I've flagged it to Max. Pavel's review is also still to come. The hub will merge once both clear.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
spm-calculator	Ready	Preview	Oct 28, 2026 2:01pm UTC

[DTrim99]

Vercel is authorized and green on 836762d.

[PavelMakarchuk]

Release Review

PR #51, "Extend the canonical forecast horizon to 2036" (DTrim99; CE and ACS rebuild commit 2d8acf0 by MaxGhenis). Round 1.

IDs: C = critical, SA = should address, G = suggestion.

Paths: V = scripts/verify_horizon_extension.py, TH = tests/test_horizon_extension.py, RC = web/public/data/release_config.json, CW = web/src/components/CalculatorWorkbench.jsx, A = spm_calculator/data/current/rolling_forecast_2026_09_09.json. Line numbers refer to the reviewed head.

Source Documents

• Reviewed head SHA: 836762d
• Mode: full
• Scope: 8 commits, 46 files, +898,613 / −115,909.
• Merge base 13ab3014d0 against head: A, the four rebuilt or regenerated inputs, both receipts, the provenance archive and the web export (type-strict JSON walks, and the library run at each snapshot). Also the retained download rolling-forecast-cc06784f….json and the archived 9/9 receipt b0a27cd8….
• CBO: workbook 51135-2026-02-Economic-Projections.xlsx, sheet "2. Calendar Year", CPI-U row (Wayback captures, because cbo.gov blocks scripts), and US-CBO/cbo-data calendar_2026-02.csv at the pinned commit 284a95665f9f, byte-identical to the pinned cbo_calendar_2026_02.csv.
• Wheels: PyPI 1.0.0.post1 and 1.0.1; 1.1.0 wheel and sdist built offline from the head and from the head plus main.
• Release and downstream: publish.yaml, the pypi environment, Vercel deployments, the live release_config.json, issue #52, policyengine-us main, policyengine-api and policyengine-sim-api spm.py, policyengine.py#618.

Branch Status

• 8 ahead of and 1 behind main. The one commit is 4ff8a8c (#45, MaxGhenis), which changes units.py, docs/api/units.md and tests/test_units.py. No file overlap; the merge is clean.
• units.py is not in A's 15 code_sha256 entries, so no digest moves. But the 1.1.0 wheel is built from main after the merge, so it ships #45's units.py (G3).

What the PR gets right

Every number in the body reproduces, by checks independent of the PR's verifier:

• C1. A type-strict walk of base against head A gives exactly the receipt's 41 allowed_changes (plus the sources length). The loader gives 3d86…/2022-2035 at base and 335e…/2022-2036 at head.
• C2. The library at base and at head writes cmp-identical packed float64 grids (2,110,752 calculations). The 2036 digest 547867b3… reproduces with the library and with a raw re-implementation; no 2036 amount is NaN, zero or negative.
• C3. aaf2d3d4… recomputes from the base library, the head library and a re-implementation without spm_calculator. All four receipt fields agree.
• C4. 335e6755… and 6ef10c97… recompute without the library (base gives 3d86…, d81c…). 53d2b4b3… is the package file, and the web mirror is cmp-identical. Receipt, RC and docs/spm-releases.md:97-99 agree, and every component_sha256 and code_sha256 matches its file.
• C5. The CBO workbook and the pinned CSV agree byte for byte. 416.542 / 407.346 − 1 = 0.02257540272888403, bit-equal, from the calendar-year series (fiscal-year would give 2.25734%). Exactly five values change, and build_forecast_inputs.py --check passes offline. year_maps["2036"] = "2023" follows the existing rule.
• C6. calculate_unit gives $60,388.65 and $55,970.71. zero_real is 2035 × 416.542 / 407.346 to 1.5e-11; ce_trend is 2035 × (1 + CPI-U) × e^0.0091012 to 4.4e-16. All 349 areas grow at the national rate.
• C7. The verifier rejects all 12 resealed mutations, each for its intended reason; the loader alone accepts all 12. The archive ships in the wheel and matches its pins.
• C8. True as worded: only the two package fields differ from a fresh export, and bare --check fails. SA1 is about what that staged pin now means.
• A full-grid rerun of the verifier exits 0 in 65 s with a receipt byte-identical to the committed one.

Critical (Must Fix)

None.

Should Address

SA1. The merge deploys a "published spm-calculator 1.0.0.post1" label on 2036 content that 1.0.0.post1 cannot load.

• Where: RC; CW:225-234, CW:339-341, CW:1272-1276, CW:1334-1336; tests/test_web_config.py:76-104; scripts/export_web_release.py:56.

• Evidence:

  • RC says status published, version 1.0.0.post1, content 335e6755…. Run on that file, the card logic shows "Reproduce with published spm-calculator 1.0.0.post1", links to PyPI 1.0.0.post1, offers load_forecast(expected_sha256="335e6755…") and lists 2036.
  • With only the PyPI 1.0.0.post1 wheel installed, that snippet raises Forecast does not match expected_sha256 for every year, 2022 included.
  • Vercel puts each main push in production within about 1 minute, with no gate. PyPI gets 1.1.0 about 20 minutes later. #52 is an issue, not a PR, so the mismatch stays live until someone pushes the promotion.
  • This is not the #41 and #50 precedent. In each earlier change to RC (#36, #38, #41), the labelled package carried the pinned content 3d86…; #50 did not change RC. This is the first time that label and content disagree. The real precedent is #36, which shipped new content as local_preview.
  • No test can catch it: test_web_config.py hard-codes the swap, and the e2e spec reads the label from RC.

• Why should address: config drift that every visitor to the public calculator hits from the merge on. Not critical, because the wheel is correct and one exporter run fixes the label.

• Fix:

  1. Run python scripts/export_web_release.py with no flag. This writes 1.1.0 with status local_preview and keeps the 335e… pin; a dry run changes only the two package fields. Bare --check then passes, which also settles MaxGhenis's wording note.
  2. In test_committed_browser_export_matches_canonical_artifact, remove the swap and assert that RC bytes equal encode_config(config).
  3. Change the body's "following #41 and #50" and bare --check text, docs/spm-releases.md:153-155 and docs/rolling-forecasts.md:348-349 to "local preview until promotion".

  #52 step 4 then flips the page to published. Do not export with --published-package-version 1.1.0 here: the site would claim a package that is not yet on PyPI, and for good if the publish job fails.

SA2. The README becomes the permanent 1.1.0 PyPI page, and its install line gives no 2036.

• Where: README.md:43,46-47 (==1.0.0.post1) against README.md:114-115 ("covers 2022–2036"); pyproject.toml:9. Also web/app/layout.jsx:13 and docs/bls-2026-correction.md:140, outside the PR.
• Evidence: the built 1.1.0 METADATA installs 1.0.0.post1 and says "covers 2022–2036". The 1.0.1 page says 2022–2035, so the contradiction is new. pip install spm-calculator==1.0.0.post1 gives 2022-2035, and --year 2036 fails. layout.jsx:13 gives the site's meta and Open Graph description "through 2035" next to a 2036 year menu. bls-2026-correction.md:140 says "now projects 2026–2035".
• Why should address: drift that a user hits, on a page that cannot change after upload. With skip-existing, only a 1.1.1 could correct it.
• Fix: set README.md:43,46,47 to ==1.1.0 (the GitHub README is ahead of PyPI for about 20 minutes, which corrects itself). Change "through 2035" to "through 2036" in layout.jsx:13, and "2026–2035" to "2026–2036" in bls-2026-correction.md:140. docs/index.md, quickstart.md and migration.md can wait for #52.

SA3. The verifier and its tests let four kinds of real change pass. One test module closes all four.

• Where: V:137-160 (_strip_added_year), V:298-350 (check_added_year), V:353-491 (grid, build_receipt), TH:96,189,203,232 (all calculate=False), TH:101-118.

• Evidence. Each mutant went through the normal flow: mutate, reassemble with build_rolling_forecast.py, --check, full-grid verifier writes the receipt, then pytest tests/. Each one passed every existing check.

  • (a) A "2036" key in any year-keyed record. _strip_added_year has no list of records that may grow. ACS price_inputs/observed_annual_values/2036 = 999.0, in an observed-CPI record that ends in 2025, passes the verifier and CI. So do county_assignments/maps/2036 and source_vintages/2036 at the verifier. Inside a retained 2035 scenario entry, only the pinned scenarios digest and the full grid catch it. V:6 and the body ("Any other difference fails") promise more.
  • (b) Retained-year changes inside the rebuilt components. The verifier only requires each component hash to be new. CE 2030 raw_ce_housing_shares × 1.10 and ACS 2030 national_median_gross_rent × 1.10 pass with the same 41 allowed_changes. No test compares a component with retained_bytes(). Both ship in the wheel, and docs/rolling-forecasts.md:89,254 cites them.
  • (c) 2036 values not bound to the method. check_added_year records growth but does not assert it, and the synthetic _extend() copies 2035 thresholds unchanged. CE zero_real 2036 renter × 1.10 passes, and $61,567.78 would ship in place of $55,970.71. Frozen 2036 thresholds, the fiscal-year CBO rate and price_projections["2036"] = 0.05 also pass. Only the 2036 rent indices are bound (tests/test_scientific_refresh.py:143-147).
  • (d) Grid and receipt builder never run in CI. TH covers 72% of V. A grid that compares nothing passes TH. CI coverage measures only spm_calculator, so codecov does not show this. The receipt test reads aaf2d3d4… and 547867b3… from the receipt itself.
  • The committed data is clean: the same 41 paths as the receipt, and 2036 follows the rule to 4.4e-16.

• Why should address: missing tests for the verifier and the 2036 year, and verifier gaps that let a real change pass. Not critical: no value or digest here is wrong.

• Fix: add the module below. At the head: 6 passed in about 9 s, and V coverage rises to 91%. Each mutant fails at least one test:

  • test_retained_content_equals_the_archived_bytes (artifact, CE, ACS) closes (a) and (b).
  • test_added_year_follows_the_documented_rule closes (c); it fails on the CE mutant and on every artifact probe.
  • test_receipt_builder_reproduces_the_committed_receipt and test_calculation_grid_rejects_a_retained_amount_change close (d). The first compares the whole receipt except amount_comparison. The second fails on the grid mutant.

  A full-grid variant of the receipt test also gives a byte-identical receipt (78 s); add it as an opt-in test if you want. The module needs no data change. Optional: make V reject (a) itself with an allow-list of the 21 records that may gain the year. That patch passes TH and rejects the (a) probes, but any edit to V needs a regenerated receipt (audit_script).

tests/test_horizon_bindings.py

"""Bind the 2036 extension to its archived evidence and its documented rule."""

import copy
import hashlib
import itertools
import json

import pytest

from scripts import verify_horizon_extension as verifier
from spm_calculator.forecast_inputs import load_horizon_inputs
from spm_calculator.release import canonical_bytes
from spm_calculator.rolling_forecast import SPMForecast, seal_forecast

ADDED = str(verifier.ADDED_YEAR)
LAST = str(max(verifier.RETAINED_YEARS))
ACS = "spm_calculator/data/current/acs_rolling_forecast.json"
CE = "spm_calculator/data/current/ce_rolling_forecast.json"
CPI_SERIES = ("CUUR0000SA0", "CUUR0000SAA", "CUUR0000SAF", "CUUR0000SAF11",
              "CUUR0000SAH1", "CUUR0000SAH2", "CUUR0000SEED", "CUUR0000SEFV")
CODE = {"scripts/build_acs_forecast.py", "spm_calculator/forecast_inputs.py"}
INPUT_SOURCES = {"sources/id=acs-2021-source-inputs/sha256",
                 "sources/id=forecast-horizon-inputs/sha256"}


def _added(*records):
    return {f"{record}/{ADDED}" for record in records}


# Every path at which a current file may differ from its retained bytes.
EXPECTED_DIFFERENCES = {
    verifier.CURRENT: {
        "assumption_sha256", "content_sha256", "county_assignments/sha256",
        "assumptions/acs/provenance_adaptation",
        "assumptions/acs/relative_index_stabilization/through_year",
        "sources/id=acs-code-identity-adaptation", "sources/id=ce-020/sha256",
        *INPUT_SOURCES,
        *(f"assumptions/acs/generator_identity/{name}" for name in CODE),
        *(f"code_sha256/{name}" for name in verifier.EXTENDED_CODE),
        *(f"component_sha256/{path}" for path in verifier.REBUILT_COMPONENTS),
        *_added(
            "assumptions/acs/nominal_rent_growth_by_year",
            "assumptions/acs/price_growth_by_year",
            "assumptions/acs/price_inputs/forecast_source/annual_growth_by_year",
            "assumptions/acs/puma_vintage_by_spm_year",
            *(f"assumptions/ce/annual_cpi/{s}" for s in CPI_SERIES),
            "assumptions/ce/inflation_rates",
            "assumptions/ce/price_source/annual_growth_by_year",
            "assumptions/price_projections", "county_assignments/year_maps",
            "rent_sensitivity/index_difference_by_year",
            "rent_sensitivity/nominal_rent_growth_by_year",
            "rent_sensitivity/price_growth_by_year",
            "scenarios/ce_trend/years", "scenarios/zero_real/years",
        ),
    },
    CE: {
        "code_sha256/spm_calculator/ce_forecast.py",
        "code_sha256/spm_calculator/forecast_inputs.py",
        "sources/20/sha256",
        *_added(
            *(f"assumptions/annual_cpi/{s}" for s in CPI_SERIES),
            "assumptions/inflation_rates",
            "assumptions/price_source/annual_growth_by_year",
            "scenarios/ce_trend/years", "scenarios/zero_real/years",
        ),
    },
    ACS: {
        "metadata/provenance_adaptation",
        "metadata/relative_index_stabilization/through_year",
        "sources/id=acs-code-identity-adaptation",
        *INPUT_SOURCES,
        *(f"metadata/generator_identity/{name}" for name in CODE),
        *_added(
            "diagnostics_by_year", "geography_by_year", "indices_by_year",
            "metadata/nominal_rent_growth_by_year", "metadata/price_growth_by_year",
            "metadata/price_inputs/forecast_source/annual_growth_by_year",
            "metadata/puma_vintage_by_spm_year",
            "sensitivity/rent_plus_2pp/diagnostics_by_year",
            "sensitivity/rent_plus_2pp/index_difference_by_year",
            "sensitivity/rent_plus_2pp/indices_by_year",
            "sensitivity/rent_plus_2pp/metadata/nominal_rent_growth_by_year",
            "sensitivity/rent_plus_2pp/metadata/price_growth_by_year",
        ),
    },
}


def _by_id(items):
    if all(isinstance(x, dict) and isinstance(x.get("id"), str) for x in items):
        return {f"id={x['id']}": x for x in items}
    return None


def differences(before, after, path=""):
    """Yield each path where two JSON documents differ in exact encoding."""
    if isinstance(before, list) and isinstance(after, list):
        keyed = (_by_id(before), _by_id(after))
        if None not in keyed:
            before, after = keyed
        elif len(before) == len(after):
            before, after = dict(enumerate(before)), dict(enumerate(after))
    if isinstance(before, dict) and isinstance(after, dict):
        for key in sorted(before.keys() | after.keys(), key=str):
            child = f"{path}/{key}" if path else str(key)
            if key not in before or key not in after:
                yield child
            else:
                yield from differences(before[key], after[key], child)
    elif canonical_bytes(before) != canonical_bytes(after):
        yield path


def _current(path):
    return json.loads((verifier.ROOT / path).read_bytes())


@pytest.fixture(scope="module")
def current():
    return _current(verifier.CURRENT)


@pytest.mark.parametrize("path", sorted(EXPECTED_DIFFERENCES))
def test_retained_content_equals_the_archived_bytes(path):
    archived = json.loads(verifier.retained_bytes(path))
    assert set(differences(archived, _current(path))) == EXPECTED_DIFFERENCES[path]


def test_added_year_follows_the_documented_rule(current):
    assumptions, acs = current["assumptions"], current["assumptions"]["acs"]
    pinned = load_horizon_inputs()["prices"]
    levels = assumptions["ce"]["price_source"]["cbo_index_levels"]
    growth = assumptions["price_projections"][ADDED]
    assert levels == pinned["cbo_index_levels"]
    assert growth == levels[ADDED] / levels[LAST] - 1
    assert growth == pinned["annual_growth_by_year"][ADDED]
    for record in (
        assumptions["ce"]["inflation_rates"],
        assumptions["ce"]["price_source"]["annual_growth_by_year"],
        acs["price_growth_by_year"],
        acs["nominal_rent_growth_by_year"],
        acs["price_inputs"]["forecast_source"]["annual_growth_by_year"],
        current["rent_sensitivity"]["price_growth_by_year"],
    ):
        assert record[ADDED] == growth
    sensitivity = current["rent_sensitivity"]["nominal_rent_growth_by_year"]
    assert sensitivity[ADDED] == pytest.approx(growth + 0.02, rel=1e-12)
    for series in assumptions["ce"]["annual_cpi"].values():
        assert series[ADDED] == pytest.approx(series[LAST] * (1 + growth), rel=1e-12)
    vintages = acs["puma_vintage_by_spm_year"]
    assert vintages[ADDED] == vintages[LAST]
    year_maps = current["county_assignments"]["year_maps"]
    assert year_maps[ADDED] == year_maps[LAST]
    for scenario in current["scenarios"].values():
        last, entry = scenario["years"][LAST], scenario["years"][ADDED]
        # zero_real: CPI-U only; ce_trend: CPI-U times the real trend.
        factor = (1 + growth) * (1 + scenario["real_growth_rate"])
        for tenure, value in entry["thresholds"].items():
            expected = last["thresholds"][tenure] * factor
            assert value == pytest.approx(expected, rel=1e-12, abs=0)
        assert entry["housing_shares"] == pytest.approx(
            last["housing_shares"], rel=1e-12, abs=0
        )
        assert entry["rent_indices"] == last["rent_indices"]
        assert entry["geography_by_area"] == last["geography_by_area"]


def test_receipt_builder_reproduces_the_committed_receipt(monkeypatch):
    # Full 2036 grid (150,768 calculations); no retained grid.
    full_grid = verifier._grid

    def added_year_only(forecast, scenario, year):
        if year == verifier.ADDED_YEAR:
            return full_grid(forecast, scenario, year)
        return iter(())

    monkeypatch.setattr(verifier, "_grid", added_year_only)
    receipt = verifier.build_receipt(
        (verifier.ROOT / verifier.BASELINE).read_bytes(),
        (verifier.ROOT / verifier.CURRENT).read_bytes(),
    )
    committed = json.loads((verifier.ROOT / verifier.RECEIPT).read_bytes())
    amounts = receipt["horizon_comparison"].pop("amount_comparison")
    pinned = committed["horizon_comparison"].pop("amount_comparison")
    assert receipt == committed
    for key in ("added_year_calculations", "added_year_sha256_big_endian_float64",
                "area_counts", "fields", "composition_grid", "retained_years",
                "added_year"):
        assert amounts[key] == pinned[key]
    assert amounts["retained_calculations"] == 0


def test_calculation_grid_rejects_a_retained_amount_change(current):
    original = SPMForecast.from_dict(
        json.loads((verifier.ROOT / verifier.BASELINE).read_bytes()),
        expected_sha256=verifier.BASELINE_CONTENT_SHA256,
    )
    changed = copy.deepcopy(current)
    entry = changed["scenarios"]["ce_trend"]["years"]["2022"]
    entry["thresholds"] = {k: v * 1.01 for k, v in entry["thresholds"].items()}
    changed["assumption_sha256"] = hashlib.sha256(
        canonical_bytes(changed["assumptions"])
    ).hexdigest()
    mutant = SPMForecast.from_dict(seal_forecast(changed))
    full_grid = verifier._grid
    with pytest.MonkeyPatch.context() as patch:
        patch.setattr(verifier, "_grid",
                      lambda f, s, y: itertools.islice(full_grid(f, s, y), 3))
        with pytest.raises(ValueError, match="Calculation output changed"):
            verifier.compare_calculations(original, mutant)

SA4. The CE and ACS rebuild in 2d8acf0 is not recorded, and the documented procedure misses two steps that it needed.

• Where: commit 2d8acf0; V:495; docs/rolling-forecasts.md:323-345; docs/spm-releases.md:112-113.
• Evidence:
  • The commit message is a title and a co-author line: no command, cache mode or builder --check output.
  • V:495 writes "scientific_component_rebuild_performed": True as a constant that nothing computes or tests. The raw-source pins are identical to base, and the normalized ACS products stay bound through the unchanged acs_normalized_products.json. Missing: which ACS load path the rebuild used (normalized products or a raw reparse), and whether build_ce_forecast.py --check and build_acs_forecast.py --check passed.
  • The docs give three rebuild commands. 2d8acf0 also needed build_forecast_inputs.py and build_acs_2021_inputs.py first, and no doc names either. The wrong order fails with "Historical county assignment changed", which was DTrim99's first CI failure.
  • No reviewer can rerun the raw-cache steps (about 4.3 GB of ACS PUMS, 11 BLS CE bundles that bls.gov blocks for scripts, no hosted copy). The offline steps all reproduce.
• Why should address: a reproducibility gap; the body's claim "from the 9/9 raw caches" is recorded nowhere durable. Low weight: no value or digest is at risk, and the receipt is outside component_sha256, so the fix moves no digest.
• Fix:
  1. MaxGhenis: post the commands and the ACS cache mode, or the output of build_acs_2021_inputs.py --check, build_ce_forecast.py --check and build_acs_forecast.py --check at the head.
  2. After docs/spm-releases.md:120, document the order: build_forecast_inputs.py, build_acs_2021_inputs.py --cache-dir …, build_ce_forecast.py, build_acs_forecast.py, build_rolling_forecast.py, verify_horizon_extension.py --output …, with each --check form, the cache directories and the environment. Add that 2036 is the last year of the CBO February 2026 file, so 2037 needs a new CBO vintage, which also changes the 2026-2036 rates.
  3. Replace the constant with recorded facts (commands, ACS records source, normalized-products digest, a computed "external source pins unchanged"), or label it operator-attested. Regenerate the receipt in the same pass as any change to V.

Suggestions

G1. Expected message per mutation case. TH:188 uses a bare pytest.raises(ValueError). Add match= per case ("remaining document fields", "Undeclared code identity change", "Undeclared source removal", and so on), so that a regression cannot pass by failing elsewhere.

G2. Verifier command line. --current does not change final_artifact.path (V:505) or the hashes under ROOT (V:483-485, V:510-518). Add a --check mode in place of overwrite plus git diff (docs/spm-releases.md:146-148). retained_bytes() (V:111) ships in neither wheel nor sdist; only the archive bytes do.

G3. Say that 1.1.0 ships #45. #45 changes the public spm_unit_id (foster children by relationship label; an unrelated child under 15 with a linked parent leaves the reference unit). No digest moves, and policyengine-us does not call it. Add a line to the body and the release notes; the repository has no changelog.

G4. Doc drift. docs/spm-releases.md:116-117 leaves out the removed adaptation link and the through_year change. web/e2e/README.md:3 says "through 2035". After SA1, CW:60 still links the preview to PR #36.

G5. One horizon constant. The end year is in at least 8 code places (forecast_inputs.py:24, ce_forecast.py:1052, build_acs_forecast.py:279,511, build_forecast_inputs.py:192,205, build_rolling_forecast.py:309, validate_axiom_integration.py:369) and about 15 test sites. It changes code identities, so do it in a later release.

G6. Skipped 2036 adapter tests. CI installs only .[dev], so test_microcosm_adapter.py and the Axiom [2036-*] cases skip, though the body lists them as moved to 2036. Say so, or run them once by hand. A fast test of build_forecast_inputs.py --check would also help.

G7. Wheel size. The wheel grows 2.49 MB (+33.6%) to 9,880,943 bytes, mostly the horizon archive, which no packaged code reads. It is within PyPI limits. Consider repository-and-sdist-only archives in future.

Investigated and cleared

• Post-merge failure. test_tracked_repository_has_no_stale_pre_repair_markers failed in a merged export only because git grep exits 128 outside a git repository. Its six markers have no hit on 4ff8a8c or the head, and it passes in main CI and in PR CI on the merge ref. Optional follow-up elsewhere: the sdist ships this test, so skip it outside a git work tree.
• Bare --check wording. The body is true as worded; SA1's fix makes the command pass.
• County map. The 2022 and 2023 maps differ only for Sumter County, SC. 2036 resolves like 2035 for all 3,143 counties, a 2022 map fails validation, and 2037 is rejected.
• Stale 2035 bounds. None in spm_calculator/ or scripts/. verify_provenance_equivalence.py:183 keeps 2035 on purpose, because it is pinned 9/9 evidence.
• Component changes outside 2036. Metadata only (seals, code identities, source digests, through_year, the removed adaptation link); none feeds calculate_unit. acs_2021_source_inputs.json changes two digests by deterministic propagation.
• 2036 completeness. Same keys and statuses as 2035, 349 areas, windows 2031Q2-2036Q1 and cohorts 2031-2035, flat area growth (largest deviation 4.4e-14 pp).
• Packaging. 1.1.0 is the right semver step. All four archive files are in the wheel and sdist. uv lock --check passes, and ruff is clean.

Release and downstream

• The merge publishes 1.1.0 with no manual gate. publish.yaml tests on push to main, then uploads with trusted publishing and skip-existing. The pypi environment has no required reviewer or wait timer. The upload comes about 20 minutes after the merge, built from main with #45. A later fix to package content needs a new version.
• The site changes about 1 minute after the merge (SA1).
• Sign-off. MaxGhenis approved at 836762d in a comment (no findings, one wording note). It is not a formal review. SA4 needs MaxGhenis's input, so ask for a re-check after the fixes.
• Publishing cannot break policyengine-us. It pins spm-calculator>=1.0.0,<=1.0.0.post1 (pyproject.toml:27) and locks 1.0.0; policyengine.py pins ==1.0.0.
• After the merge:
  1. #52 (an issue): confirm 1.1.0 on PyPI, years 2022-2036 and content 335e6755…; run export_web_release.py --published-package-version 1.1.0 and --check; push the promotion.
  2. policyengine-us: pin >=1.1.0,<1.2, relock, test_spm_unit_spm_threshold.py:77 to range(2022, 2037), a 2036 household case, docs/spm.md, README.md:15, .github/release-lock.md, and a changelog fragment.
  3. Callers that pass forecast_content_sha256 move to 335e6755… with the pin: the policyengine.py bundle (#618) and the API defaults (policyengine_api/spm.py:70,125,246,283; policyengine-sim-api spm.py:107,139). A caller still on 3d86… fails once policyengine-us resolves 1.1.0.
  4. policyengine-api#3926: the 2036 hold releases after these.

Validation Summary

Check	Result
Tests at head	527 passed, 83 skipped, 0 failed; TH 16/16
Merged tree	548 passed, 83 skipped, 1 environmental failure (cleared)
CI	All checks pass at 836762d
Full-grid verifier	Exit 0 in 65 s; receipt byte-identical
Digests	335e6755…, 53d2b4b3…, 6ef10c97… recomputed without the library; all places agree
Retained grid	2,110,752 calculations cmp-identical base against head; aaf2d3d4… three ways
2036	150,768 calculations, 547867b3…; 349 areas; growth follows the rule to 4.4e-16
CBO input	Workbook and pinned CSV agree; build_forecast_inputs.py --check passes
Offline rebuild	build_rolling_forecast.py --check passes; raw-cache builders not rerun (SA4)
Verifier gaps	4 mutant classes pass every existing check; the proposed module catches each (SA3)
Wheel	9,880,943 bytes, 72 files, 53d2b4b3…, 2022-2036, 4 archive files; merged build adds #45
Web after merge	"Published 1.0.0.post1" with content 335e6755… (SA1)
Downstream	policyengine-us and policyengine.py pins exclude 1.1.0

Review Severity: REQUEST_CHANGES

Four should-address items are open (SA1 web label, SA2 README, SA3 verifier tests, SA4 rebuild record). There is no critical item: every value and digest in the body reproduces.

Next Steps

1. Regenerate the web export as local_preview 1.1.0, change the web config test, and fix the SA1 text.
2. Fix the README install lines, layout.jsx:13 and bls-2026-correction.md:140 (SA2).
3. Add tests/test_horizon_bindings.py (SA3), and run the full suite.
4. Get the rebuild record from MaxGhenis, then document the order and replace the constant flag (SA4). If V or the receipt changes, regenerate the receipt once with the full grid, and confirm that 335e6755…, 53d2b4b3… and 6ef10c97… do not move.
5. Update the body for G3 and G6. Take the other suggestions that you agree with.
6. Run ruff format, push and wait for CI. Merge when someone can run the #52 promotion right after the upload.

[DTrim99]

@PavelMakarchuk round 1 is addressed in ea3362c:

• SA1: the browser export is a local_preview of 1.1.0 again, as Replace legacy SPM calculations with canonical rolling CE and ACS forecasts #36 shipped new content. A bare --check passes. test_web_config.py compares the export with a fresh one, and test_release.py reproduces either status. The docs say "local preview until promotion", and the preview link points here.
• SA2: the README install lines target ==1.1.0. layout.jsx and bls-2026-correction.md now say 2036.
• SA3: your tests/test_horizon_bindings.py is added as written (ruff-formatted). The mutation cases now name their messages (G1).
• SA4: scientific_component_rebuild_performed: true is replaced by computed facts, and the verifier requires them: all 56 external source pins and the normalized ACS products (49b12d77…) are unchanged. The commands, the ACS cache mode and each builder's --check result are labelled operator-attested, with this PR as the record. docs/spm-releases.md gives the full rebuild order, the caches and the CBO 2037 note. The verifier gains --check and drops --original/--current (G2). The receipt is regenerated with the full grid; only audit_script and the rebuild block change, and 335e6755…, 53d2b4b3… and 6ef10c97… don't move.
• G3, G4, G6: Preserve parent links and foster age limits in SPM unit assembly #45 is noted in the description, along with the doc drift and the tests that skip without microcosm or AXIOM_CORE_BIN. I left G5 (the single horizon constant changes code identities) and G7 for a later release.

@MaxGhenis, for SA4 the receipt points at this PR for the rebuild record. Could you post the commands you ran in 2d8acf0, the ACS cache mode (normalized products or a raw reparse), and the output of build_acs_2021_inputs.py --check, build_ce_forecast.py --check and build_acs_forecast.py --check at the head?

Ready for round 2.

[PavelMakarchuk]

Release Review

PR #51, "Extend the canonical forecast horizon to 2036" (DTrim99; CE and ACS rebuild commit 2d8acf0 by MaxGhenis). Round 2.

IDs: C = critical, SA = should address, G = suggestion. Round-1 IDs keep their numbers. New suggestions start at G8.

Paths: V = scripts/verify_horizon_extension.py, TB = tests/test_horizon_bindings.py, TH = tests/test_horizon_extension.py, REL = docs/spm-releases.md, RC = web/public/data/release_config.json, CW = web/src/components/CalculatorWorkbench.jsx. Line numbers refer to the reviewed head.

Source Documents

• Reviewed head SHA: ea3362c
• Mode: incremental from 836762d
• Scope: commit ea3362c (15 files, +434 / −74) and every round-1 item (SA1–SA4, G1–G7).
• Round 1 against the head: type-strict JSON walks of the receipt and RC, and a file-by-file diff of both trees.
• Merge base 13ab3014d0 against the head: all digests, the retained grid and the new rebuild facts, recomputed without spm_calculator.
• Wheels: 1.1.0 built offline from the head and from the head plus main.
• PR thread: DTrim99's round-1 reply (2026-10-27, 15:01 UTC) and its request to MaxGhenis. At review time the head is still ea3362c, and MaxGhenis has not replied (last comment 2026-10-22).
• Downstream: policyengine-us main pyproject.toml, issue #52, policyengine-api#3926.

Branch Status

• 9 ahead of and 1 behind main. The one commit is still 4ff8a8c (#45). main has not moved since round 1. No file overlap; GitHub reports the PR as mergeable.
• CI at ea3362c: 14 pass, 1 skipping (existing-deployment). This covers Python 3.9–3.14, web, docs, Vercel, codecov, and browser QA standalone and under the subpath (15/15 on merge ref baff205).

What round 2 fixed

• No value moved. In spm_calculator/data, only the receipt changes, and in web/public/data, only RC. The artifact, both components, the inputs, the canonical downloads and the provenance archive are byte-identical to round 1.
• Identities. 335e6755…, 53d2b4b3…, 6ef10c97… and the county digest 7bcfd3f2… recompute from the JSON without the library. The retained grid (2,110,752 calculations, aaf2d3d4…) and the 2036 grid (150,768, 547867b3…) reproduce with a raw re-implementation.
• Receipt. Exactly three paths change: audit_script/sha256 (now the hash of the head V), the removed constant flag and the new scientific_component_rebuild block. The amount digests, the artifact references and the predecessor link b0a27cd8… do not change.
• Verifier. V --check passes at the head (exit 0, about 1.5 minutes with the full grid). It fails with exit 1 on a tampered receipt.
• Tests. The full suite gives 533 passed, 83 skipped, 0 failed (round 1: 527 passed). The 6 new tests are TB. ruff format --check and ruff check are clean.

Round-1 status

ID	Status	Evidence
SA1	RESOLVED	RC is local_preview 1.1.0 with pin 335e… and years 2022–2036. Only packageVersion and the four packageDistribution fields changed, and a fresh export is byte-equal. The card now says "Reproduce with local build / development preview 1.1.0" and shows no PyPI link. With the 1.1.0 wheel built from the head plus main, the snippet passes in all 270 runs (15 years, 2 scenarios, 3 tenures, 3 areas per year). A bare export_web_release.py --check passes. --published-package-version 1.1.0 --check fails, as it must until #52. test_web_config.py compares the committed bytes with build_config(), and test_release.py covers both statuses. The docs and the body say "local preview until promotion".
SA2	RESOLVED	README.md:43,46,47 give ==1.1.0. The built 1.1.0 METADATA installs 1.1.0 and says 2022–2036. layout.jsx:13 says "through 2036", and bls-2026-correction.md:140 says "2026–2036". The remaining 1.0.0.post1 hits are in docs/index.md, quickstart.md and migration.md (deferred to #52, as round 1 accepted; see G9), in dated logs, in retained-range text and in tests on the synthetic 2022–2035 fixture. None is stale.
SA3	RESOLVED	TB equals the round-1 module after ruff format. Each round-1 mutant class went through your full flow again (mutate, reassemble, --check, regenerate the receipt and the export, full CI command). Each one now fails 1 to 5 CI tests: (a) a 2036 key in observed CPI, (b) CE and ACS 2030 values × 1.10, (c) CE zero_real 2036 renter × 1.10, (d) a grid that compares nothing. All 12 artifact probes from round 1 fail a CI test. Residual: V alone, without the CI tests, still accepts the (a) and (c) probes, so V:5-6 describes V plus the tests. No action is needed while CI is the gate.
SA4	RESOLVED in the PR; MaxGhenis's record is a merge condition	See below.
G1	RESOLVED	All 12 cases name a message. A spy on the catch-all shows that each case fails for its own mutated path and no other. Seven cases share the generic text (G14).
G2	RESOLVED	--output and --check are mutually exclusive, --check --no-calculate is refused, and --original/--current are gone. retained_bytes() still lives in scripts/ and ships in neither the wheel nor the sdist. This is harmless.
G3	RESOLVED	The body says that 1.1.0 ships #45. The merged wheel contains the units.py from 4ff8a8c, and no digest moves. Optional: say it in the 1.1.0 section of REL too.
G4	PARTIAL	REL:116-122 now lists the stabilization disclosure and the removed adaptation link, and web/e2e/README.md:3 says "through 2036". PREVIEW_URL now opens #51, but the visible label still says "calculator PR36" (G8).
G5	DECLINED, defensible	The end year sits in at least 8 hashed code identities. One constant would change 335e… and every downstream pin, on the critical path of the 2036 extension. For now, list the sites in REL (G13).
G6	RESOLVED	The body names the skipped tests and says that their 2036 coverage is unverified. Nit: the body says that they need "microcosm and pandas". pandas is a core dependency (pyproject.toml:28), so only microcosm is missing. The optional fast test of build_forecast_inputs.py --check was not added; the check passes offline in 1 s.
G7	DECLINED, defensible	The wheel is about 9.9 MB, well within PyPI limits. Removing the archive from the wheel changes no digest, but the sdist and the tests need it, so a later release is the right place.

SA4 in detail. Round 1 asked for the rebuild order, a computed record in place of the constant, and MaxGhenis's commands. Everything that you can record is now in the PR:

• rebuild_facts() (V:504-528) replaces the constant. It requires that the 56 non-rebuilt source pins and the normalized ACS products digest are unchanged. The receipt writes true only after both checks pass.
• The facts are true. The sources go from 60 to 59: the adaptation link is removed, 3 rebuilt inputs change, and 56 do not change. acs_normalized_products.json hashes to 49b12d77… at the merge base and at the head.
• A changed pin or ACS product fails. A rebuild from a different CE bundle makes the verifier fail and fails 4 CI tests. A rebuild from different normalized products fails 10 CI tests. A changed products file with no rebuild fails 7 CI tests. (The failure message comes from the catch-all, not from rebuild_facts; see G11.)
• REL:133-159 gives the full order, the cache directories and their defaults (they match the scripts), uv sync --frozen, the Windows line-ending note, why build_acs_2021_inputs.py comes first, the "Historical county assignment changed" failure and the CBO 2037 note. docs/rolling-forecasts.md:337-348 agrees.

What is left is MaxGhenis's operator record. It needs no commit, so it is a merge condition and not a code change. Doc improvements for a second operator are in G13.

Critical (Must Fix)

None. No value, digest or receipt field moved except as declared, and no test or CI check fails.

Should Address

None open. SA1, SA2 and SA3 are resolved. SA4 is resolved in the PR, and its last part is in Merge conditions.

Suggestions

G8. The preview link label names PR36 but opens #51. CW:234 gives "Preview Python API: calculator PR36", and PREVIEW_URL (CW:60) now opens /pull/51. Every visitor sees this label on the Python card from the merge until the #52 promotion. The tests check only the href. Fix: one line, for example "Preview Python API: release PR 51". Optionally, assert the link name in CalculatorWorkbench.test.jsx:985-995.

G9. The #52 checklist misses two steps.

1. test_committed_browser_export_matches_canonical_artifact now compares RC with build_config(), which gives local_preview. The promotion export fails it, so the promotion must change the test to build_config(published_package_version="1.1.0"), as #38 and #41 did. CI would catch this step.
2. #52 step 4 names docs/index.md, but not docs/quickstart.md:6,9,10 (and its 2035 examples at :86, :87, :136) or docs/migration.md:9. No check catches this step. docs/index.md:8-9 now installs ==1.0.0.post1 next to "through 2036", which is the round-1 README contradiction on GitHub (the docs are not deployed).

Add both steps to #52.

G10. CI does not bind part of amount_comparison. TB:208-221, TH:240-251. If only original_sha256_big_endian_float64 changes in the committed receipt, the receipt says that the retained digests differ while max_absolute_difference is 0. All CI tests still pass, and only the manual V --check catches it. No CI test compares original_sha256…, exact_field_comparisons, all_result_fields_equal_except, scenario_count, resources or ordering. Fix: add these lines to test_receipt_builder_reproduces_the_committed_receipt. They change no data and no receipt.

    assert pinned["original_sha256_big_endian_float64"] == pinned["current_sha256_big_endian_float64"]
    assert pinned["exact_field_comparisons"] == pinned["retained_calculations"] * len(pinned["fields"])
    for key in ("all_result_fields_equal_except", "scenario_count", "resources", "ordering", "max_absolute_difference"):
        assert amounts[key] == pinned[key]

G11. The rebuild checks cannot fire through build_receipt, and no test covers them. rebuild_facts (V:511-520) runs after compare_forecasts (V:533). That step already requires the same fields to be byte-equal, so it rejects every such change first with the catch-all message. If someone deletes the two _require calls, the block becomes a constant true again and every test still passes. Fix, either:

• (a) add a test that calls verifier.rebuild_facts(baseline, mutated) for a changed ce-000 pin and for a changed normalized-products digest, with match="External source pins changed" and match="Normalized ACS products changed"; or
• (b) call rebuild_facts before compare_forecasts. This edits V, so the receipt must be regenerated (audit_script only).

Option (a) changes no data.

G12. The rebuild wording says more than the receipt shows. REL:124-126 says that the rebuild "used the same external source pins and normalized ACS products as 1.0.x". 15 of the 56 "external" pins are package files under spm_calculator/ (the CBO CSV, the Census workbooks, the crosswalks and the products manifest itself). Also, build_acs_forecast.py:570-578 records the manifest digest in both ACS load paths. So "normalized ACS products unchanged" proves that the manifest did not change, not which path the rebuild took. Fix: write "It records that the rebuild kept all 56 source pins (raw caches and packaged inputs) and the normalized-products manifest (49b12d77…) of 1.0.x. Whether the ACS build reused the retained normalized products or reparsed raw PUMS is attested by the operator …". The receipt keys can stay.

G13. The rebuild section does not yet let a second person start from nothing. REL:133-159. Add:

1. One paragraph: the url and cache_path of each raw file are in the sources of the CE and ACS components; bls.gov blocks scripted downloads of the 11 CE bundles; ACS PUMS is about 4.3 GB.
2. That without the retained normalized/*.npz files the ACS builder reparses raw PUMS, and whether that path reproduces the committed bytes. MaxGhenis's cache-mode answer settles this.
3. The two --check forms in full, for example build_ce_forecast.py --cache-dir … --output spm_calculator/data/current/ce_rolling_forecast.json --check, and the same for ACS.
4. For a future horizon: the code sites from G5, the ADDED_YEAR, retained-year and baseline pins in V, and the new provenance archive.

G14. The catch-all message names no path. V:110-114, V:296; 7 of the 12 cases in TH:132-150 match on it. Each case is correct today, but a future regression could pass a case through a different difference, and the message would not show it. Fix: name the first differing path in the message, and match on the mutated path in TH (for example match="thresholds/renter"). This edits V, so a later release is a good place for it.

Merge conditions

1. MaxGhenis posts the rebuild record on #51. It must give the commands run for 2d8acf0, the ACS cache mode (retained normalized products or raw reparse), and the output of build_acs_2021_inputs.py --check, build_ce_forecast.py … --check and build_acs_forecast.py … --check at the final head. The receipt names PolicyEngine/spm-calculator#51 as its operator record (V:86, V:527), and the receipt ships in the 1.1.0 wheel. If the merge comes first, the published receipt cites a thread that does not contain the record. A comment is enough, because the receipt cites the PR and not a comment. A raw reparse keeps the receipt and the docs correct, but the record must say so. If a --check fails or contradicts the receipt, do not merge; ask for a re-review.
2. MaxGhenis signs off on the final head with a formal review. The merge publishes 1.1.0 to PyPI, and that upload cannot be replaced. The round-1 approval at 836762d was a comment, not a review.
3. Merge only when someone can run the #52 promotion right after the PyPI upload (about 20 minutes after the merge).
4. Keep the 2036 hold in policyengine-api#3926 until 1.1.0 is on PyPI with content 335e6755…, the #52 promotion is live, and policyengine-us and the policyengine.py bundle have adopted the new pin.

Release and downstream

• The merge publishes 1.1.0 with no manual gate. publish.yaml uploads on push to main with skip-existing, and the pypi environment has no required reviewer. The wheel is built from main, so it also ships #45.
• The site changes about 1 minute after the merge. It now shows a local preview of 1.1.0 with pin 335e…, which the 1.1.0 package loads. For about 20 minutes, 1.1.0 is not yet on PyPI. The card labels it as a preview and gives no PyPI link, so this is correct.
• Publishing cannot break policyengine-us. Its main still pins spm-calculator>=1.0.0,<=1.0.0.post1 (pyproject.toml:27, checked at this review).
• After the merge:
  1. #52: confirm 1.1.0 on PyPI with years 2022–2036 and content 335e6755…, run the promotion export and --check, apply the two G9 steps, and push.
  2. policyengine-us: pin >=1.1.0,<1.2, relock, extend test_spm_unit_spm_threshold.py:77 to range(2022, 2037), add a 2036 household case, update the docs, and add a changelog fragment.
  3. Callers that pass forecast_content_sha256 move to 335e6755… with the pin: the policyengine.py bundle (#618), the policyengine-api defaults and policyengine-sim-api spm.py. A caller still on 3d86… fails once policyengine-us resolves 1.1.0.
  4. policyengine-api#3926: the 2036 hold releases after these.

Validation Summary

Check	Result
Tests at head	533 passed, 83 skipped, 0 failed; TB and TH 22/22
CI	14 pass, 1 skipping at ea3362c; browser QA 15/15 on the merge ref
ruff	Format and lint clean; TB equals the round-1 module after formatting
Data, round 1 → head	Only the receipt and RC change; all other data files byte-identical
Digests	335e6755…, 53d2b4b3…, 6ef10c97…, 7bcfd3f2… recomputed without the library
Grids	Retained 2,110,752, aaf2d3d4…; 2036 150,768, 547867b3…
Receipt, round 1 → head	3 paths: audit_script, the old flag removed, the rebuild block added
V --check	Exit 0 at head (about 1.5 minutes); exit 1 on a tampered receipt
Round-1 mutants	Each class fails 1–5 CI tests (round 1: three classes passed all CI)
Artifact probes	12 of 12 fail a CI test; V alone still accepts them
Rebuild facts	56 pins and 49b12d77… unchanged; changed pins or products fail the verifier or 4–10 CI tests
Offline builders	build_forecast_inputs.py --check and build_rolling_forecast.py --check pass; raw-cache builders wait on MaxGhenis
Web	Local preview 1.1.0; fresh export byte-equal; bare --check passes; snippet passes 270 of 270 runs with the 1.1.0 wheel
Wheel	72 files; merged build 9,881,340 bytes; receipt byte-equal, names #51 as the operator record
README / METADATA	==1.1.0, 2022–2036
Downstream	policyengine-us main pins <=1.0.0.post1
Operator record	Not posted at review time

Review Severity: APPROVE

No critical or should-address item is open; MaxGhenis's rebuild record and formal sign-off are merge conditions, not code changes.

Next Steps

1. MaxGhenis: post the rebuild record on #51 (merge condition 1).
2. Change the G8 label and add the G9 steps to #52.
3. If you want, add the G10 assertions and the G11 option (a) test. Both are test-only and change no data.
4. Fix the G6 nit in the body, and make the G12 and G13 doc edits.
5. Leave G14 and G11 option (b) for a later release, because they edit V. If you do edit V now, regenerate the receipt once with --output, run --check, and confirm that 335e6755…, 53d2b4b3… and 6ef10c97… do not move.
6. Run ruff format, push and wait for CI.
7. MaxGhenis: review and approve the final head. Merge when someone can run the #52 promotion right after the upload.

[DTrim99]

@MaxGhenis Pavel approved round 2. His remaining merge conditions need you:

1. The rebuild record for 2d8acf0. The receipt in the 1.1.0 wheel cites this PR as its operator record, so it needs to be here before the merge. Please post:

   • the commands you ran;
   • the ACS cache mode: the retained normalized/*.npz products, or a raw PUMS reparse;
   • the output of these three checks at the final head (b145fcc):

   python scripts/build_acs_2021_inputs.py --cache-dir <acs-pums cache> --check
   python scripts/build_ce_forecast.py --cache-dir <ce-pumd cache> --output spm_calculator/data/current/ce_rolling_forecast.json --check
   python scripts/build_acs_forecast.py --cache-dir <acs-pums cache> --output spm_calculator/data/current/acs_rolling_forecast.json --check

   If a check fails or contradicts the receipt, please don't merge; we'll ask Pavel to re-review.

2. A formal GitHub review (approve) on the final head. Your earlier approval was a comment, and the merge publishes 1.1.0 to PyPI, which can't be replaced.

3. Merge timing. Merge only when the Release spm-calculator 1.1.0 with the 2022–2036 forecast #52 promotion can run right after the PyPI upload, about 20 minutes after the merge. The promotion branch is ready, with Pavel's G9 steps included. If you tell me when you plan to merge, I'll push it as soon as 1.1.0 appears.

b145fcc also takes Pavel's round-2 suggestions:

• G8: the preview label;
• G10 and G11: the receipt test bindings;
• G12 and G13: the rebuild docs, including the full --check forms above.

G14 waits for a later release, because it edits the verifier and would need a regenerated receipt.

[MaxGhenis]

Rebuild record for 2d8acf0 and the three checks at b145fcc (US + core hub, an agent, posted from Max's account)

@DTrim99, this answers item 1 of your 10/28 comment. The hub ran the component rebuild that produced 2d8acf0, from the 9/9 raw caches. Items 2 and 3 are Max's call, and the hub has asked him.

Environment for 2d8acf0. Python 3.13.9, numpy 2.4.1, pandas 3.0.0, pytest 9.0.2. The rebuild ran uv sync --locked --extra dev --extra ce, with five thread caps of 4 and core.autocrlf=false, from base 5045c41. An earlier attempt under Python 3.11 stayed unpushed: main's pre-extension CE builder reproduces the 9/9 file byte for byte under 3.12 and 3.13, but not under 3.11.

Commands, run in order from the repository root. Each exited 0.

python scripts/build_acs_2021_inputs.py --cache-dir <acs-pums cache>
python scripts/build_ce_forecast.py --cache-dir <ce-pumd cache> --output spm_calculator/data/current/ce_rolling_forecast.json
python scripts/build_acs_forecast.py --cache-dir <acs-pums cache> --output spm_calculator/data/current/acs_rolling_forecast.json
python scripts/build_rolling_forecast.py

The same four commands with --check then matched byte for byte. Six suites passed, 81 tests in all: test_horizon_extension, test_rolling_forecast_artifact, test_scientific_refresh, test_forecast_inputs, test_provenance_adaptation and test_provenance_equivalence.

ACS cache mode: the retained normalized/*.npz products, not a raw PUMS reparse. load_pums_records first checks each raw csv_hus.zip against its pinned SHA-256. _pinned_normalized then loads the retained product when every input in acs_normalized_products.json agrees. I called it against the same cache at b145fcc, and it returned the pinned product for all four vintages:

Vintage	Records
2021	616,858
2022	612,666
2023	607,890
2024	604,668

The three checks at b145fcc, run today with Python 3.13.15, numpy 2.4.1, pandas 3.0.0 and uv sync --frozen:

Check	Exit	Output
build_acs_2021_inputs.py --cache-dir <acs-pums cache> --check	0	1bd9a40d7d56895ded24424a0fd916aed8752afea45ca132044ae6696c330e56, the same digest as the 2d8acf0 run
build_ce_forecast.py --cache-dir <ce-pumd cache> --output spm_calculator/data/current/ce_rolling_forecast.json --check	0	"checked": true
build_acs_forecast.py --cache-dir <acs-pums cache> --output spm_calculator/data/current/acs_rolling_forecast.json --check	0	ACS artifact matches pinned offline rebuild

Each script rebuilds its artifact and raises on any byte difference from the committed file, so exit 0 means an exact match. The checks left the working tree unchanged. Between 2d8acf0 and b145fcc, no builder script, spm_calculator module or forecast input changed. The range touches only docs, the verifier, the receipt, tests and web files.

Against the receipt. scientific_refresh_receipt.json at b145fcc lists 1bd9a40d…6c330e56 under rebuilt_components for acs_2021_source_inputs.json, the digest the first check prints. It also records acs_normalized_products_unchanged: true (49b12d77…), which matches the cache mode above.

[MaxGhenis]

Approving b145fcc for the merge. Max asked the US + core hub (an agent) to post this review from his account.

Pavel's merge conditions:

1. The rebuild record is above (#51 (comment)). All three builder --check commands exit 0 at b145fcc, and the ACS 2021 digest matches the receipt's rebuilt_components entry.
2. This review.
3. The merge waits for a time when David can run the #52 promotion right after 1.1.0 reaches PyPI.

Since Pavel's round-2 approval at ea3362c, one commit (b145fcc) changes docs/spm-releases.md, tests/test_horizon_bindings.py, and the preview label and its web test. It changes no data, builder, verifier or receipt. CI passed at b145fcc.

🤖 Generated with Claude Code