Beta deploy verification fails on three US simulations, blocking prod deploys since 9 Oct #747

Activity

1. MaxGhenis commented on Oct 12, 2026

   

   MaxGhenis

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Taking this (W4 owns the sim-api deploy).

   Cause. I ran the failing request (country: us, region: state/ut, the refundable CTC reform, time_period: 2026) directly on the staging policyengine-simulation-py6-2-3 worker with the Modal SDK. It raises:

   ValueError: Cannot map stored 'would_claim_wic' onto 'takes_up_wic_if_eligible' for 2024: the stored column has missing values.
     policyengine/tax_benefit_models/us/legacy_inputs.py, _live_values (via apply_legacy_input_renames)
     policyengine/tax_benefit_models/us/datasets.py, create_datasets / ensure_datasets
     policyengine_simulation_executor/simulation_runtime.py, _load_dataset
   

   So the worker fails while building the per-region US dataset, in policyengine 6.2.3's legacy-input rename. That is why the national default-model test passes and the state, region and budget-window tests fail.

   Fix. It is not in policyengine-us 2.2.1 or the populace data, and not in sim-api's code. policyengine.py fixed it on 9 Oct in PolicyEngine/policyengine.py 48f3a784 and 4d2541d2: missing ACS WIC decisions are assumed claimed, and missing values anywhere else are still refused. The fix first shipped in policyengine 6.2.5. sim-api main still pins 6.2.3.

   The updater's PR to 6.2.7, #746, carries the fix. It also moves:

   • UK from policyengine-uk 2.102.3 to 2.125.1;
   • UK data to 1.58.0.

   Its checks are green and an independent review is running. I'll merge it when that agrees, watch the deploy, and close this issue only once beta verification passes and prod serves the new bundle. The US verification tests stay as they are.
2. MaxGhenis

   self-assigned this

   on Oct 12, 2026
3. MaxGhenis commented on Oct 14, 2026

   

   MaxGhenis

   on Oct 14, 2026

   ContributorAuthor

   More actions

   Fixed by policyengine 6.3.4, deployed to production on 14 October.

   • Cause: from policyengine 6.2.2, US state and congressional-district regions used the ACS-local Populace dataset. Its SPM units lack a source-backed independent-minor role, so canonical SPM refused them (SPM_COMPOSITION_REQUIRED).
   • Interim fix: Temporarily filter US state and district regions from the national dataset policyengine.py#604 points those regions back at the certified national file, as 6.2.1 did. Max approved it as d1482. It was released in policyengine 6.3.4.
   • Permanent fix: ACS local release: supply source-backed SPM independence (is_spm_independent_minor_role) so every SPM unit has a classified adult microcosm#1263, which certifies an ACS-local release in which every SPM unit has a classified adult.

   Production evidence (run 38545891000, on the #797 merge, da2e7f5: success at 12:17Z):

   • Beta "Verify deployed three-service stack" passed at 10:20Z. That includes the three tests that failed here: test_calculate_us_state_region_model and test_calculate_specific_model (both state/ut), and test_budget_window_multi_year_batch_completes. All three are beta_only, so prod does not run them.
   • Prod passed the stack verification (12:02Z), the authenticated deployment tests (12:10Z) and promotion (12:15Z).
   • Production /versions: policyengine latest 6.3.4, US 2.2.1, UK 2.126.12.

   A per-release numerical sentinel against production is being added in #743 and #744.
4. MaxGhenis

   closed this as completedon Oct 14, 2026