Add a reusable unlisted-input sweep and keep local taxes and credits out of the state outputs

[MaxGhenis]

Tooling only: this PR changes no reference, exclusion, snapshot or published number.

What it adds

• policybench unlisted-input-sweep (policybench/unlisted_input_sweep.py). Given a reference system (a fix module, e.g. reference_audit/2026-09-28/fixes/latest_final.py) and a frozen run's scenarios.csv, references and exclusions, it:

  • rebuilds every household as the reference builder did and checks the result against every scored published reference;
  • recomputes every output with each registered engine estimate of an unlisted input replaced by the prompt's literal reading (unlisted numbers 0, unlisted facts false) or a documented alternative;
  • reports every output that moves by more than $1 or flips a flag, as scored, acknowledged, prompt_rules_out, excluded_same_input or excluded_other_reason.

  It builds the system once per worker process (spawn ProcessPoolExecutor), runs the longest households first, and skips a reading without simulating when it sets every input to the value the engine already computes. Exit 1 means the sweep can't be trusted (baseline mismatch, a local entry nonzero, an unconverged fixed point). Exit 2 means --strict found a scored move with no exclusion or --acknowledged entry.

• Registry (ESTIMATES). Each entry gives the engine behavior, why no prompt lists the input, where it was found, and its readings:

  Estimate	Readings
  state_withheld_income_tax	zero (literal); liability and net (fixed points, as in Audit the state income tax withheld in federal SALT; propose three exclusions #191)
  local_sales_tax (20% of the state table amount)	zero
  medicare_part_b_premium (takes_up_medicare_if_eligible=True)	zero
  county (first county of the state: Allegany for MD)	every county of the household's state
  weekly_hours_worked_before_lsr (r14)	40 hours for people with none stated
  mortgage origination year (0 takes the $750,000 cap)	pre-TCJA
  months_receiving_social_security_disability	24 months for SSDI recipients
  meets_ssi_disability_criteria	true for people listed as disabled

  Also one reading that applies every literal reading at once.

• policybench/output_scope.py: the adapter that keeps local taxes and credits out of the state outputs. It removes:

  • nyc_income_tax_before_refundable_credits (and Maryland county tax) from the state income tax list;
  • nyc_refundable_credits from state refundable credits;
  • San Francisco's ca_sf_wftc from California's refundable list.

  The sweep composes the adapter and checks every entry it removes is zero for every household. A CI test reads the installed engine's YAML lists and fails if an upgrade adds a local entry the adapter does not name.

• docs/runbook.md §2b, Reference-Build Gates: newest release, reviewed moves, the sweep (--strict), output scope.

Results on the frozen run (policyengine-us 2.15.17, latest_final + adapter)

100 households, 1,984 outputs, 9,165 simulations, 23 minutes on 12 workers.

Gates

• The baseline reproduces all 1,928 scored references.
• The three removed local entries are zero in every household.
• Every fixed point converged.

Scored outputs that move (4). These are exactly the open proposals:

Output	Reference	Under the readings	Readings	Proposal
022 federal	11,113.57	11,895.23 to 12,234.66	SALT withholding; all literal	#191 (d963)
081 federal	26,932.27	26,991.31 to 28,763.81	SALT withholding	#191 (d963)
114 federal	10,729.61	11,137.31 to 11,969.80	SALT withholding, Part B; all literal	#191 (d963)
114 state (VA)	3,514.15	3,654.15 to 4,127.67	Part B; all literal	#193

prompt_rules_out (5). NY households 004, 071, 082, 104 and 118 owe NYC tax (or get NYC credits) in the five NYC counties. That moves local_income_tax. Living in NYC is an unlisted household fact, which the prompt's rule makes false, so these moves are reported but not gated.

Already excluded (14): 10 for the same input, 4 for another reason. The four other-reason outputs are new dependencies a release should note on their records:

• 056 SNAP is 0 in Morris County, NJ.
• 118 SNAP is 3,276 in Nassau/Suffolk and 3,540 in NYC.
• 120 federal is 40,192.03 with a pre-TCJA mortgage.
• 078 federal moves under the SALT withholding readings (already in Audit the state income tax withheld in federal SALT; propose three exclusions #191's notes).

Verification

• Differential against the 2026-10-05 audit's independent scripts (Audit the state income tax withheld in federal SALT; propose three exclusions #191's sweep_salt_withholding.csv and variants.json):

  • max |difference| 0.0 on all 1,984 outputs for the baseline and for the SALT zero, liability and net readings;
  • 0.0 on the 100 fixed-point amounts and on the 48 Maryland county federal values;
  • 6e-14 for local sales tax and Part B at zero.

• Mechanism audit. One Claude Code workflow agent per claim group checked every claim in the registry against the engine source. It corrected five descriptions:

  • Maryland county tax reaches SALT only through md_withheld_income_tax's county estimate;
  • the Part B premium is net of MSP;
  • the local sales tax file path;
  • TANF readers of weekly hours;
  • the rename's provenance.

  It also found that the pre-TCJA shortcut was not exact for negative mortgage interest; it now tests the deductible share.

• Bug caught by the slow real-engine test. policyengine-us applies a reform more than once, which emptied the record of what the adapter removed, so the scope check checked nothing. Fixed in 38b7d87 (with a test that applies the composed reform twice).

• Tests

  • New tests: 59 fast and 1 slow. The fast tests run on a fake engine whose toy formulas mirror the engine's SALT, county, Medicare and SNAP paths.
  • Full fast suite: 1,622 passed, 8 skipped.
  • The slow test reproduces the recorded weekly-hours and SSDI exclusions on the real engine.

Invariants (property-tested with Hypothesis)

• Moves are symmetric and never reflexive. An amount moves exactly when |Δ| > $1, and a flag exactly when it flips.
• Every move row is a real move, and its status is scored exactly when the output has no exclusion record.
• Applying an override is pure and idempotent, and merging two overrides is commutative when it succeeds. This property found that 0.0 and False merged silently; fixed.
• The SALT amount paid is the same with and without the adapter, for any component amounts. This is a differential across two system configurations.
• At a converged fixed point, re-simulating at the reported amount reproduces it within $0.005.
• Scoping a list keeps every state entry in order, removes every local one, and is idempotent.

For the next release (not done here)

• Notes on the four excluded_other_reason records above.
• Once d972 is ruled, register Audit optional employer pass-through in the payroll tax references; propose excluding four outputs #194's optional employer pass-through as an estimate.

Run outputs: results/local/unlisted_input_sweep_20261005/ on the build machine (report, moves, values, summary, differential).

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 5, 2026 6:39pm UTC

[MaxGhenis]

Session hand-off at head 4485065 (not ready to merge yet).

Done

• The independent review (Opus 5.5 on Subfleet, results/local/unlisted_input_sweep_20261005/review_pr196.md) requested changes. 4485065 addresses them:
  • B1 (blocking): locality moves are now attributed per output, using a rerun with the switched-on flags held off. NY SNAP's region moves stay scored. A new slow real-engine test covers scenario_118.
  • N1 to N10, including a Medicare not-enrolled reading, the nonrefundable lists in the CI guard, the recomputed-rows gate, the scenario-year guard, and per-variant caching.
  • New: reference_audit/unlisted_input_acknowledged.json, listing the four scored moves pending d963 and d974.
• Fast tests for the two new modules pass, and both slow real-engine tests pass. The full fast suite passed (1,622) on 567f668.
• The final full sweep and the differential (bit-identical to Audit the state income tax withheld in federal SALT; propose three exclusions #191's scripts) ran on 38b7d87, before the review fixes. Outputs are in results/local/unlisted_input_sweep_20261005/.

Still owed before merge

1. Rerun the full sweep on 4485065. The Medicare not-enrolled reading and the per-output locality rule can change the moved-output list; update the PR body and the acknowledgements file to match.
2. Rerun the full fast suite.
3. Get a second review of 4485065 (subfleet run --task review --tier standard) that answers each finding.
4. Confirm CI is green and the PR is MERGEABLE, then squash-merge.