Add a reusable unlisted-input sweep and keep local taxes and credits out of the state outputs

docs/runbook.md

@@ -54,6 +54,110 @@ uv run python -m policybench.cli reference-outputs \
   --output "$RUN_DIR/us/reference_outputs.csv"
 ```
 
+## 2b. Reference-Build Gates
+
+The published US references are not a bare `reference-outputs` run. They come
+from a reference system: a fix module that composes the newest policyengine-us
+with the pre-freeze publication conventions and the output-definition adapters
+(currently `reference_audit/2026-09-28/fixes/latest_final.py`, built by
+`reference_audit/2026-09-28/scripts/build_references_latest.py`; see that
+directory's README). Before a reference build is published, it must pass every
+gate below. The September 29 build applied gates 1 and 2. Gates 3 and 4 follow
+the October 5 audit, which found references resting on engine estimates of
+inputs no prompt lists.
+
+1. **Newest release.** The references come from the newest policyengine-us
+   release when the sweep begins, and the newest release on PyPI must give the
+   same values before publishing (rule 1 of
+   `reference_audit/2026-09-28/README.md`).
+2. **Reviewed moves.** The reference builder refuses any scored output that
+   moves without a reviewed action, a listed action the engine does not
+   reproduce, and any excluded output that moves without a recheck.
+3. **Unlisted-input sweep.** Recompute every output with each engine estimate
+   of an unlisted input replaced by the prompt's literal reading (unlisted
+   numbers 0, unlisted facts false) or a documented alternative, and list
+   every output that moves:
+
+   ```bash
+   uv run policybench unlisted-input-sweep \
+     --fix reference_audit/2026-09-28/fixes/latest_final.py \
+     --fix-support reference_audit/2026-09-22/fixes/r19_irs_sales_tax_2025.json \
+     --run-dir paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace \
+     --out-dir results/local/unlisted_input_sweep \
+     --acknowledged reference_audit/unlisted_input_acknowledged.json \
+     --strict
+   ```
+
+   `--fix-support` copies a data file the fix module reads from its own
+   directory (the 2026-09-28 sales tax convention reads the September 22 IRS
+   table). Each worker process builds the reference system once (about 40
+   seconds); a household simulation takes about half a second on an idle
+   machine and over a second under load. The county reading tries every county
+   of every household's state, about 9,000 simulations for the 100-household
+   run (23 minutes on 12 workers on a loaded machine); `--county-states MD`
+   limits it. `--estimate` and `--scenario` restrict a run for a quick check.
+
+   The command fixes nothing. It exits 1 when the sweep cannot be trusted (and
+   on a configuration error or a worker crash):
+   - its recomputed reference differs from any scored published reference
+     (the wrong reference system, or a changed engine; pass
+     `--allow-baseline-mismatch` only to preview a new engine);
+   - a reference row of a swept household was not recomputed;
+   - a local tax or credit is nonzero in a state output (gate 4);
+   - a reading's fixed point did not converge.
+
+   With `--strict` it exits 2 when a scored output moves. One kind of move is
+   reported but not gated. A county can switch on one of the engine's locality
+   flags (`in_nyc`, `in_san_francisco`, `in_ny_mctd_zone_2` and its other
+   `in_*` variables), which makes an unlisted household fact true. The prompt's
+   rule makes such facts false. The sweep reruns that county with the flags held
+   at their reference values, and a move that this undoes is marked
+   `prompt_rules_out` (flag in the `localities` column). A move that survives,
+   such as New York SNAP's utility allowance region, which reads the county
+   itself, stays `scored`.
+
+   Each scored move needs one of:
+   - an exclusion record with reason `reference_depends_on_unlisted_input`;
+   - an entry in the `--acknowledged` file while a ruling is pending:
+     `{"acknowledged": [{"scenario_id", "variable", "estimate" ("*" for every
+     reading), "reading" (optional), "status", "note"}]}`.
+
+   `reference_audit/unlisted_input_acknowledged.json` lists the scored moves on
+   the current references that await Max's rulings (d963, d974). Remove an
+   entry when its release lands. A move on an excluded output whose record does
+   not name the estimate's input is `excluded_other_reason`, and the next
+   release should add a note to that record.
+
+   The out directory holds `report.md` (the gates and every moved output),
+   `moves.csv` (one row per output and reading that moved, marked `scored`,
+   `acknowledged`, `prompt_rules_out`, `excluded_same_input` or
+   `excluded_other_reason`),
+   `readings.csv` (what each reading set in each household),
+   `values.csv.gz` (every simulated value), `baseline.csv` (the recomputed
+   reference against the CSV), `households.json` (overrides, fixed-point
+   traces and the local taxes checked) and `summary.json` (provenance:
+   engine versions and the sha256 of the fix files, scenarios, references and
+   exclusions).
+
+   `policybench/unlisted_input_sweep.py` registers the estimates
+   (`ESTIMATES`): the state income tax withheld in the federal SALT deduction
+   (zero, liability and net readings), the 20% local sales tax, the modeled
+   Medicare Part B premium (zero) and Medicare enrollment (not enrolled), the
+   county (Allegany for Maryland), weekly hours
+   for people with none stated, the mortgage origination year, months of SSDI
+   receipt, and SSI's disability criterion, plus one reading that applies
+   every literal reading at once. When an audit finds another estimate,
+   register it there, with a test, before the next build.
+4. **Output scope.** The state income tax and state refundable credit outputs
+   are state-only. policyengine-us 2.15.17 adds Maryland county tax and New
+   York City tax into its state income tax, and New York City's refundable
+   credits and San Francisco's Working Families Tax Credit into its state
+   refundable credits. The sweep adds `policybench/output_scope.py`'s adapter,
+   which removes each of them from the engine's lists, and checks that each one
+   it removes is zero for every household, so the adapter changes no published
+   reference. `tests/test_output_scope.py` fails when a policyengine-us upgrade
+   adds a local entry the adapter does not name.
+
 ## 3. Run Claude Separately
 
 Run Claude models serially. Claude calls need the main-thread wall timeout, and

policybench/cli.py

@@ -970,6 +970,16 @@ def main():
         ),
     )
 
+    sweep_parser = subparsers.add_parser(
+        "unlisted-input-sweep",
+        help="Recompute a frozen run's references with each engine estimate of "
+        "an unlisted input replaced by its literal or documented reading, and "
+        "list every output that moves",
+    )
+    from policybench.unlisted_input_sweep import add_arguments as add_sweep_arguments
+
+    add_sweep_arguments(sweep_parser)
+
     args = parser.parse_args()
 
     # Enable disk cache for ordinary eval calls. Contract-failure retry/repair
@@ -1571,6 +1581,13 @@ def main():
         print(f"sha256: {pointer['sha256']}")
         print(f"Pointer written to {args.pointer_output}")
 
+    elif args.command == "unlisted-input-sweep":
+        from policybench.unlisted_input_sweep import run as run_unlisted_input_sweep
+
+        code = run_unlisted_input_sweep(args)
+        if code:
+            sys.exit(code)
+
     else:
         parser.print_help()
         sys.exit(1)

policybench/output_scope.py

@@ -0,0 +1,218 @@
+"""Keep local taxes and credits out of the state outputs.
+
+PolicyBench's state outputs are state-only (``benchmark_specs.json``):
+
+- ``state_income_tax_before_refundable_credits``: "state individual income tax after
+  nonrefundable credits and before refundable credits, excluding local income and
+  payroll taxes";
+- ``state_refundable_credits``: "total refundable state individual income tax
+  credits".
+
+Local income taxes have their own output, ``local_income_tax``.
+
+policyengine-us builds each engine variable of those names by adding the variables
+a parameter list names. On 2.15.17 three of those lists carry local entries
+(``SCOPED_LISTS``):
+
+- ``gov.states.household.state_income_tax_before_refundable_credits``:
+  ``md_local_income_tax_before_refundable_credits`` (Maryland county income tax,
+  added by upstream #8888; the 2026-09-28 reference system removes it with
+  ``reference_audit/2026-09-28/fixes/latest_md_local_output_scope.py``) and
+  ``nyc_income_tax_before_refundable_credits`` (New York City income tax, which
+  nothing removed);
+- ``gov.states.household.state_refundable_credits``: ``nyc_refundable_credits``;
+- ``gov.states.ca.tax.income.credits.refundable``: ``ca_sf_wftc``, San Francisco's
+  Working Families Tax Credit, which reaches ``state_refundable_credits`` through
+  ``ca_refundable_credits``.
+
+``reform`` removes every local entry, so the engine variables match the benchmark
+outputs. Only the lists change; no formula or amount does. Maryland county tax still
+reaches the federal SALT deduction through ``md_withheld_income_tax``'s county
+estimate, and NYC tax net of NYC refundable credits (``nyc_income_tax``) is still in
+``local_income_tax``.
+
+Every one of these is zero for every frozen household: ``in_nyc`` is true only in
+the five NYC counties and ``in_san_francisco`` only in San Francisco County, and a
+household with no county takes its state's alphabetically first county (Albany
+County for New York, Alameda County for California). ``scope_violations`` is the
+check: it returns each listed local entry that is nonzero in a simulation.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Sequence
+from typing import Any
+
+STATE_AGGREGATE = "state_income_tax_before_refundable_credits"
+STATE_AGGREGATE_PARAMETER = (
+    "gov.states.household.state_income_tax_before_refundable_credits"
+)
+STATE_REFUNDABLE_PARAMETER = "gov.states.household.state_refundable_credits"
+CA_REFUNDABLE_PARAMETER = "gov.states.ca.tax.income.credits.refundable"
+# Each list, the local entries 2.15.17 puts in it, and state entries it must keep.
+SCOPED_LISTS = {
+    STATE_AGGREGATE_PARAMETER: {
+        "local": (
+            "md_local_income_tax_before_refundable_credits",
+            "nyc_income_tax_before_refundable_credits",
+        ),
+        "keeps": (
+            "md_income_tax_before_refundable_credits",
+            "ny_income_tax_before_refundable_credits",
+        ),
+    },
+    STATE_REFUNDABLE_PARAMETER: {
+        "local": ("nyc_refundable_credits",),
+        "keeps": ("ny_refundable_credits", "ca_refundable_credits"),
+    },
+    CA_REFUNDABLE_PARAMETER: {
+        "local": ("ca_sf_wftc",),
+        "keeps": ("ca_eitc",),
+    },
+}
+LOCAL_INCOME_TAX_COMPONENTS = SCOPED_LISTS[STATE_AGGREGATE_PARAMETER]["local"]
+LOCAL_COMPONENTS = tuple(
+    name for spec in SCOPED_LISTS.values() for name in spec["local"]
+)
+# Spelled as engine variable names so a new local entry is caught by name: any
+# list entry containing one of these fragments is local.
+LOCAL_NAME_FRAGMENTS = (
+    "_local_",
+    "nyc_",
+    "_sf_",
+    "san_francisco",
+    "philadelphia",
+    "kansas_city",
+    "st_louis",
+    "wilmington",
+    "denver",
+    "yonkers",
+    "multnomah",
+)
+SCOPE_INSTANT = "2026-01-01"
+# Where the lists are rewritten: the benchmark year and the years after it.
+SCOPE_PERIOD = "year:2026-01-01:10"
+FIX_ID = "policybench_output_scope"
+DESCRIPTION = (
+    "Exclude Maryland county tax and New York City and San Francisco taxes and "
+    "credits from the state income tax and state refundable credit outputs "
+    "(output definition)"
+)
+
+
+def is_local_component(name: str) -> bool:
+    """Whether a list entry names a local (county or city) tax or credit.
+
+    "state_and_local" (as in the SALT variables) is a state entry, not a local one.
+    """
+    stripped = name.replace("state_and_local", "")
+    return name in LOCAL_COMPONENTS or any(
+        fragment in stripped for fragment in LOCAL_NAME_FRAGMENTS
+    )
+
+
+def scoped_components(components: Iterable[str]) -> list[str]:
+    """A list without its local entries, order kept."""
+    return [name for name in components if not is_local_component(name)]
+
+
+def local_components(components: Iterable[str]) -> list[str]:
+    """The local entries of a list, order kept."""
+    return [name for name in components if is_local_component(name)]
+
+
+def _parameter(parameters, path: str):
+    node = parameters
+    for part in path.split("."):
+        node = getattr(node, part)
+    return node
+
+
+def listed_local_components(
+    parameters, path: str = STATE_AGGREGATE_PARAMETER, instant: str = SCOPE_INSTANT
+) -> list[str]:
+    """The local entries one parameter list adds (default: the state income tax)."""
+    return local_components(_parameter(parameters, path)(instant))
+
+
+def all_listed_local_components(parameters, instant: str = SCOPE_INSTANT) -> list[str]:
+    """Every local entry in every scoped list, in ``SCOPED_LISTS`` order."""
+    return [
+        name
+        for path in SCOPED_LISTS
+        for name in listed_local_components(parameters, path, instant)
+    ]
+
+
+def remove_local_components(parameters):
+    """``modify_parameters`` callback: drop every local entry from every list."""
+    for path, spec in SCOPED_LISTS.items():
+        param = _parameter(parameters, path)
+        current = list(param(SCOPE_INSTANT))
+        kept = scoped_components(current)
+        if kept != current:
+            param.update(period=SCOPE_PERIOD, value=kept)
+        after = param(SCOPE_INSTANT)
+        assert not local_components(after), (path, after)
+        for name in spec["keeps"]:
+            assert name in after, (path, name)
+    return parameters
+
+
+def installed_parameter_list(path: str = STATE_AGGREGATE_PARAMETER) -> list[str]:
+    """A list parameter of the installed policyengine-us, read from its YAML.
+
+    The latest value is returned. The file is located without importing
+    policyengine_us and parsed without building a tax-benefit system (about 40
+    seconds), so a test can check every engine upgrade for a new local entry.
+    """
+    import importlib.util
+    from pathlib import Path
+
+    import yaml
+
+    spec = importlib.util.find_spec("policyengine_us")
+    if spec is None or not spec.submodule_search_locations:
+        raise ModuleNotFoundError("policyengine_us is not installed")
+    root = Path(next(iter(spec.submodule_search_locations)))
+    path_on_disk = root / "parameters" / (path.replace(".", "/") + ".yaml")
+    # BaseLoader keeps the 0000-01-01 key a string; the default loader rejects it.
+    data = yaml.load(path_on_disk.read_text(encoding="utf-8"), Loader=yaml.BaseLoader)
+    values = data["values"]
+    return list(values[sorted(values)[-1]])
+
+
+def scope_violations(
+    simulation: Any,
+    components: Sequence[str],
+    period: int | str,
+    tolerance: float = 0.005,
+) -> dict[str, float]:
+    """Each listed local entry that is nonzero in ``simulation``, with its amount."""
+    violations = {}
+    for name in components:
+        amount = float(simulation.calculate(name, period).sum())
+        if abs(amount) > tolerance:
+            violations[name] = amount
+    return violations
+
+
+def _build_reform():
+    from policyengine_core.reforms import Reform
+
+    class reform(Reform):
+        def apply(self):
+            self.modify_parameters(remove_local_components)
+
+    reform.__module__ = __name__
+    return reform
+
+
+def __getattr__(name: str):
+    # ``reform`` is built on first access, as a fix module's ``reform`` is read
+    # (getattr(module, "reform")), so importing the helpers stays cheap: the CLI
+    # imports this module to build its parser.
+    if name == "reform":
+        globals()["reform"] = _build_reform()
+        return globals()["reform"]
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")