diff --git a/changelog.d/1038-cd-source-column-exclusions.fixed.md b/changelog.d/1038-cd-source-column-exclusions.fixed.md new file mode 100644 index 0000000000..dc49547865 --- /dev/null +++ b/changelog.d/1038-cd-source-column-exclusions.fixed.md @@ -0,0 +1 @@ +The US fiscal-target compiler drops congressional-district SOI facts read from an IRS column their measure id does not mean, through a new reviewed register keyed by (measure id, IRS column), `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`: N18425/A18425 (state and local income taxes, Schedule A line 5a) published as the limited SALT deduction, and N85530/A85530 (additional Medicare tax, Form 8959 line 24) published as the premium tax credit. It replaces the hard-coded congressional-district PTC-amount check, refuses a listed district fact that names no column, and appears in the exclusion receipt and the release coverage report. On the pinned feed the full surface loses the 976 district SALT specs, which summed to 1.98x the national target, and the 488 district PTC-return specs (microcosm#1038). diff --git a/docs/us-acs-local-soi-target-surface.md b/docs/us-acs-local-soi-target-surface.md index e072362569..80fcd27bb6 100644 --- a/docs/us-acs-local-soi-target-surface.md +++ b/docs/us-acs-local-soi-target-surface.md @@ -79,6 +79,20 @@ their role is `soi_fiscal_distribution`, so `totals` is unchanged: | **Admin specs** | **4,380** | **760** | **31,474** | With the 487 population marginals `state` now calibrates to 4,867 targets. + +**Since microcosm#1038** the compiler drops the district file's facts read +from the wrong IRS column (`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` in +`us_runtime/fiscal_targets.py`): the SALT rows read from N18425/A18425 +(state and local income taxes) and the premium-tax-credit rows read from +N85530/A85530 (additional Medicare tax). `state` and `state_cd` are +unchanged. `totals` loses the 487 district-file premium-tax-credit return +rows and `full` loses those and the 974 district-file SALT rows: + +| Family | `state` | `totals` | `full` | +|---|---:|---:|---:| +| `irs_soi` | 4,227 | 120 | 29,860 | +| **Admin specs** | **4,380** | **273** | **30,013** | + The rest of this section describes the 2026-09-22 measurement. The 487 population marginals (51 states and 436 congressional districts) are @@ -104,8 +118,12 @@ congressional-district file. The feed-gated test ### What `totals` contains, and what it does not -All 607 SOI specs that `totals` keeps are ACA premium tax credit measures (547 -returns, 60 amounts), 454 of them at congressional-district geography. +All 607 SOI specs that `totals` kept on 2026-09-22 are ACA premium tax credit +measures (547 returns, 60 amounts), 454 of them at congressional-district +geography. Since microcosm#1038 it keeps 120 (60 returns and 60 amounts, +all Historic Table 2: the 51 states and the 9 at-large district copies), 18 +of them at district geography: the 487 district-file return rows read +N85530, additional Medicare tax, not the premium tax credit. `_soi_target_role` in `us_runtime/fiscal_targets.py` gives named roles only to country-level all-income-range facts, the all-income-range PTC measures and the W-2 tip items; every other SOI fact gets `soi_fiscal_distribution`, @@ -292,8 +310,8 @@ bridged state row records its sibling and bridge factor. | Rows | Specs on the pinned feed | Why | |---|---:|---| -| District-file `limited_state_local_taxes_*`, both geographies | 974 | Column A18425/N18425 is state and local income taxes, not the limited SALT deduction (microcosm#1038) | -| District-file `premium_tax_credit_returns`, both geographies | 487 | Column N85530 is the additional Medicare tax, not the premium tax credit (microcosm#1038) | +| District-file `limited_state_local_taxes_*`, both geographies | 974 | Column A18425/N18425 is state and local income taxes, not the limited SALT deduction (microcosm#1038). Since #1038 the compiler drops them first; `STATE_CD_DEFECTIVE_CD_FILE_MEASURES` stays as a second fence | +| District-file `premium_tax_credit_returns`, both geographies | 487 | Column N85530 is the additional Medicare tax, not the premium tax credit (microcosm#1038); dropped first by the compiler, as above | | District `tax_filer_individual_count` | 436 | No state parent in either vintage, so it could not nest in a bound state target | | District rows of states with one district on the 117th plan (AK, DE, DC, MT, ND, SD, VT, WY) | 459 | The SOI file has no sub-state rows there. These rows are the state total copied, or for Montana split by population, so they carry no district information | | Historic Table 2 rows copied to at-large districts | 360 | The same copies from the other vintage | @@ -301,12 +319,12 @@ bridged state row records its sibling and bridge factor. | District rows of the four out-of-band rebase blocks | 34 | The factor band (above) | | Bridged state rows whose sibling ratio is out of band | 4 | The factor band (above) | -PR #1040 (awaiting a ruling) excludes the same SALT and PTC columns in the -compiler and rescales the district file's capital-gains rows by one national -factor. `state_cd` already takes district capital gains as within-state -shares of Historic Table 2, which a national factor does not move, so their -level changes only when Historic Table 2's own capital-gains rows do -(#1036). +Since microcosm#1038 the compiler excludes the same SALT and PTC columns +(`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`). PR #1040 (awaiting a ruling) +rescales the district file's capital-gains rows by one national factor. +`state_cd` already takes district capital gains as within-state shares of +Historic Table 2, which a national factor does not move, so their level +changes only when Historic Table 2's own capital-gains rows do (#1036). North Carolina's district rows are bound. Before #1043 the packaged 117th→119th crosswalk was built from NC's 2016 plan rather than the 2019 plan @@ -556,7 +574,10 @@ record one of the four modes, before any release directory exists. calibrate stage's peak is the artifact write. - A checkpoint written before the sparse matrix (measures as dense H5 columns) is refused by `--stage calibrate`; re-run `--stage materialize`. -- To reproduce the 2026-09-22 hours rebuild (#974), pass `--soi-mode totals`. +- To reproduce the 2026-09-22 hours rebuild (#974), pass `--soi-mode totals` + on a compiler from before microcosm#1038; after it, `totals` no longer + carries the 487 district-file premium-tax-credit return rows that run + calibrated to. To reproduce a build that ran as `full` after `b7922b089`, pass `--soi-mode full`. - A checkpoint materialized under an earlier default records its own mode and diff --git a/docs/us-fact-to-target.md b/docs/us-fact-to-target.md index e4ae417a8d..af5b185b75 100644 --- a/docs/us-fact-to-target.md +++ b/docs/us-fact-to-target.md @@ -204,6 +204,19 @@ should calibrate, add its id to `US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES` with a reason. `us_source_coverage.json`'s `fiscal_target_exclusion_receipt` lists the ids each rule dropped or allowed in a release. +A fact whose package reads the wrong IRS column for its measure id is a +different defect: every fact read from that column is wrong, at every +geography and vintage, while the same measure read from the right column is +fine. `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` is keyed by (measure id, +`layout.source_column_id`) for these, so a package corrected to read the +right column passes untouched; its receipt rule is +`source_column_concept_exclusion` (microcosm#1038). Check the column against +the IRS documentation guide before adding an entry, and list a column there +only when it carries another concept. A column that is a component or a +superset of the measure's concept (the district file's line 8a mortgage +interest under `interest_paid_deduction_*`, its line 7 capital gains) needs a +share-or-drop decision instead. + ## 5. Add a reform-coverage probe Probes are **generated data**: author them in diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index 74e4757cba..ac03bb85c4 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -291,6 +291,7 @@ US_FISCAL_TARGET_LEDGER_REFERENCES, US_FISCAL_TARGET_REFERENCES, US_FISCAL_TARGET_REGISTRY, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SPECS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, @@ -1165,6 +1166,7 @@ "US_FISCAL_TARGET_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES", + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS", "US_FISCAL_TARGET_COVERAGE_REQUIREMENTS", "US_FISCAL_TARGET_LEDGER_REFERENCES", "US_JCT_TAX_EXPENDITURE_REFORMS", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 7df8db9b95..a395b64b06 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -55,6 +55,7 @@ "US_FISCAL_TARGET_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES", + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS", "US_FISCAL_LEDGER_PARITY_REGISTRY", "US_FISCAL_LEDGER_PARITY_REPORT", "US_JCT_TAX_EXPENDITURE_REFORMS", @@ -877,6 +878,74 @@ def _us_hierarchy_seed( # every vintage. US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES: dict[str, str] = {} +# Reviewed source-column concept exclusions (microcosm#1038). Each entry is an +# IRS column that a Chronicle package publishes under a measure id whose +# concept the column does not carry, so no fact read from that column +# compiles, whatever its package, geography or vintage, and a package +# corrected to read the right column passes untouched. The key is (measure_id, +# layout.source_column_id) because the IRS column is the definition. The +# congressional-district documentation guide (22incddocguide.docx) defines: +# +# - N18425/A18425 as "State and local income taxes" (Schedule A line 5a) and +# N18460/A18460 as "Limited state and local taxes" (line 5e). salt_deduction, +# like the Historic Table 2 rows of these measure ids, is the limited line 5e +# deduction. +# - N85530/A85530 as "Additional Medicare tax" (Form 8959 line 24) and +# N85770/A85770 as "Total premium tax credit" (Form 8962 line 24; the guide +# prints "8926:24"). N85770/A85770 is what the Historic Table 2 rows of +# these measure ids read and what assigned_aca_ptc measures. The amount was +# dropped before by a hard-coded congressional-district check; the returns +# row compiled as premium-tax-credit recipients. A package corrected to read +# N85770/A85770 compiles as assigned_aca_ptc targets with no further gate, +# which is intended: those columns are the premium tax credit. +# +# Chronicle's congressional_district_2022 package reads these four columns +# under the limited_state_local_taxes_* and premium_tax_credit_* measure ids; +# the 46 other measures it shares with a Historic Table 2 package read the +# same column there. Of the six it has alone, interest_paid_deduction_* reads +# N19300/A19300, "Mortgage interest paid" (Schedule A line 8a), while +# interest_deduction is the line 10 total (lines 8a-8c plus investment +# interest on line 9; 8a is 85.6% of it in the TY2022 US row). That is a +# component of the measure's concept, not another concept, so it is not +# listed here: like the line 7 capital-gains rows, it needs a share-or-drop +# decision of its own, which also binds the state_cd surface's itemized +# bridge. +# A congressional-district fact of a listed measure that carries no +# layout.source_column_id refuses the compile, so a feed schema change cannot +# silently lift the exclusion. +US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS: dict[tuple[str, str], str] = { + ("limited_state_local_taxes_returns", "N18425"): ( + "N18425 counts returns with state and local income taxes (Schedule A " + "line 5a), not returns with the limited state and local tax deduction " + "(line 5e, N18460). Chronicle's congressional_district_2022 package " + "reads it under this measure id: TY2022 US 10,842,580 returns against " + "14,777,040 for N18460 in the same file and 14,968,720 in Historic " + "Table 2." + ), + ("limited_state_local_taxes_amount", "A18425"): ( + "A18425 is state and local income taxes before the $10,000 limit " + "(Schedule A line 5a), not the limited deduction (line 5e, A18460) " + "salt_deduction measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US $250.4B against " + "$121.2B for A18460 in the same file, so on the full surface the " + "congressional-district rows summed to 1.98x the national Historic " + "Table 2 target." + ), + ("premium_tax_credit_returns", "N85530"): ( + "N85530 counts returns with additional Medicare tax (Form 8959 line 24), " + "not returns with the premium tax credit (Form 8962 line 24, N85770) " + "assigned_aca_ptc measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US 6,848,330 returns " + "against 7,713,000 for N85770 in the same file." + ), + ("premium_tax_credit_amount", "A85530"): ( + "A85530 is additional Medicare tax (Form 8959 line 24), not the premium " + "tax credit (Form 8962 line 24, A85770). Chronicle's " + "congressional_district_2022 package reads it under this measure id: " + "TY2022 US $14.2B against $53.1B for A85770 in the same file." + ), +} + @dataclass(frozen=True) class SimpleTaxExpenditureReform: @@ -1033,7 +1102,7 @@ def compile_us_fiscal_target_registry( auditable waiver instead of raising; every waived target carries ``period_contract_waiver`` metadata in diagnostics. """ - materialized_facts = tuple(facts) + materialized_facts = _without_source_column_concept_exclusions(tuple(facts)) if congressional_district_vintage_crosswalk is not None: materialized_facts = translate_congressional_district_facts_to_current_vintage( materialized_facts, @@ -1114,6 +1183,10 @@ def us_fiscal_target_exclusion_receipt( vintage included (dropped). - ``m_chip_state_chip_enrollment``: CMS CHIP enrollment facts for an ``_M_CHIP_STATE_FIPS`` state, at every period (dropped). + - ``source_column_concept_exclusion``: SOI facts read from an IRS column + that ``US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`` lists for their + measure id, at every vintage, as the feed carries them, before the + congressional-district crosswalk (dropped). - ``allowlisted_vintage_bypass``: facts latest-vintage selection activates at ``target_period`` that are another vintage of an excluded cell and carry a reviewed ``US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES`` entry @@ -1124,7 +1197,13 @@ def us_fiscal_target_exclusion_receipt( it raises on a vintage bypass nobody reviewed. """ - materialized_facts = tuple(facts) + raw_facts = tuple(facts) + source_column = { + _source_record_id(fact) + for fact in raw_facts + if _is_source_column_concept_exclusion(fact) and _source_record_id(fact) + } + materialized_facts = _without_source_column_concept_exclusions(raw_facts) if congressional_district_vintage_crosswalk is not None: materialized_facts = translate_congressional_district_facts_to_current_vintage( materialized_facts, @@ -1168,6 +1247,22 @@ def us_fiscal_target_exclusion_receipt( "entries": sorted(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS), "source_record_ids": sorted(all_vintage), }, + "source_column_concept_exclusion": { + "action": "dropped", + "scope": "SOI facts read from a listed column under a listed " + "measure id, every vintage", + "entries": [ + { + "measure_id": measure_id, + "source_column_id": source_column_id, + "reason": reason, + } + for (measure_id, source_column_id), reason in sorted( + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items() + ) + ], + "source_record_ids": sorted(source_column), + }, "m_chip_state_chip_enrollment": { "action": "dropped", "scope": "CMS CHIP enrollment rows for these states, every period", @@ -2916,6 +3011,55 @@ def _period_free_source_record_id(source_record_id: str) -> str: return _normalized_record_set_id(source_record_id) +class MissingSoiSourceColumnError(ValueError): + """A congressional-district fact of a listed measure names no IRS column.""" + + +def _is_source_column_concept_exclusion(fact: object) -> bool: + """Whether an SOI fact reads a column its measure id does not mean. + + A congressional-district fact of a measure the register lists must name + its column: without one the exclusion could not tell a mislabeled column + from a corrected one, so the compile refuses instead of guessing. + """ + if _source_name(fact) != "irs_soi": + return False + measure_id = _measure_id(fact) + if measure_id not in { + measure for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + }: + return False + column = _str_at(fact, "layout", "source_column_id") + if not column: + # Any congressional-district package, whatever its boundary vintage. + if ".congressional_district_" in _str_at(fact, "layout", "record_set_id"): + raise MissingSoiSourceColumnError( + f"{_source_record_id(fact)} is a congressional-district " + f"{measure_id} fact with no layout.source_column_id, so " + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS cannot tell which " + "IRS column it reads (microcosm#1038)." + ) + return False + return (measure_id, column) in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + + +def _without_source_column_concept_exclusions( + facts: tuple[object, ...], +) -> tuple[object, ...]: + """Drop facts read from a listed column, before the district crosswalk. + + The crosswalk sums several source districts into one current district and + keeps one contributor's layout, so the column check must see each source + fact: afterwards an excluded contribution could inherit a corrected + column, and a fact with no column could hide behind one that has it. + Raises ``MissingSoiSourceColumnError`` for a listed district fact with no + column. + """ + return tuple( + fact for fact in facts if not _is_source_column_concept_exclusion(fact) + ) + + def _is_all_vintage_support_exclusion(source_record_id: str) -> bool: return _period_free_source_record_id(source_record_id) in ( _period_free_source_record_ids(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS) @@ -2950,18 +3094,6 @@ def _is_soi_congressional_district_record_set(fact: object) -> bool: return ".congressional_district_2022." in record_set_id -def _is_soi_cd_premium_tax_credit_amount_conflict( - fact: object, *, measure_id: str -) -> bool: - # The SOI CD A85530 amount is not the same gross annual PTC control used by - # assigned_aca_ptc. Keep Historic Table 2 national/state PTC amount controls - # until Ledger has an explicit CD reconciliation for this concept. - return ( - measure_id == "premium_tax_credit_amount" - and _is_soi_congressional_district_record_set(fact) - ) - - def _is_period_token(value: str) -> bool: normalized = value.lower().replace("-", "_") if normalized.startswith("month"): @@ -3028,6 +3160,8 @@ def _reference_from_ledger_fact( return None if _is_all_vintage_support_exclusion(source_record_id): return None + if _is_source_column_concept_exclusion(fact): + return None source_name = _source_name(fact) if source_name == "irs_soi": return _soi_reference_from_fact( @@ -3085,8 +3219,6 @@ def _soi_reference_from_fact( and geography_level != "congressional_district" ): return None - if _is_soi_cd_premium_tax_credit_amount_conflict(fact, measure_id=measure_id): - return None variable = SOI_AMOUNT_MEASURE_VARIABLES.get(measure_id) is_count = False if variable is None: diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py index 4024b3a260..c948e928ab 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py @@ -956,6 +956,22 @@ def test_pinned_feed_soi_surfaces_match_their_contracts() -> None: } assert {spec.metadata["ledger_geography_level"] for spec in soi} == {"state"} assert {spec.name for spec in state_specs} <= set(full_by_name) + # The compiler drops the district file's SALT and premium-tax-credit facts + # read from the wrong IRS column (microcosm#1038), so `totals` keeps only + # the Historic Table 2 premium-tax-credit rows (51 states and 9 at-large + # district copies, returns and amounts) and `full` 29,860 SOI specs. + totals_soi = [spec for spec in totals_registry.specs if spec.family == "irs_soi"] + assert len(totals_soi) == 120 + assert {spec.metadata["source_measure_id"] for spec in totals_soi} == { + "premium_tax_credit_returns", + "premium_tax_credit_amount", + } + assert not [ + spec.name + for spec in totals_soi + if ".congressional_district_" in spec.metadata["ledger_layout_record_set_id"] + ] + assert sum(spec.family == "irs_soi" for spec in full_registry.specs) == 29_860 assert not [ spec diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 3cd976bac3..80696fca29 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -109,10 +109,14 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - _soi_congressional_district_fact( + # The pinned feed reads the CD PTC measures from A85530, additional + # Medicare tax, so they never compile (source-column exclusion). + _cd_column_fact( "premium_tax_credit_amount", - 90_000_000, + "A85530", + value=90_000_000, groupby_value_id="hi_01", + geography_level="congressional_district", geography_id="5001700US1501", ), _soi_congressional_district_fact( @@ -127,10 +131,14 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - _soi_congressional_district_fact( + # The pinned feed reads it from N19300 (line 8a mortgage interest, a + # component of the line 10 deduction), which is not excluded. + _cd_column_fact( "interest_paid_deduction_returns", - 23_456, + "N19300", + value=23_456, groupby_value_id="hi_01", + geography_level="congressional_district", geography_id="5001700US1501", ), _soi_congressional_district_fact( @@ -166,9 +174,10 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: geography_level="state", geography_id="0400000US15", ), - _soi_congressional_district_fact( + _cd_column_fact( "premium_tax_credit_amount", - 180_000_000, + "A85530", + value=180_000_000, groupby_value_id="hi_total", geography_level="state", geography_id="0400000US15", @@ -736,12 +745,14 @@ def test_exclusion_vintage_scope_registers_are_consistent() -> None: def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: - """One synthetic feed exercises all four rules. The M-CHIP and all-vintage + """One synthetic feed exercises all five rules. The M-CHIP and all-vintage rules list every vintage the feed carries (both CMS months even though a 2024 build selects only 2024-12), exact-key hits land under - reviewed_exclusion, and the allowlisted bypass is the fact selection - activates. The receipt is plain JSON.""" - synthetic, expected = _four_rule_synthetic_feed(monkeypatch) + reviewed_exclusion, the source-column rule lists the facts read from a + listed column and not the one read from the right column, and the + allowlisted bypass is the fact selection activates. The receipt is plain + JSON.""" + synthetic, expected = _exclusion_rule_synthetic_feed(monkeypatch) facts = [*packaged_reference_facts(), *synthetic] receipt = us_fiscal_target_exclusion_receipt(facts, target_period=2024) @@ -752,6 +763,7 @@ def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: assert {name: rule["action"] for name, rule in rules.items()} == { "reviewed_exclusion": "dropped", "all_vintage_reviewed_exclusion": "dropped", + "source_column_concept_exclusion": "dropped", "m_chip_state_chip_enrollment": "dropped", "allowlisted_vintage_bypass": "allowed", } @@ -763,11 +775,236 @@ def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: assert rules["m_chip_state_chip_enrollment"]["state_fips"] == sorted( _M_CHIP_STATE_FIPS ) + assert [ + (entry["measure_id"], entry["source_column_id"]) + for entry in rules["source_column_concept_exclusion"]["entries"] + ] == sorted(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) + + +def test_source_column_exclusion_drops_only_the_mislabeled_column() -> None: + """A congressional-district fact labelled limited_state_local_taxes but read + from N18425/A18425 (state and local income taxes, Schedule A line 5a), or + labelled premium_tax_credit but read from N85530 (additional Medicare + tax), never compiles, at any geography; the same measures read from + A18460 and N85770, the columns a corrected package would read, compile + (microcosm#1038).""" + mislabeled = [ + _cd_column_fact("limited_state_local_taxes_returns", "N18425"), + _cd_column_fact("limited_state_local_taxes_amount", "A18425"), + _cd_column_fact( + "limited_state_local_taxes_returns", + "N18425", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ), + ] + corrected = _cd_column_fact( + "limited_state_local_taxes_amount", + "A18460", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + medicare = _cd_column_fact("premium_tax_credit_returns", "N85530") + ptc = _cd_column_fact( + "premium_tax_credit_returns", + "N85770", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + # The amount twin too: the removed record-set check dropped every + # district PTC amount; the register keeps A85770, the premium tax credit. + ptc_amount = _cd_column_fact( + "premium_tax_credit_amount", + "A85770", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + mislabeled = [*mislabeled, medicare] + facts = [*packaged_reference_facts(), *mislabeled, corrected, ptc, ptc_amount] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + compiled = {spec.metadata["ledger_source_record_id"] for spec in registry.specs} + assert not compiled & {fact["lineage"]["source_record_id"] for fact in mislabeled} + assert corrected["lineage"]["source_record_id"] in compiled + assert ptc["lineage"]["source_record_id"] in compiled + assert ptc_amount["lineage"]["source_record_id"] in compiled + rule = us_fiscal_target_exclusion_receipt(facts)["rules"][ + "source_column_concept_exclusion" + ] + assert rule["source_record_ids"] == sorted( + fact["lineage"]["source_record_id"] for fact in mislabeled + ) + + +def test_source_column_exclusions_name_the_columns_their_guides_define() -> None: + """Each entry's reason names its own column and what its measure id means, + as 22incddocguide.docx defines them: N18425/A18425 (state and local income + taxes, Schedule A line 5a) stand in for the limited deduction N18460/A18460 + (line 5e) and N85530/A85530 (additional Medicare tax) for the premium tax + credit N85770/A85770.""" + right_columns = { + ("limited_state_local_taxes_returns", "N18425"): "N18460", + ("limited_state_local_taxes_amount", "A18425"): "A18460", + ("premium_tax_credit_returns", "N85530"): "N85770", + ("premium_tax_credit_amount", "A85530"): "A85770", + } + assert set(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) == set(right_columns) + for key, reason in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items(): + measure_id, column = key + assert column in reason + # Counts read N columns and amounts read A columns. + assert column[0] == ("N" if measure_id.endswith("_returns") else "A") + assert right_columns[key] in reason + assert right_columns[key][0] == column[0] + + +def test_source_column_exclusion_refuses_a_district_fact_without_a_column() -> None: + """A congressional-district fact of a listed measure that names no IRS + column refuses the compile and the receipt, because the register could not + tell a mislabeled column from a corrected one. A Historic Table 2 fact of + the same measure without a column (the US package publishes none) still + compiles.""" + unnamed = _soi_congressional_district_fact( + "limited_state_local_taxes_amount", + 250_000_000.0, + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + ht2 = _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.us.all.limited_state_local_taxes_amount" + ), + measure_id="limited_state_local_taxes_amount", + value=122_000_000_000.0, + ) + reference = packaged_reference_facts() + + for facts in ([*reference, unnamed], [*reference, ht2, unnamed]): + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + us_fiscal_target_exclusion_receipt(facts) + registry = compile_us_fiscal_target_registry( + [*reference, ht2], allow_unaged_dollar_targets=True + ) + assert ht2["lineage"]["source_record_id"] in { + spec.metadata["ledger_source_record_id"] for spec in registry.specs + } + + +#: Two Hawaii source districts merged into one current district, so the +#: crosswalk sums them under one contributor's layout. +_HI_MERGE_CROSSWALK = [ + { + "source_geography_id": "5001700US1501", + "target_geography_id": "5001900US1501", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US1502", + "target_geography_id": "5001900US1501", + "weight": 1.0, + }, +] + + +def _hi_salt_district_fact(number: int, column: str | None, value: float): + if column is None: + return _soi_congressional_district_fact( + "limited_state_local_taxes_amount", + value, + groupby_value_id=f"hi_0{number}", + geography_id=f"5001700US150{number}", + ) + return _cd_column_fact( + "limited_state_local_taxes_amount", + column, + value=value, + groupby_value_id=f"hi_0{number}", + geography_level="congressional_district", + geography_id=f"5001700US150{number}", + ) + + +@pytest.mark.parametrize( + "columns", + [("A18460", "A18425"), ("A18425", "A18460")], + ids=["kept-first", "excluded-first"], +) +def test_source_column_exclusion_runs_before_the_crosswalk_merges(columns) -> None: + """The crosswalk sums source districts into one current district under one + contributor's layout, so the column check runs on each source fact first. + In either order the A18425 contribution never reaches the merged target, + which carries only the A18460 district's value, and the receipt names the + excluded source fact (#1224 review).""" + facts = [ + _hi_salt_district_fact(number, column, value) + for number, column, value in zip((1, 2), columns, (100.0, 200.0), strict=True) + ] + feed = [*packaged_reference_facts(), *facts] + + registry = compile_us_fiscal_target_registry( + feed, + congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK, + allow_unaged_dollar_targets=True, + ) + receipt = us_fiscal_target_exclusion_receipt( + feed, congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK + ) + + merged = [ + spec.value + for spec in registry.specs + if spec.metadata.get("source_measure_id") == "limited_state_local_taxes_amount" + and spec.metadata.get("ledger_geography_level") == "congressional_district" + ] + assert merged == [100.0 if columns[0] == "A18460" else 200.0] + excluded = next(f for f in facts if f["layout"]["source_column_id"] == "A18425") + assert receipt["rules"]["source_column_concept_exclusion"]["source_record_ids"] == [ + excluded["lineage"]["source_record_id"] + ] + + +@pytest.mark.parametrize( + "columns", + [("A18460", None), (None, "A18460"), ("A18425", None), (None, "A18425")], +) +def test_missing_column_refuses_before_the_crosswalk_merges(columns) -> None: + """A district fact of a listed measure with no column refuses in any merge + order: the crosswalk cannot hide it behind a contributor that has one, nor + behind an excluded one (#1224 review).""" + feed = [ + *packaged_reference_facts(), + *( + _hi_salt_district_fact(number, column, 100.0) + for number, column in zip((1, 2), columns, strict=True) + ), + ] + + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + compile_us_fiscal_target_registry( + feed, + congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK, + allow_unaged_dollar_targets=True, + ) + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + us_fiscal_target_exclusion_receipt( + feed, congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK + ) def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: """Invariant (receipt completeness): each rule lists exactly the facts it - drops or allows. For any subset of the four-rule synthetic feed, each rule + drops or allows. For any subset of the five-rule synthetic feed, each rule lists exactly the full feed's ids for that rule that the subset carries. Hypothesis is a workspace dependency; the wheels job installs no test extras, so it skips there.""" @@ -775,7 +1012,7 @@ def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: from hypothesis import given, settings from hypothesis import strategies as st - synthetic, expected = _four_rule_synthetic_feed(monkeypatch) + synthetic, expected = _exclusion_rule_synthetic_feed(monkeypatch) ids = [fiscal_targets._source_record_id(fact) for fact in synthetic] reference = packaged_reference_facts() @@ -861,9 +1098,12 @@ def check(years: list[int], reviewed: bool) -> None: def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> None: """On the pinned feed the receipt names the 40 M-CHIP CHIP ids (20 states - x 2 CMS months), the 16 other-income ids (ty2020-ty2023) and the ty2020 - and ty2023 tips return counts, and allows no vintage bypass (decision - d179).""" + x 2 CMS months), the 16 other-income ids (ty2020-ty2023), the ty2020 and + ty2023 tips return counts, and the 1,920 congressional-district SALT and + premium-tax-credit facts read from the wrong IRS column, as the feed + carries them before the district crosswalk (480 per measure: the US row, + 51 states, 428 source districts), and allows no vintage bypass (decision + d179, microcosm#1038).""" _, _, receipt = pinned_feed_national_state_surface rules = receipt["rules"] m_chip_ids = rules["m_chip_state_chip_enrollment"]["source_record_ids"] @@ -888,15 +1128,27 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No ] ) assert rules["allowlisted_vintage_bypass"]["source_record_ids"] == [] + column_ids = rules["source_column_concept_exclusion"]["source_record_ids"] + assert all(".congressional_district_2022." in i for i in column_ids) + assert { + measure: sum( + f".{measure}" in source_record_id for source_record_id in column_ids + ) + for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + } == {measure: 480 for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS} + assert len(column_ids) == 4 * 480 def test_pinned_feed_national_state_surface_restores_the_fences( pinned_feed_national_state_surface, ) -> None: - """The release surface on the pinned feed: 33,250 compiled targets after - Medicaid substitution, 6,102 national_state targets at registry - e1f9906bf0e0, 32 CHIP rows none of them for an M-CHIP state, no - other-income row and no tips return count. microcosm#1206 moved the + """The release surface on the pinned feed: 31,786 compiled + targets after Medicaid substitution (the full surface, at f38dbd749180), + 6,102 national_state targets at registry e1f9906bf0e0, 32 CHIP rows none + of them for an M-CHIP state, no other-income row and no tips return count. + microcosm#1038 dropped the 1,464 congressional-district SALT and + premium-tax-credit specs read from the wrong IRS column, all on the full + surface, from 33,250 (full f5bd921d6b21). microcosm#1206 moved the version from f96be9f41781 by binding the Part B premium target to gross_medicare_part_b_premium_if_enrolled; the counts are unchanged. The 408 state AGI-band rows of @@ -905,8 +1157,13 @@ def test_pinned_feed_national_state_surface_restores_the_fences( 32,867 / 5,719 at d5f9d854fe11, and before decision d179 dropped the ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77 (docs/us-chronicle-feed-repin.md).""" + from microcosm.calibrate import TargetRegistry + registry, surface, _ = pinned_feed_national_state_surface - assert len(registry.specs) == 33_250 + assert len(registry.specs) == 31_786 + builder = _load_repo_tool("build_us_fiscal_refresh_release") + full, _ = builder._select_target_surface(registry.specs, "full") + assert TargetRegistry(full, country="us").version == "f38dbd749180" assert len(surface.specs) == 6_102 assert surface.version == "e1f9906bf0e0" chip = [ @@ -935,6 +1192,33 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) +def test_pinned_feed_no_district_target_reads_an_excluded_column( + pinned_feed_national_state_surface, +) -> None: + """No congressional-district SALT or premium-tax-credit target compiles on + the pinned feed: every district-file fact of those measures reads a + listed column. Before microcosm#1038 the 488 SALT amount specs summed to + 1.98x the national Historic Table 2 target on the full surface (A18425, + state and local income taxes) and 488 premium-tax-credit return specs + counted additional-Medicare-tax payers (N85530).""" + registry, _, _ = pinned_feed_national_state_surface + excluded = {measure for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS} + assert not [ + spec.name + for spec in registry.specs + if spec.metadata.get("source_measure_id") in excluded + and ".congressional_district_" + in spec.metadata.get("ledger_layout_record_set_id", "") + ] + # The Historic Table 2 rows of the same measures still compile. + assert { + spec.metadata["source_measure_id"] + for spec in registry.specs + if spec.metadata.get("source_measure_id") in excluded + and ".historic_table_2." in spec.metadata.get("ledger_layout_record_set_id", "") + } == excluded + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." diff --git a/test_support/microcosm_build/us_fiscal_targets.py b/test_support/microcosm_build/us_fiscal_targets.py index f3c4f5c83c..1fe88d05bc 100644 --- a/test_support/microcosm_build/us_fiscal_targets.py +++ b/test_support/microcosm_build/us_fiscal_targets.py @@ -18,6 +18,7 @@ US_FISCAL_TARGET_COVERAGE_REQUIREMENTS, US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES, US_FISCAL_TARGET_REFERENCES, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, US_NONNEGATIVE_SOURCE_OUTPUTS, @@ -160,12 +161,37 @@ def _w2_tips_return_count_fact(tax_year: int) -> dict[str, object]: ) -def _four_rule_synthetic_feed(monkeypatch): +def _cd_column_fact( + measure_id: str, + source_column_id: str, + *, + groupby_value_id: str = "us", + geography_level: str = "country", + geography_id: str = "0100000US", + value: float = 1_000_000.0, +) -> dict[str, object]: + """A congressional-district fact read from IRS column ``source_column_id``, + as every pinned-feed fact records in ``layout.source_column_id``.""" + fact = _soi_congressional_district_fact( + measure_id, + value, + groupby_value_id=groupby_value_id, + geography_level=geography_level, + geography_id=geography_id, + ) + fact["layout"]["source_column_id"] = source_column_id + return fact + + +def _exclusion_rule_synthetic_feed(monkeypatch): """Synthetic facts that exercise every receipt rule, and each rule's ids. The register holds no bypass since d179, so this reviews a synthetic one: a taxable-interest cell excluded at ty2023 with its ty2020 vintage - allowlisted. + allowlisted. The congressional-district SALT facts read the income-tax + column N18425/A18425 that the pinned feed labels as the limited deduction, + and one reads N18460, the column a corrected package would read, which + must compile. """ tanf = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." @@ -217,8 +243,30 @@ def _four_rule_synthetic_feed(monkeypatch): ) ), ] + salt_excluded = [ + _cd_column_fact("limited_state_local_taxes_returns", "N18425"), + _cd_column_fact("limited_state_local_taxes_amount", "A18425"), + _cd_column_fact( + "limited_state_local_taxes_amount", + "A18425", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ), + ] + salt_kept = _cd_column_fact( + "limited_state_local_taxes_returns", + "N18460", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + facts = [*facts, *salt_excluded, salt_kept] expected = { "reviewed_exclusion": sorted([tanf, interest_excluded]), + "source_column_concept_exclusion": sorted( + fact["lineage"]["source_record_id"] for fact in salt_excluded + ), "all_vintage_reviewed_exclusion": sorted( [ *( diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index 6b64ce7059..35e639ec18 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -92,6 +92,7 @@ SIPP_2023_VOLUNTARY_FILING_DONOR_SIZE_BYTES, SOI_VARIABLE_MAP, US_FISCAL_TARGET_COVERAGE_REQUIREMENTS, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, US_MEDICAID_ENROLLMENT_TARGET_TABLE, @@ -15847,6 +15848,16 @@ def _main(argv: Sequence[str] | None = None) -> int | None: US_FISCAL_TARGET_SUPPORT_EXCLUSIONS.items() ) ] + coverage["fiscal_target_source_column_exclusions"] = [ + { + "measure_id": measure_id, + "source_column_id": source_column_id, + "reason": reason, + } + for (measure_id, source_column_id), reason in sorted( + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items() + ) + ] coverage[US_FISCAL_TARGET_EXCLUSION_RECEIPT_KEY] = fiscal_target_exclusion_receipt write_us_source_coverage_diagnostics( coverage, release_dir / "us_source_coverage.json" diff --git a/tools/us_acs_local_cd_surface.py b/tools/us_acs_local_cd_surface.py index edd37055fa..1348e0e4fc 100644 --- a/tools/us_acs_local_cd_surface.py +++ b/tools/us_acs_local_cd_surface.py @@ -50,8 +50,10 @@ STATE_CD_VINTAGE_RULE = "state_cd.historic_table_2_level_cd_file_shares.v1" #: District-file measures that read the wrong IRS column (microcosm#1038). -#: PR #1040 excludes the same columns in the compiler; until it lands this -#: register keeps them off the ``state_cd`` surface at both geographies. +#: The compiler's ``US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`` drops these +#: facts by (measure id, IRS column) before this surface sees them; this +#: register stays as a second fence on the ``state_cd`` surface, at both +#: geographies, and records the reason in the receipt. STATE_CD_DEFECTIVE_CD_FILE_MEASURES: Mapping[str, str] = { "limited_state_local_taxes_amount": ( "22incd.csv column A18425 is state and local income taxes (Schedule A "