Add IRS Historic Table 2 state facts for refunds, taxes paid, mortgage interest and charitable contributions

changelog.d/irs-soi-ht2-state-schedule-a.added.md

@@ -0,0 +1,9 @@
+Add four IRS SOI Historic Table 2 column pairs to every state row of
+`soi-historic-table-2-state-broad-2022`, each pinned to the TY2022
+documentation guide: taxable refunds of state and local income taxes
+(`N00700`/`A00700`, Schedule 1 line 1), the taxes paid deduction
+(`N18300`/`A18300`, Schedule A line 7), home mortgage interest paid to
+financial institutions (`N19300`/`A19300`, line 8a) and the contributions
+deduction (`N19700`/`A19700`, line 14). The last two use Table 2.1's ids,
+concepts and labels, and the congressional-district package's `charitable_*`
+move to Table 2.1's concept and label for the same line 14 columns.

docs/concept-migrations.md

@@ -149,3 +149,38 @@ The line 10 ids may not be read from a geographic file. Two differential checks
 compare published values: the district US row against Historic Table 2's, and
 the line 8a column and four-part sum against Table 2.1. A property check holds
 every limited SALT row to the $10,000 line 5e cap, within publisher rounding.
+
+## 2026-10-12: IRS SOI Schedule A line 14 takes Table 2.1's contributions concept
+
+| Package | Columns | Measure id | Old concept / label | New concept / label |
+|---|---|---|---|---|
+| `soi-congressional-district-2022` | `N19700` | `charitable_returns` | `irs_soi.returns_with_charitable_contributions_deduction` / "Returns with charitable contributions deduction" | `irs_soi.returns_with_contributions_deduction` / "Returns with contributions deduction" |
+| same | `A19700` | `charitable_amount` | `irs_soi.charitable_contributions_deduction` / "Charitable contributions deduction" | `irs_soi.contributions_deduction` / "Contributions deduction amount" |
+
+`measure_id`, `source_record_id`, values and lineage stay.
+
+**Why.** The congressional-district and Historic Table 2 guides define
+`N19700`/`A19700` as "Total charitable contributions", Schedule A line 14 (the
+2022 form's "Add lines 11 through 13"). Table 2.1's `charitable_*` read its
+"Contributions deduction: Total" columns, the same line, under
+`irs_soi.contributions_deduction`. Two ids named one line. For TY2022 the
+district file's US `A19700` is $206.1B and Historic Table 2's is $221.0B,
+0.927 and 0.994 of Table 2.1's $222.4B.
+
+**Added alongside.** `soi-historic-table-2-state-broad-2022` now reads the same
+line 14 columns, and line 8a's `N19300`/`A19300`, for every state under Table
+2.1's ids, concepts and labels. It also reads two columns no package selected
+before, under new ids: `N00700`/`A00700` (Schedule 1 line 1, taxable refunds of
+state and local income taxes) as `taxable_state_local_income_tax_refunds_*`, and
+`N18300`/`A18300` (Schedule A line 7, taxes paid) as `taxes_paid_deduction_*`.
+
+**Consumer effect.** Consumers that select by `measure_id` see no change from
+the rename. The district file's state-total rows and the new Historic Table 2
+state rows publish the same lines 8a and 14 for the same states, so they share
+semantic fact keys, as the other columns of the two files already do.
+
+**Pinned by** `tests/test_chronicle_soi_doc_guide_columns.py`:
+`test_schedule_a_lines_shared_across_grains_carry_table_2_1_ids` holds every
+package that reads `N19300`, `A19300`, `N19700` or `A19700` to Table 2.1's
+measure id, concept and label, and `test_line_7_and_refund_columns_keep_one_identity`
+gives `N00700`, `A00700`, `N18300` and `A18300` one identity in every package.

docs/us-legacy-etl-cleanup.md

@@ -121,14 +121,14 @@ counted 32,067,810. The retired tests pinned the invented levels: for example
 | `db/etl_soi_credits.py` `ctc_claims`, `ctc_amount`, `actc_claims`, `actc_amount` (state) | `soi-historic-table-2-state-broad-2022` `ctc_*` (`N07225`/`A07225`), `actc_*` (`N11070`/`A11070`) | None for 2022. |
 | `db/etl_soi_credits.py` EITC by number of children (national) | `soi-table-2-5-eitc-agi-children-2022`, `soi-table-2-5-eitc-agi-children-2023`; by state: `soi-historic-table-2-state-eitc-2022` `eitc_no_children_*` through `eitc_three_or_more_children_*` | None. |
 | `db/etl_soi_credits.py` CTC and ACTC by number of children (national) | — | Gap: no packaged IRS table publishes CTC or ACTC by number of qualifying children. |
-| `db/etl_soi_deductions.py` `salt_claims`, `salt_amount` (state) | `soi-historic-table-2-state-broad-2022` `limited_state_local_taxes_*`, `real_estate_taxes_*` | The legacy "SALT" named no IRS line; HT2's total-taxes-paid column (`N18300`/`A18300`) is not packaged at state grain. |
-| `db/etl_soi_deductions.py` `mortgage_interest_claims`, `mortgage_interest_amount` (state) | state: `soi-congressional-district-2022` `interest_paid_deduction_*` (`N19300`/`A19300`, state rows for all 51 geographies; chronicle#345 relabels the column as Schedule A line 8a home mortgage interest); national: `soi-table-2-1` `mortgage_interest_paid_*` | None for 2022. |
-| `db/etl_soi_deductions.py` `charitable_claims`, `charitable_amount` (state) | state: `soi-congressional-district-2022` `charitable_*` (`N19700`/`A19700`, state rows for all 51 geographies); national: `soi-table-2-1` `charitable_*` | None for 2022. |
+| `db/etl_soi_deductions.py` `salt_claims`, `salt_amount` (state) | `soi-historic-table-2-state-broad-2022` `taxes_paid_deduction_*` (`N18300`/`A18300`, Schedule A line 7), `limited_state_local_taxes_*` (line 5e), `real_estate_taxes_*` (line 5b) | The legacy "SALT" named no IRS line, so no one measure is a like-for-like replacement. None for 2022. |
+| `db/etl_soi_deductions.py` `mortgage_interest_claims`, `mortgage_interest_amount` (state) | state: `soi-historic-table-2-state-broad-2022` `mortgage_interest_paid_*` (`N19300`/`A19300`, Schedule A line 8a, Historic Table 2's state rows) and `soi-congressional-district-2022` `mortgage_interest_paid_*` (the district file's state-total rows for the same column); national: `soi-table-2-1` `mortgage_interest_paid_*` | None for 2022. The two state series are separate IRS tabulations of one line and differ by state. |
+| `db/etl_soi_deductions.py` `charitable_claims`, `charitable_amount` (state) | state: `soi-historic-table-2-state-broad-2022` `charitable_*` (`N19700`/`A19700`, Schedule A line 14, Historic Table 2's state rows) and `soi-congressional-district-2022` `charitable_*` (the district file's state-total rows for the same column); national: `soi-table-2-1` `charitable_*` | None for 2022. The two state series are separate IRS tabulations of one line and differ by state (WY `A19700`: $572M in the district file, $1,204M in Historic Table 2). |
 | `db/etl_soi_deductions.py` `medical_claims`, `medical_amount` (state) | `soi-historic-table-2-state-broad-2022` `medical_dental_expense_*` | None for 2022. |
-| `db/etl_soi_deductions.py` `qbi_claims`, `qbi_amount` (state) | `soi-congressional-district-2022` `qualified_business_income_deduction_*` (`N04475`/`A04475`, state rows for all 51 geographies); `soi-historic-table-2-state-broad-2022` `qbi_claims`, `qbi_amount` | None for 2022. The broad package's `qbi_*` read `N03270`/`A03270` on main; chronicle#345 repins them to `N04475`/`A04475`. |
+| `db/etl_soi_deductions.py` `qbi_claims`, `qbi_amount` (state) | `soi-congressional-district-2022` `qualified_business_income_deduction_*` (`N04475`/`A04475`, state rows for all 51 geographies); `soi-historic-table-2-state-broad-2022` `qbi_claims`, `qbi_amount` | None for 2022. chronicle#345 repinned the broad package's `qbi_*` from `N03270`/`A03270` to `N04475`/`A04475`. |
 | `db/etl_soi_income_sources.py` taxable and tax-exempt interest, ordinary and qualified dividends, Schedule C, partnership and S corporation, rental and royalty (national and state) | national: `soi-historic-table-2` (US row); state: `soi-historic-table-2-state-broad-2022` `taxable_interest_*`, `tax_exempt_interest_*`, `ordinary_dividends_*`, `qualified_dividends_*`, `schedule_c_income_*`, `partnership_scorp_income_*`, `rental_royalty_income_*` | None for 2022. |
 | `db/etl_soi_income_sources.py` `short_term_capital_gains_*`, `long_term_capital_gains_*` | `soi-historic-table-2`, `soi-historic-table-2-state-broad-2022` and `soi-table-1-4` publish net capital gain (`net_capital_gains_*`) only | Gap: no packaged table splits gains into short and long term. |
-| `db/etl_soi_income_sources.py` `state_local_refunds_*` | — | Gap: HT2 publishes `N00700`/`A00700`, but no package selects it. |
+| `db/etl_soi_income_sources.py` `state_local_refunds_*` | state: `soi-historic-table-2-state-broad-2022` `taxable_state_local_income_tax_refunds_*` (`N00700`/`A00700`, Schedule 1 line 1) | None at state grain for 2022. Gap nationally: Historic Table 2's and the district file's US rows and Table 1.4's "State income tax refunds" columns publish it, but no package selects them. |
 | `db/etl_soi_income_sources.py` `alimony_received_*` | national: `soi-table-1-4` `alimony_received_*` | Gap at state grain: no package carries state alimony. |
 
 ## Retired: hand-typed values from other publishers

packages/irs_soi/congressional_district_2022/source_package.yaml

@@ -14412,24 +14412,24 @@ record_sets:
     value_scale: 1000
     expected_cell_type: number
   - measure_id: charitable_returns
-    label: Returns with charitable contributions deduction
+    label: Returns with contributions deduction
     ordinal: 38
     column: CR
     source_column_id: N19700
     expected_column_header_row: 1
     expected_column_header: N19700
-    concept: irs_soi.returns_with_charitable_contributions_deduction
+    concept: irs_soi.returns_with_contributions_deduction
     unit: count
     aggregation: sum
     expected_cell_type: number
   - measure_id: charitable_amount
-    label: Charitable contributions deduction
+    label: Contributions deduction amount
     ordinal: 39
     column: CS
     source_column_id: A19700
     expected_column_header_row: 1
     expected_column_header: A19700
-    concept: irs_soi.charitable_contributions_deduction
+    concept: irs_soi.contributions_deduction
     unit: usd
     aggregation: sum
     value_scale: 1000

packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml

@@ -781,6 +781,98 @@ record_sets:
     expected_column_header_row: 1
     expected_column_header: A11070
     value_scale: 1000
+  - measure_id: taxable_state_local_income_tax_refunds_returns
+    label: Returns with taxable refunds of state and local income taxes
+    ordinal: 53
+    column: AG
+    source_column_id: N00700
+    concept: irs_soi.returns_with_taxable_state_local_income_tax_refunds
+    unit: count
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: N00700
+  - measure_id: taxable_state_local_income_tax_refunds_amount
+    label: Taxable refunds of state and local income taxes
+    ordinal: 54
+    column: AH
+    source_column_id: A00700
+    concept: irs_soi.taxable_state_local_income_tax_refunds
+    unit: usd
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: A00700
+    value_scale: 1000
+  - measure_id: taxes_paid_deduction_returns
+    label: Returns with taxes paid deduction
+    ordinal: 55
+    column: CD
+    source_column_id: N18300
+    concept: irs_soi.returns_with_taxes_paid_deduction
+    unit: count
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: N18300
+  - measure_id: taxes_paid_deduction_amount
+    label: Taxes paid deduction amount
+    ordinal: 56
+    column: CE
+    source_column_id: A18300
+    concept: irs_soi.taxes_paid_deduction
+    unit: usd
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: A18300
+    value_scale: 1000
+  - measure_id: mortgage_interest_paid_returns
+    label: Returns with home mortgage interest paid to financial institutions
+    ordinal: 57
+    column: CH
+    source_column_id: N19300
+    concept: irs_soi.returns_with_home_mortgage_interest_paid_to_financial_institutions
+    unit: count
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: N19300
+  - measure_id: mortgage_interest_paid_amount
+    label: Home mortgage interest paid to financial institutions amount
+    ordinal: 58
+    column: CI
+    source_column_id: A19300
+    concept: irs_soi.home_mortgage_interest_paid_to_financial_institutions
+    unit: usd
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: A19300
+    value_scale: 1000
+  - measure_id: charitable_returns
+    label: Returns with contributions deduction
+    ordinal: 59
+    column: CP
+    source_column_id: N19700
+    concept: irs_soi.returns_with_contributions_deduction
+    unit: count
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: N19700
+  - measure_id: charitable_amount
+    label: Contributions deduction amount
+    ordinal: 60
+    column: CQ
+    source_column_id: A19700
+    concept: irs_soi.contributions_deduction
+    unit: usd
+    aggregation: sum
+    expected_cell_type: number
+    expected_column_header_row: 1
+    expected_column_header: A19700
+    value_scale: 1000
 - record_set_id: irs_soi.ty2022.historic_table_2.state_broad.ak
   provenance_class: administrative
   record_set_spec_id: irs_soi.historic_table_2.state_broad_totals.v1

tests/test_chronicle_bundle.py

@@ -383,13 +383,20 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
     # year 2023: no new geography, period or entity keys, duplicate keys or
     # warnings. Its SALT and PTC column fixes, its line 8a relabel and the
     # Historic Table 2 QBI column fix change no count.
+    # Historic Table 2's state package adds Schedule 1 line 1 and Schedule A
+    # lines 7, 8a and 14: 408 TY2022 tax-unit facts (eight measures at 51
+    # states), eight at each state geography. The district file's state-total
+    # rows publish lines 8a and 14 under the same concepts, so 51 states x four
+    # measures add 204 semantic duplicate keys. Measured with the 20 IRS SOI
+    # packages at bundle year 2023, with and without these measures: no new
+    # geography, period or entity keys and no new warnings.
     assert summary["counts"] == {
         "aggregate_duplicate_key_count": 0,
         "entity_count": 12,
         "error_count": 0,
         # chronicle#322 adds 4,947 ONS observations and 3,200 DESNZ ones.
         # Inner/Outer London also add two region-level geography keys.
-        "fact_count": 1007134,
+        "fact_count": 1007542,
         "geography_count": 20548,
         "period_count": 511,
         # 467 before chronicle#292 moved the congressional-district and
@@ -409,7 +416,8 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
         # export: 5,244 NRS + 5,406 UC + 12 DfI overlap groups are retired.
         # Independent census cuts and the older-only combined Metro/Glider
         # journey series remain publisher facts with their original provenance.
-        "semantic_duplicate_key_count": 2324,
+        # Historic Table 2's state lines 8a and 14 add 204 (see above).
+        "semantic_duplicate_key_count": 2528,
         "skipped_source_count": 10,
         "source_count": 56,
         "source_package_count": 377,
@@ -447,7 +455,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
         if row_count == 1:
             assert row["aggregate_fact_key"].startswith("ledger.aggregate_fact.v2:")
             assert row["semantic_fact_key"].startswith("ledger.semantic_fact.v2:")
-    assert row_count == 1007134
+    assert row_count == 1007542
     assert provenance_classes <= {
         "administrative",
         "census",
@@ -468,7 +476,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
         "jct-obbba-revenue-estimates-2025",
         "jct-tax-expenditures-2024",
     ]
-    assert coverage["fact_count"] == 1007134
+    assert coverage["fact_count"] == 1007542
     assert coverage["counts"]["by_source"] == {
         "bea": 446,
         "bfp_economic_outlook": 5,
@@ -494,7 +502,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
         "hmrc": 31515,
         "ici": 12,
         "ird": 4362,
-        "irs_soi": 43963,
+        "irs_soi": 44371,
         "isc": 4,
         "jrc_euromod_be": 90,
         "kff": 52,
@@ -1771,6 +1779,9 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
     # The congressional-district file's Schedule A line 8b, 8c and 9 columns:
     # six measures at its 480 rows.
     expected_period_counts["tax_year:2022"] += 2880
+    # Historic Table 2's state Schedule 1 line 1 and Schedule A lines 7, 8a
+    # and 14: eight measures at 51 states.
+    expected_period_counts["tax_year:2022"] += 408
     # BEA NIPA Table 7.8 line 17 (B4923C): the bundle year's own column.
     expected_period_counts["calendar_year:2023"] += 1
     assert coverage["counts"]["by_period"] == expected_period_counts
@@ -1800,7 +1811,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
     assert coverage["counts"]["by_geography"]["nuts1:BE3"] == 3662
     assert coverage["counts"]["by_geography"]["commune:11002"] == 1
     assert coverage["counts"]["by_geography"]["country:0100000US"] == 2116
-    assert coverage["counts"]["by_geography"]["state:0400000US06"] == 255
+    assert coverage["counts"]["by_geography"]["state:0400000US06"] == 263
     assert (
         coverage["counts"]["by_geography"]["congressional_district:5001700US0601"] == 62
     )
@@ -1841,10 +1852,10 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path):
         "person": 496036,
         "return": 14600,
         "social_protection_scheme": 36,
-        "tax_unit": 45268,
+        "tax_unit": 45676,
     }
     assert not coverage["duplicates"]["aggregate_fact_keys"]
-    assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2324
+    assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2528
     assert Counter(warning["code"] for warning in summary["warnings"]) == {
         "conflicting_geography_name_across_packages": 78,
         "conflicting_groupby_value_label": 23,