Add IRS Historic Table 2 state facts for refunds, taxes paid, mortgage interest and charitable contributions

[MaxGhenis]

Summary

soi-historic-table-2-state-broad-2022 now reads four more column pairs of IRS Historic Table 2 (22in55cmcsv.csv) for each of the 51 states. That is 408 TY2022 facts. Each column is pinned to the definition in the TY2022 documentation guide (22incmdocguide.doc), not to the column-name pattern.

Columns	Guide definition (line)	2022 form line	Measure ids	Concepts
N00700/A00700 (AG/AH)	"state and local income tax refunds" (Schedule 1:1)	"Taxable refunds, credits, or offsets of state and local income taxes"	taxable_state_local_income_tax_refunds_returns / _amount (new)	irs_soi.returns_with_taxable_state_local_income_tax_refunds / irs_soi.taxable_state_local_income_tax_refunds (new)
N18300/A18300 (CD/CE)	"taxes paid" (Schedule A:7)	"Add lines 5e and 6"	taxes_paid_deduction_returns / _amount (new)	irs_soi.returns_with_taxes_paid_deduction / irs_soi.taxes_paid_deduction (new)
N19300/A19300 (CH/CI)	"mortgage interest paid" (Schedule A:8a)	"Home mortgage interest and points reported to you on Form 1098"	mortgage_interest_paid_returns / _amount	Table 2.1's and the district file's (#345)
N19700/A19700 (CP/CQ)	"Total charitable contributions" (Schedule A:14)	"Add lines 11 through 13"	charitable_returns / _amount	Table 2.1's irs_soi.returns_with_contributions_deduction / irs_soi.contributions_deduction

What was there before. No package selected Schedule 1 line 1 or Schedule A line 7 at state grain; #348's docs/us-legacy-etl-cleanup.md lists both as gaps. For lines 8a and 14, the only state values were the district file's state-total rows. Those are a separate IRS tabulation, and they differ from Historic Table 2's state rows: WY line 14 is $572M in the district file and $1,204M in Historic Table 2. This PR adds Historic Table 2's own state rows for both lines.

Labels for the line 8a and line 14 pairs are Table 2.1's. Amounts use value_scale: 1000. Each measure guards its header with source_column_id and expected_column_header.

District concept move. The district package and Table 2.1 used different concept ids for line 14.

• District package: irs_soi.(returns_with_)charitable_contributions_deduction.
• Table 2.1: irs_soi.(returns_with_)contributions_deduction.

Both read the same line. The district file's N19700/A19700 and Table 2.1's "Contributions deduction: Total" (DG/DH) are both Schedule A line 14. So the district charitable_* move to Table 2.1's concept and label, and state, district and national rows join on concept as well as measure_id. measure_id, source_record_id, values and lineage are unchanged. docs/concept-migrations.md records the move.

New ids. No package reads Schedule 1 line 1 or Schedule A line 7 at any grain, so these ids are new.

• taxes_paid_deduction_* follows Table 2.1's naming for its line totals (interest_paid_deduction_* for line 10). Table 2.1 publishes line 7 as "Taxes paid deduction: Total" (CA/CB), which its package does not yet select.
• The refunds ids carry "taxable", following the taxable_* convention. Schedule 1 line 1 reports only the taxable part of these refunds; the guide's wording drops that word.

Evidence

Scripts, downloads, bundle measurements, mutation results and both review outputs are in ~/PolicyEngine/_reviews/chronicle-ht2-state-schedule-a/ on the build Mac.

Publisher files

• 22incmdocguide.doc (sha256 d99db44b…) and 23incmdocguide.doc (94279069…) were downloaded from irs.gov on 2026-10-12. The bytes match the extracts and hashes in tests/data/irs_soi_doc_guides/ from Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345.
• The 2022 Schedule 1 and Schedule A came from irs.gov/pub/irs-prior.
• 22in55cmcsv.csv matches its manifest sha256 (f72f8333…).

California, TY2022 (amounts in thousands of dollars, as published):

Column	Value
N00700	638,320
A00700	1,061,330
N18300	2,836,990
A18300	25,884,637
N19300	2,327,780
A19300	35,409,552
N19700	2,287,850
A19700	34,813,548

Do the columns mean what the guide says? I compared Historic Table 2 with the national publications (computed from the repo's publisher files):

Line	Ratio, TY2022	Ratio, TY2023
8a: 51-state sum / Table 2.1, returns and amount	1.000, 0.996	0.986, 0.986
14: 51-state sum / Table 2.1, returns and amount	0.992, 0.989	0.983, 0.992
7: US row / Table 2.1 "Taxes paid deduction: Total", returns and amount	0.996, 0.999	0.987, 0.989
Schedule 1 line 1: US row / Table 1.4 "State income tax refunds", returns and amount	1.001, 1.027	0.959, 0.994

Observations, kept as published. These are cells as published; no mechanism is claimed.

• Schedule A line 7 adds lines 5e and 6. Even so, in 6 states the published A18300 is below A18460 (line 5e): DC, ME, MD, MT, OH and OR. The largest gap is MD, $20.8M or 0.4%. TY2023 has the same pattern in 8 states.
• The district file's state-total rows differ from Historic Table 2's state rows for these columns. For example, WY A19700 is $572M in the district file and $1,204M in Historic Table 2.
• The tests assert neither relation.

Invariants (stated and tested)

1. Cell preservation. All 408 new facts are checked against an independent csv parse of 22in55cmcsv.csv. Each fact equals its cell × scale and names its source column. The check is exhaustive, not sampled, because the input is finite.
2. Accounting identity. The 51 state facts plus the file's OA and PR rows equal its US row: exactly for amounts, and within 5 × 53 for return counts, which are rounded to tens.
3. Differential. The 51-state sums for lines 8a and 14 are pinned against Table 2.1's TY2022 national facts, to four decimal places.
4. One identity per IRS variable.
   • Every package that reads N19300, A19300, N19700 or A19700 declares Table 2.1's measure id, concept and label.
   • N00700, A00700, N18300 and A18300 each carry one measure id and concept in every package.
5. Guide pins. Every new measure id reads the column its guide definition names, in both the TY2022 and TY2023 guides. These are the existing test_every_measure_reads_the_column_its_guide_definition_names and test_every_definition_names_the_same_variable_in_every_guide.
6. TY2023 forward guard. 23in55cmcsv.csv has the same eight variables at the same column letters, so a TY2023 twin generated from this package reads them unchanged.

Mutation check. I ran seven mutants and the tests killed all seven:

• taxes paid amount moved to A18460;
• district concept reverted;
• value_scale dropped;
• state charitable concept set to the old district id;
• refunds measure id renamed;
• mortgage returns moved to N19500;
• state label changed.

The script and results are in the evidence folder.

Bundle counts

I built build-bundle --year 2023 over the 20 IRS SOI sources at the parent commit and at this head. The differences:

Count	Change
fact_count	43,963 → 44,371 (+408)
tax_year:2022	+408
tax_unit	+408
Each state geography	+8 (CA 255 → 263 in the full bundle)
semantic_duplicate_key_count	+204

The +204 comes from the district file's state-total rows, which publish lines 8a and 14 under the same concepts for the same 51 states: 51 × 4. This is how the two files' shared columns already behave.

Nothing else moved: no new geography, period or entity keys, no warnings and no errors. The full-bundle pins in tests/test_chronicle_bundle.py move by the same amounts: fact count 1,007,134 → 1,007,542 and semantic duplicate keys 2,324 → 2,528.

Consumer effect

Chronicle stores these as raw publisher facts; it activates nothing. What microcosm does with them is microcosm's call on its next re-pin, tracked in PolicyEngine/microcosm#1245. As read at microcosm 7253402fb (analysis in the evidence folder; I grep-verified the mappings):

• Feed. Microcosm's feed builder selects Chronicle record sets and periods, not measures. All 408 facts therefore enter its rebuilt feed, and us/chronicle_feed.json rows and digest move: 40,178 → 40,586 rows if nothing else changes.
• charitable_*. These are the only new ids microcosm maps, to charitable_deduction over itemizers. They become state targets, and in state_cd they replace the charitable sibling bridges with these Historic Table 2 state parents.
• The other six ids. mortgage_interest_paid_* is unmapped on purpose. taxes_paid_deduction_* and taxable_state_local_income_tax_refunds_* map to nothing.
• District concept move. It moves no microcosm target. Microcosm selects by measure_id, and none of its code names either charitable concept id.

TY2023

On main, no TY2023 state package carries the broad columns. soi-historic-table-2-state-agi-2023 holds only return count and AGI, so this PR adds no TY2023 measures. #295 adds a TY2023 state broad twin derived from this TY2022 package, and I left a heads-up there. Its mirror test will expect these eight measures, and they sit at the same columns in 23in55cmcsv.csv.

Dependencies

• Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345 (merged as 8826247b). This PR extends Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345's guide-pinning test and reuses its district mortgage_interest_paid_* ids.
• Retire US legacy ETLs whose values don't trace to publisher bytes #348 (merged as 130b95bd). Commit 7d096e57 updates five rows of Retire US legacy ETLs whose values don't trace to publisher bytes #348's docs/us-legacy-etl-cleanup.md:
  • SALT row: names taxes_paid_deduction_* (line 7) beside the line 5e and 5b measures;
  • mortgage interest and charitable rows: list Historic Table 2's state rows next to the district file's state totals, and note that the two tabulations differ;
  • state_local_refunds_*: covered at state grain; the national gap stays;
  • QBI row: the Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345 note is now in the past tense, at Retire US legacy ETLs whose values don't trace to publisher bytes #348's owner's request.
• Follow-ups (chips, not in this PR):
  • select line 7 and Schedule 1 line 1 nationally (Table 2.1, Table 1.4, Historic Table 2's US row) and in the district file;
  • unify Historic Table 2 and district concept ids for other shared IRS variables (N01700, N06500; N00900/N26270/N25870 overlap IRS SOI Table 1.4: add business, rental, royalty, farm-rental, partnership and S-corp net income/loss legs; give HT2/district 'net income (less loss)' columns their own concept ids #346).

Chronicle Governance

• Approved Chronicle agent role:
  • ledger-source-ingestor
• Deterministic checks run:
  • Source-package validation: validate-package soi-historic-table-2-state-broad-2022 --year 2023: valid, 3,111 facts, no errors or warnings.
  • Source-cell preservation: build-suite: 3,111 facts, all lineaged, 8,476 source cells unchanged. test_every_new_fact_is_its_published_cell checks every new fact exhaustively.
  • Consumer-contract validation: build-suite consumer facts are valid, and so is the 20-source IRS SOI bundle. CI: CI pending on a838ee45 (it passed on the pre-rebase head d2d27d9c: run 38291765431, Chronicle bundle, Chronicle surface and Chronicle checks).
  • Raw-facts boundary validation: build-suite agent acceptance passes: 0 missing lineage or provenance, 0 row-semantic errors. The reviewer reran raw-fact validation for both changed packages: 3,111 state facts and 29,760 district facts.
• LLM judge verdicts. Round 1 was GPT-6.1 Sol via subfleet run --task review --tier hard, at e3bd56e9. Round 2 confirmed both verdicts at 7d096e57, after the rebase onto main and the cleanup-doc commit. The Sol lanes were closed until 04:00Z, so round 2 was an independent Opus 5.5 agent, the fallback in ~/.claude/CLAUDE.md.
  • ledger-source-fidelity: PASS at e3bd56e9 (round 1) and 7d096e57 (round 2). The reviewer independently checked CA, NY, TX, DC and more states against the CSV, the guide hashes and wording, the 2022 form lines, the accounting identity, the Table 2.1 ratios and every bundle delta. "No values are invented or repaired." It raised three nits and no blockers or should-fix findings. All three are fixed in 9261668e, rebased as d2d27d9c: the state-only identity test now checks labels, it accepts matching later readers such as a TY2023 twin, and the bundle comment says what the measurement compared.
  • Round 2 checks, executed. The WY cells; the district file's 51 state-total facts for N19300/A19300/N19700/A19700; the national refunds gap across all four Table 1.4 vintages; Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345's QBI repin; the bundle-pin lineage across a90ad379 → 8826247b → 130b95bd; and 11 out-of-tree mutants against the rebased identity tests.
  • What the rebase onto Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345's final test module changed. Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345 added concept and label to the selector dedup key. The identity tests now also catch a concept or label that differs in only one record set, which they could not before.
  • Round 2 nits. It raised two, both fixed in a838ee45:
    • the identity constant's name: SINGLE_IDENTITY_COLUMNS, test test_line_7_and_refund_columns_keep_one_identity;
    • the refunds row's national gap now also names the district file's US row.
  • Round 2 follow-up (chip). Retire US legacy ETLs whose values don't trace to publisher bytes #348's doc test checks package ids, not the measure ids beside them.
  • ledger-target-profile: n/a (no target profile touched)
  • ledger-contract: n/a (no schema or identity code touched; the concept-id move is a package declaration recorded in docs/concept-migrations.md)
  • ledger-boundary: PASS at e3bd56e9 (round 1) and 7d096e57 (round 2). The change makes source selections and one concept alignment, and adds no reconciliation, aging, imputation, activation, solver construction or PolicyEngine-computed facts. Cross-source comparisons happen only in tests. docs/concept-migrations.md and changelog.d/ are outside the ingestor's literal allowed paths, but they "directly document the authorized source changes".

Tests

• New tests/test_chronicle_soi_ht2_state_schedule_a.py covers:
  • exhaustive cell preservation;
  • the accounting identity;
  • the Table 2.1 differential;
  • the TY2023 same-column guard.
• tests/test_chronicle_soi_doc_guide_columns.py:
  • guide definitions for the two new id pairs;
  • test_schedule_a_lines_shared_across_grains_carry_table_2_1_ids;
  • test_line_7_and_refund_columns_keep_one_identity.
• tests/test_chronicle_source_package.py: CA values for all eight measures and package counts 2,703 → 3,111.
• tests/test_chronicle_bundle.py: full-bundle pins moved by the measured deltas.
• Run locally:
  • the touched SOI modules, governance and poverty-coverage tests: 64 passed;
  • all of tests/test_chronicle_source_package.py plus the consumer-contract, boundary and catalog tests: 182 passed;
  • ruff check: clean.
• Mutants: eight run, all killed (mutation-results.txt). After the rebase onto Read IRS SOI district and Historic Table 2 measures from the columns their guides define #345's d61c069a, which compiles selectors from source-record specs, I reran the seven package mutants and all were killed again (mutation-results-d61c069a.txt).
• The full bundle runs only in CI.

🤖 Generated with Claude Code