Repository navigation
Read IRS SOI district and Historic Table 2 measures from the columns their guides define - #345
Conversation
The IRS documentation guides define each geographic-file column. Four mappings read a column the guide defines as something else: - congressional_district_2022 limited_state_local_taxes_* read N18425/A18425 (state and local income taxes, Schedule A line 5a); now N18460/A18460 (limited state and local taxes, line 5e), as Historic Table 2 reads. - premium_tax_credit_* read N85530/A85530 (additional Medicare tax, Form 8959 line 24); now N85770/A85770 (total premium tax credit). - interest_paid_deduction_* (Table 2.1's line 10 total) read N19300/A19300, line 8a mortgage interest. Renamed to Table 2.1's line 8a ids and concepts (mortgage_interest_paid_*), with new line 8b, 8c and 9 measures. - historic_table_2 and historic_table_2_state_broad_2022 qbi_* read N03270/A03270 (self-employed health insurance); now N04475/A04475. tests/test_chronicle_soi_doc_guide_columns.py pins every measure id these packages declare to its guide definition (extracts in tests/data/irs_soi_doc_guides), with differential checks against Historic Table 2 and Table 2.1 and the $10,000 line 5e cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nsumers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PolicyEngine/chronicle#345 relabels the district file's line 8a column to mortgage_interest_paid_* and adds lines 8b, 8c and 9 under Table 2.1's own ids. The column-keyed rule binds the relabeled 8a rows unchanged and the new ids stay unmapped, so nothing double-binds. Also drop an unused import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups (GPT-6.1 Sol, APPROVE with one P2 and two P3s): - The guide check read only measure.column. The build reads row.column or measure.column and row-level header guards, so the test now walks the compiled source-record specs. Row ranges must stay in one column, and a header guard must sit on the file's header row. A new mutation test moves one Historic Table 2 row to column T and expects the check to name it. - The $10,000 line 5e bound now allows the guides' rounding: counts to the nearest ten, amounts to thousands. - The guide extracts name the "Selected Income and Tax Items" table by title. The Historic Table 2 and county guides call it section F in their contents and G in their body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Independent review (GPT-6.1 Sol via Subfleet, read-only lane; job ledger-source-fidelity: PASS Reviewed requested commit
Executed verification:
Static assessment: unknown IDs, missing guides, new filenames/vintages, and undocumented columns fail closed for discovered The full persisted bundle test was not executed. Duplicate-key and warning claims were assessed from identity construction and unchanged warning inputs. The production changes only remap or relabel publisher cells and normalize their units; test-only comparisons and sums do not create computed Chronicle facts. |
Re-review of d61c069 (GPT-6.1 Sol, APPROVE, three P3s): - An explicit source_column_id equal to the measure id looked like the layout's fallback. The check now takes the declared id from the compiled record set (row before measure), so it is checked against the header. - Deduplication could drop a selector whose concept or label differed. Both are now in the key, and a measure id with more than one concept or label in a package fails. - Divisor selectors were not inspected. These packages publish levels, so a divisor column now fails until the test checks divisors. The mutation test is parametrized over all four cases on a temp copy of historic_table_2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re-review (GPT-6.1 Sol via Subfleet, read-only lane; job ledger-source-fidelity: PASS Reviewed Three nonblocking coverage issues remain:
Executed verification:
One wording correction: Historic Table 2’s contents and body disagree on F/G; the cached county guide uses F in both. The committed title-only headers remain accurate. |
|
Confirmation review (GPT-6.1 Sol via Subfleet, read-only lane; job ledger-source-fidelity: PASS Reviewed
|
|
Merged as |
Summary
The IRS documentation guide for each SOI geographic file (Historic Table 2, congressional district, county) defines every CSV column. Four Chronicle measures read a column that the guide defines as a different item. This PR points each one at the column its guide defines, and adds a regression test that pins every
measure_idin these packages to its guide definition.soi-congressional-district-2022limited_state_local_taxes_returns/_amountN18425/A18425, "State and local income taxes", Schedule A:5aN18460/A18460, "Limited state and local taxes", Schedule A:5epremium_tax_credit_returns/_amountN85530/A85530, "Additional Medicare tax", 8959:24N85770/A85770, "Total premium tax credit", 8926:24 (8962:24 in the TY2023 guide)interest_paid_deduction_*→mortgage_interest_paid_*N19300/A19300, "Mortgage interest paid", Schedule A:8a, under Table 2.1's line 10 idssoi-historic-table-2,soi-historic-table-2-state-broad-2022qbi_claims/qbi_amountN03270/A03270, "Self-employment health insurance deduction", Schedule 1:17N04475/A04475, "Qualified business income deduction", 1040:13N18460andN85770for the same measure ids. The column letters,source_column_idandexpected_column_headerguards all move (BX/BY → CF/CG, EX/EY → DV/DW).interest_paid_deduction_*for the Schedule A line 10 total. The 2022 Schedule A computes line 10 as lines 8a, 8b, 8c and 9 combined (8d is reserved), and no geographic file publishes line 10. The district line 8a columns now carry Table 2.1's line 8a ids. The other three parts are added with Table 2.1's ids, concepts and labels:home_mortgage_personal_seller_*:N19500/A19500, line 8b;deductible_points_*:N19530/A19530, line 8c;investment_interest_paid_*:N19570/A19570, line 9.qbi_*measure, so no target moves.What the IRS files define
Guides. Each guide's section G table is extracted to
tests/data/irs_soi_doc_guides/<guide>.tsv, with the irs.gov URL, sha256 and extraction method in its header. I downloaded each file from irs.gov today; the bytes match the copies in~/PolicyEngine/_reviews/cg-returns-control-20260925/irs/.22incddocguide.docx22incd.csv(congressional district TY2022)4e265d3c…22incmdocguide.doc22in55cmcsv.csv(Historic Table 2 TY2022)d99db44b…23incmdocguide.doc23in55cmcsv.csv(Historic Table 2 TY2023)94279069…22incydocguide.docx22incyallnoagi.csv(county TY2022)a6acb748…A00101in22incd.csvand22incyallnoagi.csv, which no package reads.N/A18425andN/A18450, rewordsN/A11070("additional child tax credit"), and prints the premium tax credit lines as 8962:24 and 8962:25 where the TY2022 guides print 8926.Schedule A (2022) (
f1040sa--2022.pdffrom irs.gov, sha2561e2e4ee7…):Publisher cells, TY2022 US (district
STATE=US, agi_stub=0; Historic Table 2STATE=US, AGI_STUB=0; Table 2.1 row "All returns, total"):A19300vs Table 2.1 line 8a (mortgage_interest_paid_amount)A19300vs Table 2.1 line 10 (interest_paid_deduction_amount)A19300 + A19500 + A19530 + A19570vs Table 2.1 line 10A18460vs Table 2.1 limited SALTA18425/N18425qbi_amountChronicle effects (measured)
Row-level diff. I built
build-bundle --year 2023over all 20 IRS SOI packages onmainand on this branch (41,083 → 43,963 facts) and compared consumer rows onsource_record_id:limited_state_local_taxes_*rows per measure and 480premium_tax_credit_amountrows.premium_tax_credit_returnschanges on 479: in TN-05,N85530andN85770are both 20,780. Values also change on the 62 Historic Table 2qbi_*rows per measure (11 national, 51 state). On all of these,layout.source_column_idandlineage.source_cell_keyschange with the value.interest_paid_deduction_*rows becomemortgage_interest_paid_*rows with the same values. Theirsource_record_id,measure_id, concept, label and keys change.layout.record_set_spec_hashchanges on every row of the three edited packages, because it hashes the whole record-set spec. No other field changes.Bundle counts. Measured with the same 20-package build:
irs_soiby source,tax_unitby entity,tax_year:2022by period, and the "Congressional District Data 2022" table count (26,880 → 29,760).tests/test_chronicle_bundle.pymove by those amounts: fact and row count 1,004,254 → 1,007,134; US 2,110 → 2,116; California 249 → 255; CA-01 56 → 62. CI's bundle job checks them; the full default bundle doesn't fit on this host.Package validation.
validate_source_packageis valid at--year 2022and2023for all three edited packages: 29,760, 605 and 2,703 source records.Consumers (Microcosm)
I read Microcosm
origin/main51a263726. Its SOI compiler selects facts bymeasure_id:_measure_id(fiscal_targets.py:4074) readsobserved_measure.source_measure_id._soi_reference_from_fact(:3069) looks that id up inSOI_AMOUNT_MEASURE_VARIABLES/SOI_RETURN_MEASURE_VARIABLES(:113,:199).Noneat:3101, so the fact is skipped.Nothing changes in Microcosm until its Chronicle feed is re-pinned (
us/chronicle_feed.json). This PR does not re-pin it. On the next re-pin:interest_paid_deduction_*rows map tointerest_deduction(:125,:210). After the rename they aremortgage_interest_paid_*, which Microcosm leaves unmapped on purpose. Its comment at:176reads: "the two leg measures (mortgage_interest_paid_, home_mortgage_personal_seller_) stay unmapped to avoid double-counting".main, the district interest targets would drop out.state_cdbridges forinterest_paid_deduction_*(STATE_CD_LEVEL_BRIDGES,tools/us_acs_local_cd_surface.py:90-97) would stop matching.layout.source_column_idN19300/A19300, compiled as a share, so the rename is safe as long as those columns stay. They do: the 960 rows keepN19300/A19300and their values.N18460/A18460andN85770/A85770. microcosm#1224'sUS_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONSis keyed on (measure id,N18425/A18425/N85530/A85530), so it stops matching them, and the corrected rows compile.main,_is_soi_cd_premium_tax_credit_amount_conflict(:2953) drops every districtpremium_tax_credit_amountrow, whatever its column. #1224 removes that check.test_pinned_feed_exclusion_receipt, 4 × 488 ids) and its 31,786-spec pin need updating on re-pin.STATE_CD_DEFECTIVE_CD_FILE_MEASURES(tools/us_acs_local_cd_surface.py:53-68) is keyed on measure id alone, so it keeps the corrected rows offstate_cduntil that register is edited.qbi_*entry, andtest_soi_qbi_historic_table_2_measures_are_not_direct_targetsguards that. No target moves.TY2023 Historic Table 2 (#295)
#295 adds TY2023 national, state-broad and state-EITC packages. Its QBI measures already read
N04475/A04475. Itstest_packages_mirror_their_2022_twinscarries aTY2022_QBI_CORRECTIONshim for the wrong TY2022 twin, and once this PR lands the twin matches.Simulation: this branch with #295's packages and tests overlaid.
assert variable == wrongno longer holds;pr295-qbi-followup.patch, posted on Add IRS SOI Historic Table 2 TY2023 national, state broad and state EITC facts #295), all 54 tests intest_chronicle_soi_ht2_2023.py,test_chronicle_soi_doc_guide_columns.py,test_chronicle_soi_capital_gain_concepts.pyandtest_chronicle_soi_state_agi_2023.pypass.Chronicle governance
ledger-source-ingestor. Three paths sit outside that role'sallowed_paths:docs/concept-migrations.md(the rename notedocs/architecture.mdrequires),changelog.d/andtests/data/irs_soi_doc_guides/.validate_source_packagefor the three edited packages at 2022 and 2023: valid.mainand this branch: valid, 0 errors, 0 aggregate-duplicate keys, with the row diff above.ruff check chronicle policyengine_chronicle db scripts tests(locked ruff 0.14.10): clean.test_chronicle_source_package.py, the SOI and all-package suites listed under Tests.a90ad379:Chronicle bundlepassed (1h56m), confirming the default-bundle pins, andChronicle surfacepassed (47m). CI reruns on the follow-up head.a90ad379; posted below):ledger-source-fidelity: PASS. It re-extracted all four guides withtextutil(582/582 TSV rows match), verified the guide and CSV hashes and every changed declaration's header, and built the district facts in memory (29,760).ledger-boundary: PASS. The production change only remaps or relabels publisher cells; the test-only sums create no Chronicle facts.ledger-contract: no schema or consumer-contract change.d61c069a. The re-review ofd61c069awas APPROVE, PASS/PASS, with three P3 coverage gaps, all fixed inc542be25. The confirmation review ofc542be25, the current head, is APPROVE with no new findings. It ran all 42,720 compiled records across the seven packages and repeated every mutation.Invariants (
tests/test_chronicle_soi_doc_guide_columns.py)Every invariant below is checked by enumerating every package, measure and row, not by sampling.
A00101). Every guide variable is a column of its file.GUIDE_DEFINITIONSentry names exactly one variable, and the same one in every guide.irs_soipackage that parses a delimited IRS-variable file, in any vintage. It reads the compiled source-record specs, so row-level column overrides, row ranges and row-level header guards count as the build reads them.source_column_idandexpected_column_header, where declared, must equal that header.measure_id, an undocumented column, a data file with no guide, or a package withoutartifact_yeareach fail.historic_table_2, and the check names each one. They cover a row moved to column T (A00100), an explicit column id equal to the measure id, a divisor, and a second record set with anotherqbi_amountconcept. The reviewer's own mutations were caught the same way: an Autauga County row moved to column V with matching guards, a county divisor, and a duplicated district record set with the old line 10 concept.interest_paid_deduction_*may not be read from them and stays declared by Table 2.1.On
mainthe module fails 12 of its 20 tests. Invariant 3 names all ten wrong declarations in the guide's wording, and the line 5e cap fails on the district package independently of the YAML pins.Not changed here
historic_table_2declare only a column letter. Their consumer rows carry the measure id aslayout.source_column_id(for examplepremium_tax_credit_returns). The new test resolves their letters against the file's header, so a column shift fails CI. Addingsource_column_idguards would change those rows'layout.source_column_id, which microcosm#1224 reads, so it belongs with a Microcosm re-pin.charitable_*(districtirs_soi.charitable_contributions_deductionvs Table 2.1irs_soi.contributions_deduction, both Schedule A line 14). Alsoctc_*readsN07225, which the guides define as "nonrefundable child and other dependent tax credit", under the conceptirs_soi.child_tax_credit. Noted on IRS SOI measure_id net_capital_gains_* names two different measures (Form 1040 line 7 vs Table 1.4 Schedule D gain) #307.Tests
tests/test_chronicle_soi_doc_guide_columns.py(20 tests) andtests/data/irs_soi_doc_guides/*.tsv(4 guide extracts).test_soi_congressional_district_2022_builds_all_return_facts: 29,760 facts, plus the corrected SALT, PTC and line 8a values and columns, the new parts, and nointerest_paid_deduction_*.A04475.c542be25.test_chronicle_source_package.py,test_source_package_alias_drift.py,test_chronicle_artifact_year_restamp.py,test_issue_265_non_uk_labels.py,test_chronicle_issue_272.py,test_chronicle_governance.py,test_chronicle_facts_only.py,test_chronicle_boundaries.py,test_chronicle_soi_targets.py,test_etl_soi.py,test_etl_soi_state.py,test_us_poverty_target_coverage.py,test_issue_261_dimension_labels.pyandtest_chronicle_source_label_authority.py: 323 passed. The default bundle and the full suite run in CI.Review follow-ups (
d61c069a)measure.column, but the build readsrow.column or measure.columnand row-level guards. The test now walks the compiled source-record specs, and adds the mutation test above.Re-review follow-ups (
c542be25)source_column_idequal to the measure id looked like the layout fallback. The declared id now comes from the compiled record set.axiom: n/a: source-data column mapping, no policy rule change.
🤖 Generated with Claude Code