diff --git a/changelog.d/1038-cd-interest-paid-mortgage-shares.fixed.md b/changelog.d/1038-cd-interest-paid-mortgage-shares.fixed.md new file mode 100644 index 000000000..9a7a82b83 --- /dev/null +++ b/changelog.d/1038-cd-interest-paid-mortgage-shares.fixed.md @@ -0,0 +1 @@ +Bind the congressional-district file's interest-paid rows as mortgage interest shares. 22incd.csv publishes Schedule A line 8a, not the line 10 total its measure ids name, so the rows summed to 0.673x the national line 10 target. Each now binds `deductible_mortgage_interest` as its share of the file's US row times Table 2.1's home mortgage interest control (8a + 8b). Any other column under those ids refuses the compile. The ACS-local `state_cd` surface binds these state rows as compiled instead of bridging them, which brings back Wyoming and South Dakota. diff --git a/docs/us-acs-local-soi-target-surface.md b/docs/us-acs-local-soi-target-surface.md index e07236256..c0e93ea15 100644 --- a/docs/us-acs-local-soi-target-surface.md +++ b/docs/us-acs-local-soi-target-surface.md @@ -241,7 +241,7 @@ defect inside the file. On the pinned feed all 2,193 blocks agree to float precision (largest relative gap 4e-16; the receipt records it under `internal_consistency`, and the feed-gated test pins it). -On the pinned feed the band drops eight: +On the pinned feed the band drops six: | State | Measure | Basis | Ratio | Median | Relative | |---|---|---|---:|---:|---:| @@ -249,34 +249,54 @@ On the pinned feed the band drops eight: | Hawaii | `ordinary_dividends_amount` | rebase | 2.17 | 1.07 | 2.03 | | New York | `rental_royalty_income_amount` | rebase | 0.72 | 1.05 | 0.68 | | Utah | `tax_exempt_interest_amount` | rebase | 1.43 | 1.09 | 1.32 | -| Wyoming | `charitable_amount`, `interest_paid_deduction_amount` | bridge (itemized) | 1.75 | 1.06 | 1.65 | -| South Dakota | `charitable_amount`, `interest_paid_deduction_amount` | bridge (itemized) | 1.44 | 1.06 | 1.36 | +| Wyoming | `charitable_amount` | bridge (itemized) | 1.75 | 1.06 | 1.65 | +| South Dakota | `charitable_amount` | bridge (itemized) | 1.44 | 1.06 | 1.36 | -That removes 34 district rows (Hawaii 2 x 2, New York 26, Utah 4) and four +That removes 34 district rows (Hawaii 2 x 2, New York 26, Utah 4) and two bridged state rows (Wyoming and South Dakota are at-large, so they had no -district rows). The largest gaps the band keeps are 1.22x (Wisconsin +district rows). Their `interest_paid_deduction_amount` rows crossed the same +bridge and were dropped too, until microcosm#1038 bound them as national +shares (below). The largest gaps the band keeps are 1.22x (Wisconsin partnership and S-corporation income, West Virginia capital gains) and 1.20x (Mississippi capital gains); every other kept block is within 1.17x. So on this feed any tolerance above the largest kept gap (1.220) and below Utah's -(1.316) drops the same eight. The band +(1.316) drops the same six. The band is a reviewed tolerance, not an estimate; changing it moves the contract counts below, which a feed-gated test pins. -The six measures only the district file has (`charitable_*`, +Six measures only the district file has (`charitable_*`, `interest_paid_deduction_*`, `qualified_business_income_deduction_*`) have no -Historic Table 2 parent. Their parent is the district file's own state row, -**lifted onto the Historic Table 2 basis** by a sibling concept both files -carry in the same state (`STATE_CD_LEVEL_BRIDGES`): +Historic Table 2 parent. Their parent is the district file's own state row. +Four of them are **lifted onto the Historic Table 2 basis** by a sibling +concept both files carry in the same state (`STATE_CD_LEVEL_BRIDGES`): - counts use `return_count`; -- charitable and interest-paid amounts use `itemized_deductions_amount`; +- charitable amounts use `itemized_deductions_amount`; - QBI deduction amounts use `adjusted_gross_income`. +The other two need no bridge. `interest_paid_deduction_*` reads Schedule A +line 8a, and the compiler binds it as `deductible_mortgage_interest`: each +row's share of the file's US row, times Table 2.1's home mortgage interest +control (docs/us-soi-interest-paid-concepts.md). Such a row names its share's +denominator (`soi_share_total_source_record_id`), and its states already sum +to the control, so it has neither the coverage gap nor the stamp the bridge +corrects. `state_cd` binds its state row as compiled +(`state_cd_level_basis = national_control_share`), and its districts are +rebased to it with basis `cd_file_state_total_national_share`, at factor 1 to +float precision. Bridging it as well would have lifted its amounts about 6%, +and its counts about 2%, a second time (the bridges' medians). It would also have tied it to the band's verdict on itemized +deductions, which Wyoming and South Dakota fail because of charitable +giving: their HT2/CD charitable ratios are 2.10 and 1.98, while their line +8a ratios are 1.007 and 1.028. On the pinned feed the 51 interest-paid state +rows sum to the national control, $186.31B and 11,644,348 returns aged to +2024. Before, 49 bridged amount rows summed to $161.58B on +`interest_deduction`, and 51 count rows to 11,514,598. + The bridge assumes the measure shares its sibling's Historic Table 2 / district-file ratio in that state: the coverage and aging gap above, plus whatever the two publications disagree about for the sibling. It is an estimate, not an identity, which is why the factor band applies to it too. -Without it these state rows and their 2,562 district rows would sit 2–6% +Without it these state rows and their 1,708 district rows would sit 2–6% below every related concept. On the pinned feed the bridge factors are 1.00–1.03 for counts (median 1.016), 1.03–1.75 for the itemized-deduction amounts (median 1.062; the two above 1.25x the median are dropped) and @@ -299,7 +319,7 @@ bridged state row records its sibling and bridge factor. | Historic Table 2 rows copied to at-large districts | 360 | The same copies from the other vintage | | District-file state rows for concepts Historic Table 2 carries | 2,295 | Second vintage of a state concept | | District rows of the four out-of-band rebase blocks | 34 | The factor band (above) | -| Bridged state rows whose sibling ratio is out of band | 4 | The factor band (above) | +| Bridged state rows whose sibling ratio is out of band | 2 | The factor band (above) | PR #1040 (awaiting a ruling) excludes the same SALT and PTC columns in the compiler and rescales the district file's capital-gains rows by one national @@ -323,14 +343,17 @@ change forces the same review. | `usda_snap` | 102 | | `cms_medicaid` (enrollment) | 51 | | `irs_soi` state, Historic Table 2 | 4,227 | -| `irs_soi` state, district file (district-file-only measures) | 302 | +| `irs_soi` state, district file (district-file-only measures) | 304 | | `irs_soi` district (427 districts x 51 measures, less 34 banded) | 21,743 | -| **Admin specs** | **26,425** | +| **Admin specs** | **26,427** | Of the 21,743 district rows, 19,181 are rebased to a Historic Table 2 parent -and 2,562 keep a district-file parent. The 2,189 (state, concept) district -blocks each sum to their parent within 1e-9. Adding the 487 population -marginals gives 26,912 targets, before the holdout. The Historic Table 2 +and 2,562 keep a district-file parent: 1,708 bridged and 854 national shares +(interest paid). The 2,189 (state, concept) district blocks each sum to their +parent within 1e-9. Adding the 487 population marginals gives 26,914 +targets, before the holdout. Before microcosm#1038 bound interest paid as +national shares, the district-file state count was 302, with 26,425 admin +specs and 26,912 targets. The Historic Table 2 state count includes the 408 state AGI-band rows of #940 (3,819 and 26,017 admin specs before the 2026-10-04 re-pin). The feed-gated test `test_pinned_feed_state_cd_surface_matches_its_contract` pins these counts diff --git a/docs/us-fact-to-target.md b/docs/us-fact-to-target.md index e4ae417a8..ab1df4730 100644 --- a/docs/us-fact-to-target.md +++ b/docs/us-fact-to-target.md @@ -186,6 +186,32 @@ binds them: band. `experiments/940-state-agi-bands/` measures pre-calibration support per state and band on the #982 fix's offline outputs. +### District-file components bind as shares of a national control (microcosm#1038) + +The congressional-district file sometimes publishes one part of a concept +under the measure id of the whole. Its interest-paid column N19300/A19300 is +Schedule A line 8a, but Chronicle publishes it as `interest_paid_deduction_*`, +whose Table 2.1 rows are the line 10 total. `_SOI_DISTRICT_COMPONENT_SHARE_COLUMNS` +registers such columns by IRS column id, with the model concept the part +measures well and the national control of that concept: + +- **Which rows.** Every district-file fact that reads a registered column, + whatever its measure id, compiles onto the registered variable (line 8a + onto `deductible_mortgage_interest`). A district-file fact of a measure the + file has no column for (`_SOI_DISTRICT_MEASURES_WITHOUT_A_COLUMN`) that + reads another column, or names none, raises + `UnreviewedSoiDistrictColumnError`. +- **The value.** `_rebase_soi_district_component_shares` sets + `control x row / US row`. The US row is the same record set's national row + of the same column; the control is Table 2.1's latest home mortgage interest + row not after the build period. Ties raise, the US row retires, and a row + with no anchor is dropped, never shipped as a level. +- **Receipts.** `soi_source_concept`, `soi_control_concept`, + `soi_share_total_source_record_id`, `soi_source_column_id` and + `soi_return_universe_bridge`, beside the usual `uprating_*` fields. + +docs/us-soi-interest-paid-concepts.md has the measurements behind the choice. + ## 4. Keep the exclusion register honest `US_FISCAL_TARGET_SUPPORT_EXCLUSIONS` is keyed by `source_record_id` and its diff --git a/docs/us-soi-interest-paid-concepts.md b/docs/us-soi-interest-paid-concepts.md new file mode 100644 index 000000000..57869f278 --- /dev/null +++ b/docs/us-soi-interest-paid-concepts.md @@ -0,0 +1,205 @@ +# Congressional-district interest paid: what the column counts and what it binds + +The congressional-district SOI file has no column for the itemized interest +deduction (Schedule A line 10). Chronicle's `congressional_district_2022` +package published one of its parts under the line 10 measure ids, and the +compiler bound it as the whole. This page records what each source counts, +the three remedies we measured, and the one `fiscal_targets.py` applies +(microcosm#1038). + +## What each source counts + +| Source | Columns or measure ids | What it counts | +|---|---|---| +| District file (`22incd.csv`, `22incddocguide.docx`) | N19300 / A19300 | "Mortgage interest paid", Schedule A line 8a | +| District file | N19500 / A19500 | "Home mortgage from personal seller", line 8b | +| District file | N19530 / A19530 | "Deductible points", line 8c | +| District file | N19570 / A19570 | "Investment interest paid", line 9 | +| Table 2.1 (feed, TY2023) | `interest_paid_deduction_*` | The line 10 total: $208.18B on 11,944,817 returns | +| Table 2.1 | `home_mortgage_interest_*` | Total home mortgage interest: $171.36B on 11,644,348 returns, exactly line 8a ($167.68B) plus 8b ($3.69B) | +| Table 2.1 | `mortgage_interest_paid_*`, `home_mortgage_personal_seller_*`, `deductible_points_*`, `investment_interest_paid_*` | Lines 8a, 8b, 8c and 9 | + +Chronicle's district-file package reads N19300/A19300, line 8a, under +`interest_paid_deduction_{returns,amount}`. Its concept label, +`irs_soi.interest_paid_deduction`, is Table 2.1's line 10 concept. The file +carries the other three parts too, but the package does not publish them. + +In the file's US row, line 8a is $140.14B of the $163.71B that lines 8a + 8b ++ 8c + 9 sum to: 85.6%. It is 98.2% of 8a + 8b. + +## What the model measures + +- `interest_paid_deduction_*` maps to PE-US `interest_deduction`. That + variable adds `deductible_mortgage_interest` and `non_mortgage_interest` + (policyengine-us 2.2.1), so it is the line 10 concept. +- `home_mortgage_interest_*` maps to `deductible_mortgage_interest`, read as + `deductible_mortgage_interest_tax_unit`, the home mortgage interest left + after the acquisition-debt caps (microcosm#511). +- Every row of either measure is masked to itemizers (`itemized_only`, from + `_SOI_ITEMIZED_ONLY_VARIABLES`), whatever universe its record set names. + The district file's rows and Table 2.1's therefore materialize over the + same tax units. + +## What main did + +The district file's state, district and US rows compiled onto +`interest_deduction` as line 10 levels. On the pinned feed (`51aa40f`, +compiled at 2024 with aging and the packaged crosswalk), on the `full` +surface: + +| | District-file states (51), and districts (436), each sum | National Table 2.1 line 10 target | Ratio | +|---|---:|---:|---:| +| Amount | $152.36B | $226.33B | 0.673 | +| Returns | 11,352,500 | 11,944,817 | 0.950 | + +The amount gap is mostly concept (8a against line 10) and vintage: the file +is TY2022 data stamped 2023, set against Table 2.1 TY2023. + +## Three remedies, measured + +Every number below comes from the sha-matched `22incd.csv` (`137522878a...`) +and Historic Table 2 TY2022 (`22in55cmcsv.csv`). Scripts and outputs are in +`~/PolicyEngine/_reviews/cd-interest-paid-concept-20261012/`. + +**(a) Line 10 shares.** Each row would bind `interest_deduction` as its share +of the file's US row times the Table 2.1 line 10 control. This assumes a +district's share of line 8a equals its share of line 10. Investment interest +is concentrated where the assumption fails. Line 8a is 38.8% of line 10 in +FL-21, and 44.9% and 45.0% in NY-10 and NY-12. Across the 436 district units +of the 117th plan, a district's target would be off from its line 10 share +by: + +- a median of 7.3%, a 95th percentile of 17.4%, and at most 54.6% (FL-21); +- 4.66% of the national total misallocated between districts (half the sum + of absolute share differences), about $10.5B of the aged control. + +At state level it is off by a median of 5.3% and at most 29.8% (Wyoming), +with 3.07% of the total misallocated. Against Historic Table 2's own state +line 10 totals, an independent publication, it is off by at most 56.1% +(Wyoming). + +**(b) Drop the rows.** Interest would keep only Table 2.1's two national +rows. That discards 974 district-file targets whose information is almost +all mortgage interest, which the file measures well. Mortgage interest is +unevenly spread: a district's share of line 8a differs from its share of +returns by 30.3% of the total (half the sum of absolute share differences), +from its share of AGI by 22.2%, and from its share of itemized deductions by +14.0%. Option (c)'s proxy error is 0.21% on the same scale. + +**(c) Mortgage interest shares (applied).** Each row binds +`deductible_mortgage_interest` as its share of the file's US row times Table +2.1's home mortgage interest control (8a + 8b). This assumes a district's +share of line 8a equals its share of 8a + 8b. Line 8b is 1.8% of 8a + 8b +nationally, so the share moves little: + +- districts: a median of 0.45%, a 95th percentile of 1.4%, at most 2.4% + (CA-35), and 0.21% of the total misallocated; +- states: a median of 0.38%, at most 1.3% (West Virginia), and 0.17% + misallocated; +- against Historic Table 2's state 8a + 8b totals: a median of 0.58% and at + most 3.0% (West Virginia), with 0.29% misallocated. That includes the + district file's coverage gap, which is 1.7% for 8a (CD/HT2 0.983, WV + 0.954). + +The counts proxy is as close. At state level, a state's share of the file's +N19300 differs from its share of Historic Table 2's N19300 + N19500 by a +median of 0.43%, at most 2.8% (North Dakota). + +So (c) keeps the geography (a) and (b) would lose or distort, at about a +twentieth of (a)'s error. The concepts it leaves without district targets, +points (8c) and investment interest (line 9), are 17.7% of line 10 +nationally. Table 2.1's line 10 row still binds their national total. + +## How the rule works + +`_SOI_DISTRICT_COMPONENT_SHARE_COLUMNS` in `fiscal_targets.py` registers +N19300 and A19300, keyed by the IRS column rather than the measure id: + +- **Reference.** `_soi_reference_from_fact` sends any district-file fact + that reads a registered column to `deductible_mortgage_interest`, as a + count for N19300 and a sum for A19300, with role + `soi_fiscal_distribution`. It flags the fact for the rebase. A Chronicle + package that relabels the column, for example to Table 2.1's + `mortgage_interest_paid_*` ids, compiles to the same targets. +- **Refusal.** A district-file `interest_paid_deduction_*` fact that reads + any other column, or names none, raises `UnreviewedSoiDistrictColumnError`. + The file has no line 10 column, so such a fact is unreviewed by + construction. +- **Rebase** (`_rebase_soi_district_component_shares`). Each flagged row + becomes `control x row / US row`: + - The US row is the same record set's national all-AGI row of the same + column, filing status and period. + - The control is the latest national all-AGI `home_mortgage_interest_*` + fact in the itemized-returns universe not after the build period, from + outside the district files. + - Two different controls, or two different US rows, at the deciding period + raise. Feed order never decides (microcosm#1035). + - A row with no control at or after its own period, or no nonzero US row, + is dropped, never shipped as a level. + - The file's US row retires, because Table 2.1's row owns the national + concept. +- **Declarations.** Every kept row carries: + - `soi_source_concept = schedule_a_line_8a_home_mortgage_interest_paid_to_financial_institutions`; + - `soi_control_concept = schedule_a_lines_8a_8b_total_home_mortgage_interest`; + - `soi_share_total_source_record_id`, the US row; + - `soi_source_column_id`; + - `soi_return_universe_bridge = itemized_returns`; + - the `uprating_*` fields every rebase carries. +- **Periods.** The value lands at the control's period, and target aging + gives it the control's own remaining link, the CBO AGI series for amounts. + The file's ty2023 stamp cancels in the share: the same package stamped + 2022 gives the same targets. + +### Invariants + +These hold for every fact set the compile accepts, and +`test_us_soi_district_interest_shares.py` checks them, including with +Hypothesis: + +- No district-file row binds `interest_deduction`. +- One factor scales every row of a record set, column and filing status, so + districts keep their within-state shares. +- Where the file's states sum to its US row, as they do on the pinned feed, + its states sum to the control, and so do its districts. +- No row carries the rebase flag after the compile. +- Rows are nonnegative. +- The output does not depend on feed order, or on a common scale of the + file's values. + +## Effect on the pinned feed + +`full` surface, compiled at 2024 with aging and the packaged crosswalk: + +| | Before | After | +|---|---:|---:| +| District-file amount rows: 51 states, and 436 districts, each sum | $152.36B on `interest_deduction` (0.673x line 10) | $186.31B on `deductible_mortgage_interest` (1.000x the national home mortgage target) | +| District-file return rows, each sum | 11,352,500 (0.950x) | 11,644,348 (1.000x) | +| California amount | $38.13B | $46.63B | +| Rebase factor, amount / returns | — | 1.222854 / 1.025708 | +| Compiled specs | 33,250 | 33,248 (the file's two US rows retire) | +| `national_state` | 6,102 at `e1f9906bf0e0` | unchanged | + +No other spec changes in name, value or metadata. The amount factor includes +TY2022-to-TY2023 growth in mortgage interest: Table 2.1 TY2023 line 8a is +$167.68B, against the file's TY2022 $140.14B. + +### The ACS local `state_cd` surface + +`state_cd` used to lift the district file's interest-paid state rows onto +the Historic Table 2 basis with the itemized-deductions bridge (amounts) and +the return-count bridge (counts). The bridges corrected the file's coverage +gap and its stamp. A share of a national control has neither: its states +already sum to the control. Bridging it as well would lift its amounts about +6%, and its counts about 2%, a second time (the bridges' medians). So +`us_acs_local_cd_surface.py` +binds a district-file state row that names its share's denominator +(`is_national_share_spec`) as compiled, as its districts' parent, with basis +`cd_file_state_total_national_share`. `STATE_CD_LEVEL_BRIDGES` no longer +lists interest paid, so a row the compiler failed to rebase would refuse the +surface rather than bind unbridged. + +The bridge was also the wrong instrument here. Wyoming and South Dakota fell +out of its factor band because their charitable deductions differ between +the two files: HT2/CD A19700 is 2.10 and 1.98. Their mortgage interest +agrees, at 1.007 and 1.028. Both states' interest-paid rows now bind. The +`state_cd` contract counts are in docs/us-acs-local-soi-target-surface.md. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 7df8db9b9..e2aec45ac 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -46,6 +46,9 @@ from microcosm.calibrate.geography_constants import US_STATE_FIPS_TO_POSTAL __all__ = [ + "AmbiguousSoiDistrictComponentControlError", + "AmbiguousSoiDistrictComponentShareTotalError", + "UnreviewedSoiDistrictColumnError", "US_FISCAL_MACRO_REALISM_BANDS", "US_FISCAL_TARGET_REGISTRY", "US_FISCAL_TARGET_SPECS", @@ -1067,6 +1070,11 @@ def compile_us_fiscal_target_registry( materialized_facts, target_period=target_period, ) + registry = _rebase_soi_district_component_shares( + registry, + materialized_facts, + target_period=target_period, + ) registry = _rebase_soi_state_agi_bands(registry, materialized_facts) registry = apply_ledger_target_profile( registry, @@ -1991,6 +1999,322 @@ def _soi_total_uprating_index(measure_id: str) -> str: return "total_taxable_interest_amount" +@dataclass(frozen=True) +class _SoiDistrictComponentShare: + """A district-file IRS column that counts one part of a model concept.""" + + #: The SOI concept (``SOI_VARIABLE_MAP`` key) the share rows bind. + variable: str + is_count: bool + #: The national measure whose level the district file's shares split, and + #: the return universe its fact must come from. + control_measure_id: str + control_universe: str + #: What the column counts, and what the control and ``variable`` count. + source_concept: str + control_concept: str + + +#: The congressional-district file's interest-paid column (microcosm#1038). +#: +#: 22incd.csv has no Schedule A line 10 column. 22incddocguide.docx lists the +#: line's parts: N19300/A19300 "Mortgage interest paid" (line 8a), +#: N19500/A19500 home mortgage from a personal seller (8b), N19530/A19530 +#: deductible points (8c) and N19570/A19570 investment interest paid (line 9). +#: Chronicle's congressional_district_2022 package publishes only line 8a, +#: under the interest_paid_deduction_* measure ids, whose Table 2.1 rows are +#: the line 10 total. Those rows used to compile onto interest_deduction as +#: line 10 levels, and on the full surface they summed to 0.673x the national +#: Table 2.1 target. +#: +#: Line 8a is 85.6% of line 10 in the file's US row, but 38.8% in FL-21 and +#: under 46% in NY-10 and NY-12, where investment interest is concentrated. So +#: a district's share of line 8a misstates its share of line 10 by a median +#: 7.3% and at worst 54.6%. Line 8a is also 98.2% of lines 8a + 8b. Table +#: 2.1's home mortgage interest total is exactly 8a + 8b, and +#: deductible_mortgage_interest measures that total. A district's share of +#: line 8a is within 2.4% of its share of 8a + 8b in every district (median +#: 0.45%). So each district-file row that reads one of these columns compiles +#: onto deductible_mortgage_interest as its share of the file's own US row, +#: scaled to Table 2.1's home mortgage interest control +#: (:func:`_rebase_soi_district_component_shares`). The column is the key, so +#: the rule holds whatever measure id a package publishes it under. +#: docs/us-soi-interest-paid-concepts.md has the measurements. +_SOI_SCHEDULE_A_LINE_8A = ( + "schedule_a_line_8a_home_mortgage_interest_paid_to_financial_institutions" +) +_SOI_SCHEDULE_A_LINES_8A_8B = "schedule_a_lines_8a_8b_total_home_mortgage_interest" +_SOI_DISTRICT_COMPONENT_SHARE_COLUMNS: dict[str, _SoiDistrictComponentShare] = { + "N19300": _SoiDistrictComponentShare( + variable="deductible_mortgage_interest", + is_count=True, + control_measure_id="home_mortgage_interest_returns", + control_universe="itemized_returns", + source_concept=_SOI_SCHEDULE_A_LINE_8A, + control_concept=_SOI_SCHEDULE_A_LINES_8A_8B, + ), + "A19300": _SoiDistrictComponentShare( + variable="deductible_mortgage_interest", + is_count=False, + control_measure_id="home_mortgage_interest_amount", + control_universe="itemized_returns", + source_concept=_SOI_SCHEDULE_A_LINE_8A, + control_concept=_SOI_SCHEDULE_A_LINES_8A_8B, + ), +} +#: Measure ids for whose concept the district file has no column. A +#: district-file fact of one must read a column registered above; any other +#: column, or none, refuses the compile until it is reviewed. +_SOI_DISTRICT_MEASURES_WITHOUT_A_COLUMN = frozenset( + {"interest_paid_deduction_returns", "interest_paid_deduction_amount"} +) +_SOI_DISTRICT_COMPONENT_SHARE_FLAG = "requires_soi_district_component_share" +_SOI_SOURCE_COLUMN_ID_KEY = "soi_source_column_id" + + +class UnreviewedSoiDistrictColumnError(ValueError): + """A district-file fact names no reviewed column for its measure's concept.""" + + +class AmbiguousSoiDistrictComponentControlError(ValueError): + """Different national facts tie for one district component control.""" + + +class AmbiguousSoiDistrictComponentShareTotalError(ValueError): + """One district-file record set carries two different US totals.""" + + +def _is_soi_congressional_district_family(fact: object) -> bool: + """Whether a fact comes from a congressional-district file, any vintage.""" + + return ".congressional_district_" in _str_at(fact, "layout", "record_set_id") + + +def _soi_district_component_share( + fact: object, *, measure_id: str +) -> _SoiDistrictComponentShare | None: + """The share rule a district-file fact's column follows, if any. + + Raises: + UnreviewedSoiDistrictColumnError: For a district-file fact of a + measure the file has no column for that reads an unregistered + column or names none. + """ + + if _source_name(fact) != "irs_soi" or not _is_soi_congressional_district_family( + fact + ): + return None + column = _str_at(fact, "layout", "source_column_id") + rule = _SOI_DISTRICT_COMPONENT_SHARE_COLUMNS.get(column) + if rule is None and measure_id in _SOI_DISTRICT_MEASURES_WITHOUT_A_COLUMN: + raise UnreviewedSoiDistrictColumnError( + f"{_source_record_id(fact)} is a congressional-district " + f"{measure_id} fact read from column {column or '(none)'!r}. The " + "district file has no column for this concept; only the reviewed " + f"component columns {sorted(_SOI_DISTRICT_COMPONENT_SHARE_COLUMNS)} " + "compile, as shares (microcosm#1038)." + ) + return rule + + +def _rebase_soi_district_component_shares( + registry: TargetRegistry, + facts: tuple[object, ...], + *, + target_period: int | str, +) -> TargetRegistry: + """Bind district-file component rows as shares of a national control. + + A row :func:`_soi_reference_from_fact` flagged reads a column that counts + one part of the concept it binds (``_SOI_DISTRICT_COMPONENT_SHARE_COLUMNS``). + It never ships as a level. Its value becomes ``control x row / US row``: + the US row is the same record set's national all-AGI row of the same + column, filing status and period, and the control is the latest + national all-AGI fact of the rule's control measure and universe not + after the build period. One factor per record set, column and status + scales the file's states and districts alike, so its districts still sum + to its states and, where the file's states sum to its US row, its states + and its districts each sum to the control. The US row retires, because + the control's own row owns the national concept. A row with no control at + or after its period, or no nonzero US row, is dropped. Every kept row + declares the bridge: ``soi_source_concept``, ``soi_control_concept``, the + denominator (``soi_share_total_source_record_id``) and the control's + universe. The value lands at the control's period, so target aging ages + it with the control's own row and the file's period stamp cancels. + """ + + controls = _soi_district_component_controls(facts, target_period=target_period) + share_totals = _soi_district_component_share_totals(facts) + specs: list[TargetSpec] = [] + for spec in registry.specs: + if spec.metadata.get(_SOI_DISTRICT_COMPONENT_SHARE_FLAG) != "true": + specs.append(spec) + continue + column = spec.metadata[_SOI_SOURCE_COLUMN_ID_KEY] + rule = _SOI_DISTRICT_COMPONENT_SHARE_COLUMNS[column] + status = spec.metadata.get("filing_status", "") + source_period = spec.metadata.get("source_period", "") + control = controls.get((rule.control_measure_id, status)) + if control is None or not _period_not_before( + control.period_key, _period_key_from_value(source_period) + ): + continue + share_total = share_totals.get( + ( + spec.metadata.get("ledger_layout_record_set_id", ""), + column, + status, + source_period, + ) + ) + if share_total is None or share_total.value == 0: + continue + if _is_national_all_agi_spec(spec): + continue + factor = control.value / share_total.value + metadata = { + key: value + for key, value in spec.metadata.items() + if key != _SOI_DISTRICT_COMPONENT_SHARE_FLAG + } + metadata.update( + { + "uprating_index": rule.control_measure_id, + "uprating_from_period": source_period, + # The rebase lands the value at the CONTROL's period; target + # aging completes the remaining links from there. + "uprating_to_period": control.source_period, + "uprating_index_source_period": control.source_period, + "uprating_index_source_record_id": control.source_record_id, + "uprating_factor": _format_float(factor), + "stale_distribution_rebased_to_active_total": "true", + "soi_source_concept": rule.source_concept, + "soi_control_concept": rule.control_concept, + "soi_share_total_source_record_id": share_total.source_record_id, + } + ) + if rule.control_universe != spec.metadata.get("soi_return_universe"): + metadata["soi_return_universe_bridge"] = rule.control_universe + specs.append(replace(spec, value=spec.value * factor, metadata=metadata)) + return TargetRegistry(specs, country=registry.country) + + +def _soi_district_component_controls( + facts: tuple[object, ...], + *, + target_period: int | str, +) -> dict[tuple[str, str], _SoiTotalControl]: + """Each component rule's national control, by (measure, filing status). + + Only a national all-AGI fact of a rule's control measure and universe + from outside the district files qualifies. The latest period not after + the build period wins, and the choice depends on the fact set alone: two + different candidates at that period refuse the compile instead of letting + feed order pick one (microcosm#1035). + """ + + universes = { + rule.control_measure_id: rule.control_universe + for rule in _SOI_DISTRICT_COMPONENT_SHARE_COLUMNS.values() + } + target_period_key = _period_key_from_value(target_period) + controls: dict[tuple[str, str], _SoiTotalControl] = {} + tied: dict[tuple[str, str], set[tuple[str, float, str]]] = {} + for fact in facts: + if _source_name(fact) != "irs_soi": + continue + measure_id = _measure_id(fact) + universe = universes.get(measure_id) + if universe is None or _soi_return_universe_from_fact(fact) != universe: + continue + if _geography_level(fact) != "country" or not _is_all_agi_range_fact(fact): + continue + if _is_soi_congressional_district_family(fact): + continue + status = _filing_status_label(_dimensions(fact).get("filing_status")) + source_record_id = _source_record_id(fact) + if status is None or not source_record_id: + continue + period_key = _period_key(fact) + if not _not_after_target_period(period_key, target_period_key): + continue + candidate = _SoiTotalControl( + value=_numeric_value(fact), + source_period=str(_period_value(fact)), + source_record_id=source_record_id, + period_key=period_key, + ) + key = (measure_id, status) + identity = (source_record_id, candidate.value, candidate.source_period) + current = controls.get(key) + if current is not None and period_key[:2] == current.period_key[:2]: + tied[key].add(identity) + continue + if current is None or period_key[:2] > current.period_key[:2]: + controls[key] = candidate + tied[key] = {identity} + ambiguous = {key: sorted(found) for key, found in tied.items() if len(found) > 1} + if ambiguous: + raise AmbiguousSoiDistrictComponentControlError( + "District component control is ambiguous: different candidates " + f"tie at the latest period for {ambiguous}. Deduplicate them " + "upstream; feed order must not pick a control (microcosm#1035)." + ) + return controls + + +@dataclass(frozen=True) +class _SoiDistrictShareTotal: + value: float + source_record_id: str + + +def _soi_district_component_share_totals( + facts: tuple[object, ...], +) -> dict[tuple[str, str, str, str], _SoiDistrictShareTotal]: + """Each district file's US row of a component column, its shares' denominator. + + Keyed by (record set, column, filing status, period), so a row divides by + the US row of its own file. Two different values under one key refuse the + compile; equal values under two record ids keep the smaller id, so the + named denominator does not depend on feed order. + """ + + totals: dict[tuple[str, str, str, str], _SoiDistrictShareTotal] = {} + for fact in facts: + if _source_name(fact) != "irs_soi": + continue + column = _str_at(fact, "layout", "source_column_id") + if column not in _SOI_DISTRICT_COMPONENT_SHARE_COLUMNS: + continue + if not _is_soi_congressional_district_family(fact): + continue + if _geography_level(fact) != "country" or not _is_all_agi_range_fact(fact): + continue + status = _filing_status_label(_dimensions(fact).get("filing_status")) + source_record_id = _source_record_id(fact) + if status is None or not source_record_id: + continue + key = ( + _str_at(fact, "layout", "record_set_id"), + column, + status, + str(_period_value(fact)), + ) + candidate = _SoiDistrictShareTotal(_numeric_value(fact), source_record_id) + current = totals.get(key) + if current is not None and current.value != candidate.value: + raise AmbiguousSoiDistrictComponentShareTotalError( + f"District file has two US totals for {key}: {current.value!r} " + f"({current.source_record_id}) and {candidate.value!r} " + f"({candidate.source_record_id}). Deduplicate them upstream." + ) + if current is None or candidate.source_record_id < current.source_record_id: + totals[key] = candidate + return totals + + def _rebase_soi_state_agi_bands( registry: TargetRegistry, facts: tuple[object, ...], @@ -3098,6 +3422,12 @@ def _soi_reference_from_fact( override = _soi_layout_variable_override(fact, measure_id=measure_id) if override is not None: variable, is_count = override + # A district-file column that counts part of a concept binds that + # concept's model variable as a share, whatever measure id it is + # published under (_rebase_soi_district_component_shares; microcosm#1038). + component = _soi_district_component_share(fact, measure_id=measure_id) + if component is not None: + variable, is_count = component.variable, component.is_count if variable is None: return None @@ -3148,7 +3478,12 @@ def _soi_reference_from_fact( "source_variable": variable, "source_period": str(_period_value(fact)), "target_period": str(target_period), - "target_role": _soi_target_role(fact, measure_id), + # A component share is never its measure id's national total. + "target_role": ( + "soi_fiscal_distribution" + if component is not None + else _soi_target_role(fact, measure_id) + ), "variable": display_variable, "materializer": "irs_soi_slice", "measure_mode": _soi_measure_mode(variable, is_count=is_count), @@ -3181,6 +3516,11 @@ def _soi_reference_from_fact( metadata["requires_total_soi_uprating"] = "true" if requires_state_agi_band_rebase: metadata[_SOI_STATE_AGI_BAND_REBASE_FLAG] = "true" + if component is not None: + metadata[_SOI_DISTRICT_COMPONENT_SHARE_FLAG] = "true" + metadata[_SOI_SOURCE_COLUMN_ID_KEY] = _str_at( + fact, "layout", "source_column_id" + ) return LedgerTargetReference( name=source_record_id, ledger_source_record_id=source_record_id, diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_cd_properties.py b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_cd_properties.py index 243568e5e..886631fef 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_cd_properties.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_cd_properties.py @@ -152,7 +152,13 @@ def test_sparse_and_dense_compile_agree_on_every_target(surface) -> None: "return_count": ("count", "sum"), "wages_salaries_amount": ("employment_income", "sum"), "charitable_amount": ("charitable_deduction", "sum"), + "interest_paid_deduction_amount": ("deductible_mortgage_interest", "sum"), } +#: District-file-only measures the compiler binds as shares of a national +#: control (the line 8a mortgage interest rows, microcosm#1038): their +#: district-file specs name the share's denominator, and Historic Table 2 +#: never carries them. +_NATIONAL_SHARE_MEASURES = frozenset({"interest_paid_deduction_amount"}) def _soi_spec(name, measure, value, *, level, record_set, state, district=None): @@ -171,6 +177,8 @@ def _soi_spec(name, measure, value, *, level, record_set, state, district=None): } if district is not None: metadata["congressional_district_geoid"] = district + if measure in _NATIONAL_SHARE_MEASURES and record_set == _CD_FILE: + metadata["soi_share_total_source_record_id"] = f"cdfile.us.{measure}" return TargetSpec( name=name, entity="household", @@ -215,7 +223,9 @@ def _state_cd_inputs(draw): # file has no sub-state rows for it, whatever the current plan says. layout[state] = (districts, len(source) == 1) for measure in measures: - if measure == "return_count" or draw(st.booleans()): + if measure == "return_count" or ( + measure not in _NATIONAL_SHARE_MEASURES and draw(st.booleans()) + ): specs.append( _soi_spec( f"ht2.{state}.{measure}", @@ -256,7 +266,12 @@ def _state_cd_inputs(draw): return [specs[i] for i in order], pd.DataFrame(crosswalk), layout -_BRIDGES = {measure: "return_count" for measure in _MEASURES} +#: A national-share measure has no bridge: the surface must not need one. +_BRIDGES = { + measure: "return_count" + for measure in _MEASURES + if measure not in _NATIONAL_SHARE_MEASURES +} @_SETTINGS @@ -302,7 +317,19 @@ def test_district_targets_add_up_to_one_state_vintage(inputs) -> None: expected_parent = ht2 if ht2 in original else f"cdfile.{state}_total.{measure}" assert spec.metadata["state_cd_parent_target_name"] == expected_parent assert expected_parent in kept - if expected_parent != ht2: + if measure in _NATIONAL_SHARE_MEASURES: + # Already on its national control's basis: bound as compiled, its + # own parent, never bridged a second time. + assert kept[expected_parent].value == original[expected_parent].value + assert ( + kept[expected_parent].metadata["state_cd_level_basis"] + == "national_control_share" + ) + assert ( + spec.metadata["state_cd_parent_basis"] + == "cd_file_state_total_national_share" + ) + elif expected_parent != ht2: # A district-file-only state level is lifted onto the Historic # Table 2 basis by its bridge sibling's two levels in that state. bridge = ( @@ -812,6 +839,102 @@ def test_a_bridge_through_an_out_of_band_sibling_drops_its_state_and_districts() assert _bound(surface, "20", "charitable_amount", "state") +def test_a_national_share_level_needs_no_bridge_and_survives_its_band() -> None: + """A district-file-only row bound as a national share is its own parent. + + On the pinned feed the itemized bridge dropped Wyoming's and South + Dakota's interest-paid rows because their charitable deductions differ + between the two files. Once the compiler binds the line 8a rows as shares + of Table 2.1's control, they need no bridge, so a sibling's band verdict + cannot drop them, and they bind at their compiled value. + """ + + measure = "interest_paid_deduction_amount" + specs, crosswalk = _bridge_world( + {"10": 1.0, "11": 1.0, "12": 1.0, "13": 1.0, "14": 2.0}, {"20": 2.0} + ) + for state, districts in ( + ("10", [f"{10}01", f"{10}02"]), + ("14", [f"{14}01", f"{14}02"]), + ("20", []), # at-large: a state row only + ): + specs.append( + _soi_spec( + f"cdfile.{state}_total.{measure}", + measure, + 50.0, + level="state", + record_set=_CD_FILE, + state=state, + ) + ) + for district, share in zip( + districts, (0.3, 0.7)[: len(districts)], strict=True + ): + specs.append( + _soi_spec( + f"cdfile.{district}.{measure}", + measure, + 50.0 * share, + level="congressional_district", + record_set=_CD_FILE, + state=state, + district=district, + ) + ) + surface = _CD.state_cd_soi_surface( + specs, + state_surface_predicate=_TOOL.soi_surface_predicate("state"), + crosswalk=crosswalk, + level_bridges={"charitable_amount": "return_count"}, + ) + by_name = {spec.name: spec for spec in surface.specs} + # The bridged sibling concept is distrusted in 14 and in at-large 20 ... + assert not _bound(surface, "14", "charitable_amount", "state") + assert not _bound(surface, "20", "charitable_amount", "state") + # ... while the national-share concept binds everywhere, as compiled. + for state in ("10", "14", "20"): + (name,) = _bound(surface, state, measure, "state") + assert by_name[name].value == 50.0 + assert by_name[name].metadata["state_cd_level_basis"] == ( + "national_control_share" + ) + for state in ("10", "14"): + children = _bound(surface, state, measure, "congressional_district") + assert len(children) == 2 + assert math.fsum(by_name[name].value for name in children) == pytest.approx( + 50.0, rel=1e-12 + ) + assert { + by_name[name].metadata["state_cd_parent_basis"] for name in children + } == {"cd_file_state_total_national_share"} + assert not [ + row + for row in surface.receipt["factor_band"]["out_of_band"] + if row["measure"] == measure + ] + assert surface.receipt["national_share_levels_by_measure"] == {measure: 3} + assert measure not in surface.receipt["level_bridge_factor_by_measure"] + + +def test_a_district_file_only_level_with_neither_bridge_nor_share_is_refused() -> None: + """Without a national share declaration the level still needs a bridge. + + So an interest-paid row the compiler failed to rebase could not slip onto + the surface unbridged: ``STATE_CD_LEVEL_BRIDGES`` no longer lists it. + """ + + assert "interest_paid_deduction_amount" not in _CD.STATE_CD_LEVEL_BRIDGES + specs, crosswalk = _bridge_world({"10": 1.0, "11": 1.0}, {}) + with pytest.raises(ValueError, match="has no level bridge"): + _CD.state_cd_soi_surface( + specs, + state_surface_predicate=_TOOL.soi_surface_predicate("state"), + crosswalk=crosswalk, + level_bridges={}, + ) + + def test_a_sign_conflict_or_a_self_contradicting_district_file_is_refused() -> None: """Defects inside the data are refused, not dropped as level gaps.""" diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_sparse_cd.py b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_sparse_cd.py index 73d885170..0f353b11d 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_sparse_cd.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_sparse_cd.py @@ -904,16 +904,17 @@ def test_pinned_feed_state_cd_surface_matches_its_contract() -> None: soi = [spec for spec in specs if spec.family == "irs_soi"] assert receipt["counts"] == { "historic_table_2_state": 4227, - "cd_file_state": 302, + "cd_file_state": 304, "congressional_district": 21743, "congressional_district_by_parent_basis": { "historic_table_2": 19181, - "cd_file_state_total_bridged": 2562, + "cd_file_state_total_bridged": 1708, + "cd_file_state_total_national_share": 854, }, - "total": 26272, + "total": 26274, } - assert len(soi) == 26272 - assert len(specs) == 26272 + 102 + 51 + assert len(soi) == 26274 + assert len(specs) == 26274 + 102 + 51 reconciliation = module.cd_surface.state_parent_reconciliation(soi) assert len(reconciliation) == 2189 assert all(block["ok"] for block in reconciliation) @@ -947,6 +948,7 @@ def test_pinned_feed_state_cd_surface_matches_its_contract() -> None: # The factor band drops these (state, measure) pairs, and only these: # four district blocks whose two publications disagree about the state, # and the itemized bridge of two at-large states (no district rows). + # Interest paid no longer crosses that bridge (microcosm#1038). # The district file agrees with itself in every block, to float precision. internal = receipt["internal_consistency"] assert internal["blocks_checked"] == 2193 @@ -962,9 +964,7 @@ def test_pinned_feed_state_cd_surface_matches_its_contract() -> None: ("36", "rental_royalty_income_amount", "rebase"), ("49", "tax_exempt_interest_amount", "rebase"), ("46", "charitable_amount", "level_bridge"), - ("46", "interest_paid_deduction_amount", "level_bridge"), ("56", "charitable_amount", "level_bridge"), - ("56", "interest_paid_deduction_amount", "level_bridge"), } # 43 states carry district rows (51 minus the 8 at-large on the 117th # plan), less the band's four blocks. @@ -980,18 +980,37 @@ def test_pinned_feed_state_cd_surface_matches_its_contract() -> None: } assert {measure for measure, n in rebase_n.items() if n == 42} == banded assert set(rebase_n.values()) == {42, 43} - # Every district-file-only state level is bridged, in all 51 states but - # the band's two itemized outliers. + # Every other district-file-only state level is bridged, in all 51 + # states but the band's two itemized outliers. bridges = receipt["level_bridge_factor_by_measure"] assert set(bridges) == set(module.cd_surface.STATE_CD_LEVEL_BRIDGES) assert {measure: entry["n"] for measure, entry in bridges.items()} == { "charitable_amount": 49, "charitable_returns": 51, - "interest_paid_deduction_amount": 49, - "interest_paid_deduction_returns": 51, "qualified_business_income_deduction_amount": 51, "qualified_business_income_deduction_returns": 51, } + # The line 8a mortgage interest rows bind as compiled, shares of Table + # 2.1's home mortgage interest control, in all 51 states; the states sum + # to that control, which the national surface binds too. + assert receipt["national_share_levels_by_measure"] == { + "interest_paid_deduction_amount": 51, + "interest_paid_deduction_returns": 51, + } + for measure in ( + "interest_paid_deduction_amount", + "interest_paid_deduction_returns", + ): + states = [ + spec for spec in state_rows if spec.metadata["source_measure_id"] == measure + ] + assert len(states) == 51 + assert {spec.metadata["variable"] for spec in states} == { + "deductible_mortgage_interest" + } + assert { + spec.metadata["soi_share_total_source_record_id"] for spec in states + } == {f"irs_soi.ty2023.congressional_district_2022.all_returns.us.{measure}"} assert json.dumps(receipt) # the receipt is manifest-serializable diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 3cd976bac..0e32cdb8b 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -127,12 +127,6 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - _soi_congressional_district_fact( - "interest_paid_deduction_returns", - 23_456, - groupby_value_id="hi_01", - geography_id="5001700US1501", - ), _soi_congressional_district_fact( "charitable_returns", 34_567, @@ -195,7 +189,9 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: assert any( "hi_total.adjusted_gross_income" in source_id for source_id in source_ids ) - assert len(cd_specs) == 11 + # Interest paid binds only as a share of a national control: + # test_congressional_district_mortgage_interest_binds_as_a_national_share. + assert len(cd_specs) == 10 by_measure = { spec.metadata["source_measure_id"]: spec for spec in cd_specs @@ -239,12 +235,6 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: ) assert qbi_returns.value == 12_345 - interest_returns = by_measure["interest_paid_deduction_returns"] - assert interest_returns.metadata["measure_mode"] == "indicator_sum" - assert interest_returns.metadata["base_variable"] == "interest_deduction" - assert interest_returns.metadata["itemized_only"] == "true" - assert interest_returns.value == 23_456 - charitable_returns = by_measure["charitable_returns"] assert charitable_returns.metadata["measure_mode"] == "indicator_sum" assert charitable_returns.metadata["base_variable"] == "charitable_deduction" @@ -893,10 +883,13 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No def test_pinned_feed_national_state_surface_restores_the_fences( pinned_feed_national_state_surface, ) -> None: - """The release surface on the pinned feed: 33,250 compiled targets after + """The release surface on the pinned feed: 33,248 compiled targets after Medicaid substitution, 6,102 national_state targets at registry e1f9906bf0e0, 32 CHIP rows none of them for an M-CHIP state, no - other-income row and no tips return count. microcosm#1206 moved the + other-income row and no tips return count. microcosm#1038 retired the + congressional-district file's two interest-paid US rows (33,250 before); + national_state does not carry district-file rows, so it did not move. + microcosm#1206 moved the version from f96be9f41781 by binding the Part B premium target to gross_medicare_part_b_premium_if_enrolled; the counts are unchanged. The 408 state AGI-band rows of @@ -906,7 +899,7 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77 (docs/us-chronicle-feed-repin.md).""" registry, surface, _ = pinned_feed_national_state_surface - assert len(registry.specs) == 33_250 + assert len(registry.specs) == 33_248 assert len(surface.specs) == 6_102 assert surface.version == "e1f9906bf0e0" chip = [ @@ -935,6 +928,82 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) +def test_pinned_feed_district_interest_paid_binds_mortgage_interest_shares( + pinned_feed_national_state_surface, +) -> None: + """The district file's line 8a rows on the pinned feed (microcosm#1038). + + Its 51 state and 436 district rows of each measure bind + deductible_mortgage_interest, and each set sums to Table 2.1's national + home mortgage interest target after aging. None binds interest_deduction, + whose national line 10 row they summed to 0.673x of. The factors are + Table 2.1 TY2023 over the file's US row: $171,364,787,000 / + $140,135,155,000 and 11,644,348 / 11,352,500 returns. + """ + + registry, _, _ = pinned_feed_national_state_surface + district_file = [ + spec + for spec in registry.specs + if ".congressional_district_" + in spec.metadata.get("ledger_layout_record_set_id", "") + ] + assert not [ + spec.name + for spec in district_file + if spec.metadata.get("variable") == "interest_deduction" + or spec.metadata.get("requires_soi_district_component_share") + ] + national = { + spec.metadata["source_measure_id"]: spec + for spec in registry.specs + if spec.metadata.get("ledger_layout_record_set_id") + == "irs_soi.ty2023.table_2_1.itemized_all_returns" + and spec.metadata.get("ledger_geography_level") == "country" + } + assert national["interest_paid_deduction_amount"].metadata["variable"] == ( + "interest_deduction" + ) + factors = { + "amount": 171_364_787_000 / 140_135_155_000, + "returns": 11_644_348 / 11_352_500, + } + for kind, factor in factors.items(): + control = national[f"home_mortgage_interest_{kind}"] + rows = [ + spec + for spec in district_file + if spec.metadata["source_measure_id"] == f"interest_paid_deduction_{kind}" + ] + assert {spec.metadata["ledger_geography_level"] for spec in rows} == { + "state", + "congressional_district", + } + assert {spec.metadata["variable"] for spec in rows} == { + "deductible_mortgage_interest" + } + assert {spec.metadata["aging_factor"] for spec in rows} == { + control.metadata["aging_factor"] + } + for spec in rows: + assert float(spec.metadata["uprating_factor"]) == pytest.approx( + factor, rel=1e-12 + ) + assert spec.metadata["soi_source_column_id"] == ( + "A19300" if kind == "amount" else "N19300" + ) + for level, count in (("state", 51), ("congressional_district", 436)): + level_rows = [ + spec + for spec in rows + if spec.metadata["ledger_geography_level"] == level + ] + assert len(level_rows) == count + assert math.fsum(spec.value for spec in level_rows) == pytest.approx( + control.value, rel=1e-9 + ) + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_soi_district_interest_shares.py b/packages/microcosm-build/tests/engine_free/us/test_us_soi_district_interest_shares.py new file mode 100644 index 000000000..fd686091e --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/us/test_us_soi_district_interest_shares.py @@ -0,0 +1,577 @@ +"""Congressional-district interest paid binds as a share of a national control. + +The district file publishes Schedule A line 8a (N19300/A19300), not the line +10 total its interest_paid_deduction_* measure ids name. Its rows bind +deductible_mortgage_interest as shares of the file's US row, scaled to Table +2.1's home mortgage interest control (microcosm#1038, +docs/us-soi-interest-paid-concepts.md). + +Invariants, for every fact set the compile accepts: + +- No district-file row binds interest_deduction. +- One factor (control / the file's US row) scales every row of one record set, + column and filing status, so district rows keep their within-state shares + and the states keep their shares of the US row. +- Where the file's states sum to its US row, its states sum to the control, + and so do its districts. +- The file's US row never binds; Table 2.1's rows own the national concepts. +- A row with no control at or after its period, or no nonzero US row, never + binds (no component level ships). +- The output does not depend on feed order or on a common scale of the + district file's values (the package's period stamp cancels likewise). +""" + +# ruff: noqa: F403, F405 +import random + +from test_support.microcosm_build.us_fiscal_targets import * + +hypothesis = pytest.importorskip("hypothesis") +from hypothesis import HealthCheck, given, settings # noqa: E402 +from hypothesis import strategies as st # noqa: E402 + +from microcosm.build.us_runtime.fiscal_targets import ( # noqa: E402 + AmbiguousSoiDistrictComponentControlError, + AmbiguousSoiDistrictComponentShareTotalError, + UnreviewedSoiDistrictColumnError, +) + +_CD_RECORD_SET = "irs_soi.ty2023.congressional_district_2022.all_returns" +_T21_RECORD_SET = "irs_soi.ty2023.table_2_1.itemized_all_returns" +#: States with two districts on both plans, so the hierarchy rule's +#: completeness check passes without a crosswalk. +_TWO_DISTRICT_STATES = { + "15": "hi", + "16": "id", + "23": "me", + "33": "nh", + "44": "ri", + "54": "wv", +} +_COLUMNS = {"amount": "A19300", "returns": "N19300"} +_CONTROLS = { + "amount": "home_mortgage_interest_amount", + "returns": "home_mortgage_interest_returns", +} + + +def _cd_fact( + measure_id: str, + column: str | None, + value: float, + *, + row: str = "us", + level: str = "country", + geography_id: str = "0100000US", + record_set: str = _CD_RECORD_SET, + period: int = 2023, +) -> dict[str, object]: + fact = _dynamic_ledger_fact( + source_record_id=f"{record_set}.{row}.{measure_id}", + source_name="irs_soi", + measure_id=measure_id, + value=value, + period_value=period, + geography_level=level, + geography_id=geography_id, + dimensions={"income_range": "all", "filing_status": "all"}, + layout_record_set_id=record_set, + groupby_dimension="irs_soi.congressional_district", + groupby_value_id=row, + ) + if column is not None: + fact["layout"]["source_column_id"] = column + return fact + + +def _table_2_1_fact( + measure_id: str, + value: float, + *, + period: int = 2023, + record_set: str = _T21_RECORD_SET, +) -> dict[str, object]: + return _dynamic_ledger_fact( + source_record_id=f"{record_set}.all.{measure_id}", + source_name="irs_soi", + measure_id=measure_id, + value=value, + period_value=period, + dimensions={"income_range": "all", "filing_status": "all"}, + layout_record_set_id=record_set, + ) + + +def _district_file( + districts: dict[str, tuple[float, float]], + *, + kind: str = "amount", + measure_prefix: str = "interest_paid_deduction", + us_total: float | None = None, + scale: float = 1.0, + record_set: str = _CD_RECORD_SET, + period: int = 2023, +) -> list[dict[str, object]]: + """The file's US, state and district rows for ``{state_fips: (d1, d2)}``.""" + + measure_id = f"{measure_prefix}_{kind}" + column = _COLUMNS[kind] + facts = [] + states_total = 0.0 + for fips, (first, second) in sorted(districts.items()): + postal = _TWO_DISTRICT_STATES[fips] + states_total += first + second + facts.append( + _cd_fact( + measure_id, + column, + (first + second) * scale, + row=f"{postal}_total", + level="state", + geography_id=f"0400000US{fips}", + record_set=record_set, + period=period, + ) + ) + for index, value in enumerate((first, second), start=1): + facts.append( + _cd_fact( + measure_id, + column, + value * scale, + row=f"{postal}_{index:02d}", + level="congressional_district", + geography_id=f"5001700US{fips}{index:02d}", + record_set=record_set, + period=period, + ) + ) + facts.append( + _cd_fact( + measure_id, + column, + (states_total if us_total is None else us_total) * scale, + record_set=record_set, + period=period, + ) + ) + return facts + + +def _national_rows(kind: str = "amount", *, control: float = 171e9) -> list: + """Table 2.1's line 10 total and its home mortgage interest control.""" + + line_10 = 208e9 if kind == "amount" else 11.9e6 + return [ + _table_2_1_fact(f"interest_paid_deduction_{kind}", line_10), + _table_2_1_fact(_CONTROLS[kind], control), + ] + + +def _compile(facts) -> TargetRegistry: + return compile_us_fiscal_target_registry( + [*packaged_reference_facts(), *facts], allow_unaged_dollar_targets=True + ) + + +def _district_file_specs(registry) -> list: + return [ + spec + for spec in registry.specs + if ".congressional_district_" + in spec.metadata.get("ledger_layout_record_set_id", "") + ] + + +_WORLD = {"15": (30e9, 50e9), "23": (8e9, 12e9)} + + +@pytest.mark.parametrize("kind", ["amount", "returns"]) +def test_congressional_district_mortgage_interest_binds_as_a_national_share( + kind: str, +) -> None: + control = 171e9 if kind == "amount" else 11.6e6 + us_total = 100e9 # the two states' rows sum to it + registry = _compile( + [*_district_file(_WORLD, kind=kind), *_national_rows(kind, control=control)] + ) + rows = _district_file_specs(registry) + factor = control / us_total + # Two states and four districts bind; the file's US row retires. + assert len(rows) == 6 + assert not [spec for spec in rows if "state_fips" not in spec.metadata] + for spec in rows: + metadata = spec.metadata + assert metadata["variable"] == "deductible_mortgage_interest" + assert metadata["source_variable"] == "deductible_mortgage_interest" + assert metadata["base_variable"] == "deductible_mortgage_interest_tax_unit" + assert metadata["measure_mode"] == ( + "indicator_sum" if kind == "returns" else "sum" + ) + assert metadata["itemized_only"] == "true" + assert metadata["target_role"] == "soi_fiscal_distribution" + assert metadata["soi_source_column_id"] == _COLUMNS[kind] + assert metadata["soi_source_concept"] == ( + "schedule_a_line_8a_home_mortgage_interest_paid_to_financial_institutions" + ) + assert metadata["soi_control_concept"] == ( + "schedule_a_lines_8a_8b_total_home_mortgage_interest" + ) + assert metadata["soi_share_total_source_record_id"] == ( + f"{_CD_RECORD_SET}.us.interest_paid_deduction_{kind}" + ) + assert metadata["uprating_index_source_record_id"] == ( + f"{_T21_RECORD_SET}.all.{_CONTROLS[kind]}" + ) + assert metadata["uprating_to_period"] == "2023" + assert metadata["soi_return_universe"] == "all_returns" + assert metadata["soi_return_universe_bridge"] == "itemized_returns" + assert float(metadata["uprating_factor"]) == pytest.approx(factor, rel=1e-15) + assert "requires_soi_district_component_share" not in metadata + by_row = {spec.metadata["ledger_layout_groupby_value_id"]: spec for spec in rows} + assert by_row["hi_01"].value == pytest.approx(30e9 * factor, rel=1e-12) + assert by_row["me_total"].value == pytest.approx(20e9 * factor, rel=1e-12) + states = [ + spec for spec in rows if "congressional_district_geoid" not in spec.metadata + ] + assert math.fsum(spec.value for spec in states) == pytest.approx(control, rel=1e-12) + districts = [ + spec for spec in rows if "congressional_district_geoid" in spec.metadata + ] + assert math.fsum(spec.value for spec in districts) == pytest.approx( + control, rel=1e-12 + ) + # Table 2.1's own rows are untouched: line 10 stays on interest_deduction + # as its national total, and the control binds the same concept the + # district shares do. + national = { + spec.metadata["source_measure_id"]: spec + for spec in registry.specs + if spec.metadata.get("ledger_layout_record_set_id") == _T21_RECORD_SET + } + line_10 = national[f"interest_paid_deduction_{kind}"] + assert line_10.metadata["variable"] == "interest_deduction" + assert line_10.value == (208e9 if kind == "amount" else 11.9e6) + assert national[_CONTROLS[kind]].metadata["variable"] == ( + "deductible_mortgage_interest" + ) + assert national[_CONTROLS[kind]].value == control + + +def test_a_relabeled_district_measure_binds_the_same_shares() -> None: + """The IRS column is the key, not the measure id a package uses for it. + + A Chronicle package that relabels the line 8a column to Table 2.1's own + mortgage_interest_paid_* ids compiles to the same targets. + """ + + def values(prefix: str) -> dict[str, tuple[str, float]]: + registry = _compile( + [*_district_file(_WORLD, measure_prefix=prefix), *_national_rows()] + ) + return { + spec.metadata["ledger_layout_groupby_value_id"]: ( + spec.metadata["variable"], + spec.value, + ) + for spec in _district_file_specs(registry) + } + + assert values("mortgage_interest_paid") == values("interest_paid_deduction") + + +def test_the_other_line_10_parts_stay_inert() -> None: + """PolicyEngine/chronicle#345's district-file shape compiles only line 8a. + + That package relabels line 8a to mortgage_interest_paid_* and adds lines + 8b, 8c and 9 under Table 2.1's own ids, each on its own column. Those ids + are unmapped (Table 2.1 binds only their total), so they bind nothing at + any geography, and Table 2.1's own 8a row stays unmapped too. + """ + + others = { + "home_mortgage_personal_seller": "A19500", + "deductible_points": "A19530", + "investment_interest_paid": "A19570", + } + facts = [ + *_district_file(_WORLD, measure_prefix="mortgage_interest_paid"), + *_national_rows(), + _table_2_1_fact("mortgage_interest_paid_amount", 167e9), + ] + for prefix, column in others.items(): + facts += [ + _cd_fact(f"{prefix}_amount", column, 1e9), + _cd_fact( + f"{prefix}_amount", + column, + 1e8, + row="hi_total", + level="state", + geography_id="0400000US15", + ), + ] + registry = _compile(facts) + measures = {spec.metadata.get("source_measure_id") for spec in registry.specs} + assert measures.isdisjoint({f"{prefix}_amount" for prefix in others}) + assert "mortgage_interest_paid_amount" in { + spec.metadata["source_measure_id"] for spec in _district_file_specs(registry) + } + assert not [ + spec + for spec in registry.specs + if spec.metadata.get("ledger_layout_record_set_id") == _T21_RECORD_SET + and spec.metadata.get("source_measure_id") == "mortgage_interest_paid_amount" + ] + assert len(_district_file_specs(registry)) == 6 + + +@pytest.mark.parametrize("column", [None, "", "A19570", "A19500", "A04470"]) +def test_district_interest_paid_without_a_reviewed_column_is_refused( + column: str | None, +) -> None: + """The file has no line 10 column, so only line 8a may compile, as shares.""" + + facts = [ + *_district_file(_WORLD), + _cd_fact( + "interest_paid_deduction_amount", + column, + 1e9, + row="hi_extra", + level="state", + geography_id="0400000US15", + record_set="irs_soi.ty2023.congressional_district_2024.all_returns", + ), + *_national_rows(), + ] + with pytest.raises(UnreviewedSoiDistrictColumnError, match="no column for"): + _compile(facts) + + +def test_table_2_1_interest_paid_needs_no_column() -> None: + """The guard is the district file's: Table 2.1 publishes line 10 itself.""" + + registry = _compile(_national_rows()) + (line_10,) = [ + spec + for spec in registry.specs + if spec.metadata.get("source_measure_id") == "interest_paid_deduction_amount" + ] + assert line_10.metadata["target_role"] == "interest_deduction_total" + assert line_10.metadata["variable"] == "interest_deduction" + + +def test_district_mortgage_interest_without_an_anchor_never_binds() -> None: + """No control, a control older than the rows, or no US row: dropped.""" + + no_control = _compile(_district_file(_WORLD)) + assert not _district_file_specs(no_control) + + old_control = _compile( + [ + *_district_file(_WORLD), + _table_2_1_fact( + "home_mortgage_interest_amount", + 160e9, + period=2022, + record_set="irs_soi.ty2022.table_2_1.itemized_all_returns", + ), + ] + ) + assert not _district_file_specs(old_control) + + no_us_row = [ + fact + for fact in _district_file(_WORLD) + if fact["geography"]["level"] != "country" + ] + assert not _district_file_specs(_compile([*no_us_row, *_national_rows()])) + + zero_us_row = _district_file(_WORLD, us_total=0.0) + assert not _district_file_specs(_compile([*zero_us_row, *_national_rows()])) + + +def test_district_mortgage_interest_control_must_be_unambiguous() -> None: + """Feed order never picks a control or a denominator (microcosm#1035).""" + + rival = _table_2_1_fact( + "home_mortgage_interest_amount", + 170e9, + record_set="irs_soi.ty2023.table_2_1_revised.itemized_all_returns", + ) + with pytest.raises(AmbiguousSoiDistrictComponentControlError): + _compile([*_district_file(_WORLD), *_national_rows(), rival]) + + duplicate_us_row = _cd_fact( + "interest_paid_deduction_amount", "A19300", 99e9, row="us_duplicate" + ) + with pytest.raises(AmbiguousSoiDistrictComponentShareTotalError): + _compile([*_district_file(_WORLD), duplicate_us_row, *_national_rows()]) + + # The newest control wins outright; an older one is no rival. + older = _table_2_1_fact( + "home_mortgage_interest_amount", + 150e9, + period=2022, + record_set="irs_soi.ty2022.table_2_1.itemized_all_returns", + ) + registry = _compile([*_district_file(_WORLD), *_national_rows(), older]) + assert { + spec.metadata["uprating_index_source_record_id"] + for spec in _district_file_specs(registry) + } == {f"{_T21_RECORD_SET}.all.home_mortgage_interest_amount"} + + +def test_district_mortgage_interest_ignores_the_package_stamp() -> None: + """The file is TY2022 data stamped 2023 (microcosm#1030); either stamp + gives the same targets, landed at the control's period.""" + + def values(period: int) -> dict[str, tuple[float, str]]: + record_set = f"irs_soi.ty{period}.congressional_district_2022.all_returns" + registry = _compile( + [ + *_district_file(_WORLD, record_set=record_set, period=period), + *_national_rows(), + ] + ) + return { + spec.metadata["ledger_layout_groupby_value_id"]: ( + spec.value, + spec.metadata["uprating_to_period"], + ) + for spec in _district_file_specs(registry) + } + + stamped, truthful = values(2023), values(2022) + assert stamped.keys() == truthful.keys() + for row, (value, landed) in stamped.items(): + assert truthful[row][0] == pytest.approx(value, rel=1e-15) + assert landed == truthful[row][1] == "2023" + + +def test_district_mortgage_interest_ages_with_its_control() -> None: + """Target aging gives the shares the control's own remaining link, so + the states still sum to the aged national row.""" + + from microcosm.build.us_runtime.target_aging import age_us_dollar_targets + + registry = _compile([*_district_file(_WORLD), *_national_rows()]) + projections = ( + _cbo_income_source_projection_fact( + 2023, "adjusted_gross_income", value=15_000_000_000_000 + ), + _cbo_income_source_projection_fact( + 2024, "adjusted_gross_income", value=16_500_000_000_000 + ), + ) + aged = age_us_dollar_targets(registry, projections, target_period=2024) + link = 16_500_000_000_000 / 15_000_000_000_000 + (control,) = [ + spec + for spec in aged.specs + if spec.metadata.get("source_measure_id") == "home_mortgage_interest_amount" + ] + assert control.value == pytest.approx(171e9 * link, rel=1e-12) + states = [ + spec + for spec in _district_file_specs(aged) + if "congressional_district_geoid" not in spec.metadata + ] + assert len(states) == 2 + assert {spec.metadata["aging_factor"] for spec in states} == { + control.metadata["aging_factor"] + } + assert math.fsum(spec.value for spec in states) == pytest.approx( + control.value, rel=1e-12 + ) + + +@st.composite +def _district_worlds(draw): + fips = draw( + st.lists( + st.sampled_from(sorted(_TWO_DISTRICT_STATES)), + min_size=1, + max_size=len(_TWO_DISTRICT_STATES), + unique=True, + ) + ) + positive = st.floats(1.0, 1e11, allow_nan=False, allow_infinity=False) + world = {state: (draw(positive), draw(positive)) for state in fips} + return { + "world": world, + "kind": draw(st.sampled_from(["amount", "returns"])), + "prefix": draw( + st.sampled_from(["interest_paid_deduction", "mortgage_interest_paid"]) + ), + # The US row may exceed the bound states (states missing from the feed). + "us_excess": draw(st.sampled_from([1.0, 1.0, draw(st.floats(1.0, 3.0))])), + "control": draw(positive), + "scale": draw(st.floats(1e-3, 1e3)), + "seed": draw(st.integers(0, 2**31 - 1)), + } + + +@settings(max_examples=25, deadline=None, suppress_health_check=[HealthCheck.too_slow]) +@given(_district_worlds()) +def test_district_mortgage_interest_share_invariants(case) -> None: + world, kind = case["world"], case["kind"] + states_total = math.fsum(first + second for first, second in world.values()) + us_total = states_total * case["us_excess"] + control = case["control"] + + def compiled(scale: float, shuffle_seed: int | None) -> dict[str, TargetSpec]: + facts = [ + *_district_file( + world, + kind=kind, + measure_prefix=case["prefix"], + us_total=us_total, + scale=scale, + ), + *_national_rows(kind, control=control), + ] + if shuffle_seed is not None: + random.Random(shuffle_seed).shuffle(facts) + registry = _compile(facts) + return {spec.name: spec for spec in _district_file_specs(registry)} + + base = compiled(1.0, None) + factor = control / us_total + raw = {} + for fips, (first, second) in world.items(): + postal = _TWO_DISTRICT_STATES[fips] + raw[f"{postal}_total"] = first + second + raw[f"{postal}_01"], raw[f"{postal}_02"] = first, second + rows = { + spec.metadata["ledger_layout_groupby_value_id"]: spec for spec in base.values() + } + # The US row retires; every state and district binds. + assert set(rows) == set(raw) + for row, spec in rows.items(): + # One factor; never negative; never interest_deduction; never flagged. + assert spec.value == pytest.approx(raw[row] * factor, rel=1e-12) + assert spec.value >= 0.0 + assert spec.metadata["variable"] == "deductible_mortgage_interest" + assert "requires_soi_district_component_share" not in spec.metadata + # Districts sum to their state; states sum to the control's share of + # the US row that the bound states hold. + for fips in world: + postal = _TWO_DISTRICT_STATES[fips] + assert rows[f"{postal}_01"].value + rows[f"{postal}_02"].value == ( + pytest.approx(rows[f"{postal}_total"].value, rel=1e-12) + ) + state_sum = math.fsum(rows[f"{_TWO_DISTRICT_STATES[f]}_total"].value for f in world) + assert state_sum == pytest.approx(control * states_total / us_total, rel=1e-12) + if case["us_excess"] == 1.0: + assert state_sum == pytest.approx(control, rel=1e-12) + # A common scale of the file (its stamp, its units) changes no target + # value, only the recorded factor; feed order changes nothing at all. + scaled = compiled(case["scale"], None) + assert scaled.keys() == base.keys() + for name, spec in base.items(): + assert scaled[name].value == pytest.approx(spec.value, rel=1e-12) + assert scaled[name].metadata["variable"] == spec.metadata["variable"] + shuffled = compiled(1.0, case["seed"]) + assert shuffled == base diff --git a/tools/us_acs_local_cd_surface.py b/tools/us_acs_local_cd_surface.py index edd37055f..78d8092e3 100644 --- a/tools/us_acs_local_cd_surface.py +++ b/tools/us_acs_local_cd_surface.py @@ -87,11 +87,14 @@ #: closest amount Historic Table 2 carries. The factor band below applies to #: the sibling. The bridge is stamp-invariant: if the stamp is corrected #: (#1030) both levels move together. +#: +#: A district-file state row the compiler already bound as a share of a +#: national control (:func:`is_national_share_spec`; the line 8a mortgage +#: interest rows, microcosm#1038) has neither gap and needs no bridge: its +#: states sum to the control. Bridging it as well would lift it twice. STATE_CD_LEVEL_BRIDGES: Mapping[str, str] = { "charitable_amount": "itemized_deductions_amount", "charitable_returns": "return_count", - "interest_paid_deduction_amount": "itemized_deductions_amount", - "interest_paid_deduction_returns": "return_count", "qualified_business_income_deduction_amount": "adjusted_gross_income", "qualified_business_income_deduction_returns": "return_count", } @@ -196,6 +199,17 @@ def is_cd_file_spec(spec) -> bool: return record_set_spec_id(spec).startswith(CD_FILE_RECORD_SET_SPEC_PREFIX) +def is_national_share_spec(spec) -> bool: + """Whether the compiler bound this spec as a share of a national control. + + Such a spec names its share's denominator + (``soi_share_total_source_record_id``): its value is the control times its + share of its file's US row, already on the control's basis. + """ + + return bool(_metadata(spec).get("soi_share_total_source_record_id")) + + def soi_materializer_semantics(spec) -> tuple: """Everything the SOI slice reads from a spec, except its geography. @@ -462,14 +476,28 @@ def within_band( } # One basis for every state level: a concept Historic Table 2 lacks keeps - # the district file's state row, scaled by a sibling's two levels. + # the district file's state row, scaled by a sibling's two levels, unless + # the compiler already bound it as a share of a national control. bridged: dict[str, object] = {} bridge_factors: dict[str, list[float]] = defaultdict(list) + national_share_levels: Counter = Counter() for key, spec in sorted(cd_file_state_by_key.items()): state, identity = key measure = identity[0] if key in ht2_by_key or measure in STATE_CD_DEFECTIVE_CD_FILE_MEASURES: continue + if is_national_share_spec(spec): + metadata = dict(spec.metadata) + metadata.update( + { + "state_cd_vintage_rule": STATE_CD_VINTAGE_RULE, + "state_cd_cd_file_value": repr(float(spec.value)), + "state_cd_level_basis": "national_control_share", + } + ) + bridged[spec.name] = replace(spec, metadata=metadata) + national_share_levels[measure] += 1 + continue sibling = level_bridges.get(measure) if sibling is None: raise ValueError( @@ -574,6 +602,8 @@ def within_band( for child in children: drop(child, f"level_bridge_out_of_band:{identity[0]}") continue + if is_national_share_spec(parent): + basis = "cd_file_state_total_national_share" parent = bridged[parent.name] kept_cd_file_parents.add(parent.name) if soi_materializer_semantics(parent) != child_semantics: @@ -690,7 +720,9 @@ def within_band( "that parent. Concepts only the district file carries keep its " "own state row as their parent, lifted onto the Historic Table 2 " "basis by a sibling concept's ratio of the two files' state " - "totals in the same state (STATE_CD_LEVEL_BRIDGES)." + "totals in the same state (STATE_CD_LEVEL_BRIDGES), unless the " + "compiler bound that row as a share of a national control, " + "which is its basis already." ), "counts": { "historic_table_2_state": len(ht2_state), @@ -723,6 +755,7 @@ def within_band( measure: _factor_summary(values) for measure, values in sorted(bridge_factors.items()) }, + "national_share_levels_by_measure": dict(sorted(national_share_levels.items())), "rebase_factor_by_measure": { measure: _factor_summary(values) for measure, values in sorted(factors_by_measure.items())