diff --git a/changelog.d/1243-ht2-national-controls.fixed.md b/changelog.d/1243-ht2-national-controls.fixed.md new file mode 100644 index 000000000..2347dcce5 --- /dev/null +++ b/changelog.d/1243-ht2-national-controls.fixed.md @@ -0,0 +1 @@ +Rebase the Historic Table 2 state rows of 14 SOI columns (return count, AGI, wages, taxable IRA distributions, taxable pensions, unemployment compensation and taxable Social Security, returns and amounts; itemized deductions and taxable income amounts) onto their newer Pub 1304 Table 1.x national totals as shares, and retire the stale HT2 US rows and HT2's US-workbook return count. On the pinned feed, `national_state` no longer binds two national values for any of these quantities (they were 0.3% to 12.2% apart). The 51-state sums move from -2.2% (taxable IRA distributions) to +12.2% (taxable Social Security), and none exceeds its national target. The #940 state AGI bands move with their rebased state totals (microcosm#1243). diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index 400a897b6..d0b444fb8 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -213,7 +213,10 @@ chosen by concept and period alone counts are unchanged and `national_state` is `65e4dde11c83` on this feed. On the 2026-10-04 `51aa40f` re-pin, whose capital-gains cells are identical, the same 60 return counts move and `national_state` goes from `e1f9906bf0e0` to -`882f4764a248` at 6,102 targets. +`882f4764a248` at 6,102 targets. microcosm#1243 then rebases 14 more Historic +Table 2 state families onto their Table 1.x controls and retires 15 stale +national rows: 33,235 compiled and 6,087 on `national_state` (`0e899cd7f893`) +([us-soi-national-controls.md](us-soi-national-controls.md)). The labelled feed exposed one latent defect in microcosm, fixed in this change: `apply_us_medicaid_enrollment_substitutions` built Rhode Island's diff --git a/docs/us-fact-to-target.md b/docs/us-fact-to-target.md index e4ae417a8..1f9e11b8a 100644 --- a/docs/us-fact-to-target.md +++ b/docs/us-fact-to-target.md @@ -111,6 +111,28 @@ exclusion register instead. wiring is safe and normal (the keogh ALD facts rode the feed unmapped for weeks). +### One national level per quantity (microcosm#1243) + +When a new fact gives a quantity a second national all-AGI row, the surface +binds two levels of one model quantity, and no weights can meet both. The +usual case is a newer Pub 1304 Table 1.x row beside the stale Historic Table 2 +(HT2) US row. HT2 state rows become shares of the newer row in three passes: +taxable interest (#489), capital gains (#1035) and +`_rebase_stale_soi_ht2_distributions` for the 14 columns in +`_SOI_HT2_NATIONAL_CONTROLS`. That last register pairs each HT2 column with +the Table 1.x column of the same Form 1040 line. + +- **Before you register a column**, verify the concept match. Check the HT2 + documentation guide's line reference against the Table 1.x header, and + check that the same-year HT2 US row and Table 1.x total agree. The 14 + registered columns agree within 2.7% in TY2020-TY2023; capital gains + differ by 2.36x and need a declared bridge instead + ([us-soi-national-controls.md](us-soi-national-controls.md), + [us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). +- `test_pinned_feed_no_soi_state_family_sums_past_its_national_target` + fails a re-pin that adds a second national value for a compared quantity, + or a state family past its national target. + ### State AGI bands bind as shares of the state total (microcosm#940) A state's total return count and AGI leave its top tail free, and the top @@ -146,7 +168,9 @@ binds them: stay the same share of the state total they bind beside, at every stage. The control may be older than the bands (TY2023 bands on the TY2022 state total): the newest published shares scale onto the level the state total - binds at, rather than binding a second, differently aged level. + binds at, rather than binding a second, differently aged level. A state + total that is itself rebased onto a national control (#1243) sits at that + control's period, and its bands land there too. - **One vintage.** State bands are the one family selected by vintage, not per row. Every eligible vintage (not after the build period) of a binding band stays a candidate through latest-vintage selection, and diff --git a/docs/us-soi-national-controls.md b/docs/us-soi-national-controls.md new file mode 100644 index 000000000..ef1841ce2 --- /dev/null +++ b/docs/us-soi-national-controls.md @@ -0,0 +1,208 @@ +# SOI Historic Table 2 national controls + +Fourteen IRS SOI quantities reach the US target surface from two sources: the +Historic Table 2 (HT2) state file and a Pub 1304 Table 1.x national table. +This page records why the two count the same thing, which one supplies the +national level, and how `fiscal_targets.py` combines them (microcosm#1243). +The taxable-interest (#489, #518) and capital-gains (#1035, #1036) families +follow the same doctrine through their own passes. + +## The defect + +On the pinned feed (`51aa40f`, compiled at 2024 as the release does), the +`national_state` surface bound two national all-AGI targets for each of these +quantities: the TY2022 HT2 US row and a TY2023 Table 1.x row. Both measure +the same model variable, filing status, return universe and filters, so no +weight vector can meet both. They were 0.3% to 12.2% apart. For +`ira_distributions` sum and `taxable_pension_income` count, the 51 HT2 state +rows also summed past the Table 1.4 target (1.019x and 1.009x). The return +count had a fourth national row: HT2's US workbook (`22in54us.xlsx`, +"Historic Table 2 state data, United States total"), which the feed packages +as `irs_soi.ty2023.state_2022.us`: TY2022 data stamped ty2023. + +## The columns and what they count + +Each HT2 column is defined at the same Form 1040 line as its Table 1.4 +column. The HT2 documentation guides (`20incmdocguide.doc` to +`23incmdocguide.doc`) give the line for every year, and the Table 1.4 +headers (`20in14ar.xls` to `23in14ar.xls`) name the same item. The HT2 US +row also matches Table 1.4's "All returns, total" cell in the same tax year, +for every year: + +| HT2 column | Form 1040 line | Control | HT2 / Table 1.4, TY2020-TY2023 | Model variable | +|---|---|---|---:|---| +| N1 | returns | Table 1.1 `return_count` | 0.990-0.998 | `count` | +| A00100 | 11 | Table 1.1 `adjusted_gross_income` | 0.997-0.999 | `adjusted_gross_income` | +| N00200 / A00200 | 1z (1 before TY2022) | Table 1.4 `wages_salaries_*` | 0.995-0.999 / 0.997-0.999 | `employment_income` | +| N01400 / A01400 | 4b | Table 1.4 `taxable_ira_distributions_*` | 0.999-1.009 / 0.992-1.001 | `ira_distributions` | +| N01700 / A01700 | 5b | Table 1.4 `taxable_pension_income_*` | 0.996-1.005 / 0.986-0.996 | `taxable_pension_income` | +| N02300 / A02300 | Schedule 1, 7 | Table 1.4 `unemployment_compensation_*` | 0.983-1.002 / 0.982-0.994 | `unemployment_compensation` | +| N02500 / A02500 | 6b | Table 1.4 `taxable_social_security_*` | 0.991-0.996 / 0.993-1.002 | `taxable_social_security` | +| A04470 | 12 (12a in TY2021) | Table 1.2 `total_itemized_deductions_amount` | 0.974-0.998 | `itemized_taxable_income_deductions` | +| A04800 | 15 | Table 1.2 `taxable_income_amount` | 0.997-0.999 | `taxable_income` | + +- The return count and AGI come from Table 1.1, whose "all returns" cells + Table 1.4 repeats. +- The feed carries the Table 1.4 itemized-deduction and taxable-income + totals (cols 135-136 and 141-142) only in Table 1.2. Table 1.2 prints the + same TY2023 cells: $690.845B and $11,944.447B. +- The same-year differences are small, and HT2 is usually the lower. The + widest is itemized deductions in TY2022 (HT2 is 2.6% below). None + approaches the 2.36x of the capital-gains columns, which count different + populations ([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). +- Their cause is not established. HT2 tabulates the full population of + returns the IRS processed in the following calendar year, using prior-year + returns as a proxy for late filers. The HT2 guide's endnote 1 says that + practice is consistent with SOI's national tabulations, and Pub 1304's + sample is drawn from the same processing year (Pub 1304, "Domain of + Study"). Sampling error and SOI's record editing are candidates, untested + here. Either way, the share step below removes the difference. +- Both rows of each pair map to the same model variable and measure mode. + `test_each_registered_column_and_its_control_bind_the_same_model_quantity` + checks this against the compile's measure maps. + +The identities are reproduced by +`~/PolicyEngine/_reviews/soi-national-controls-20261012/identities.py`, which +locates each Table 1.4 column by its header text. The Table 1.4 workbooks +for TY2020-TY2023 and the HT2 files for TY2022 and TY2023 are byte-identical +(sha256) to the feed's sources; the TY2020 and TY2021 HT2 files come from +irs.gov. + +## Where the gap came from + +The aged HT2 US row differs from the aged control by two factors: + +- **Vintage.** HT2 amounts age from TY2022 to TY2023 by realized AGI growth + (wages by realized wage growth); counts do not age. Each column's own + change differed. +- **Same-year difference.** HT2's TY2022 total differs slightly from Table + 1.4's (previous section). + +| Quantity | Table 1.4 TY2023 / TY2022 | HT2 aging link | Vintage factor | HT2 / Table 1.4, TY2022 | National max / min | +|---|---:|---:|---:|---:|---:| +| Taxable Social Security, amount | 1.1495 | 1.0305 | 1.1155 | 0.9943 | 1.1219 | +| Taxable Social Security, returns | 1.0425 | 1 | 1.0425 | 0.9922 | 1.0507 | +| Itemized deductions, amount | 1.0342 | 1.0305 | 1.0036 | 0.9738 | 1.0306 | +| Taxable IRA distributions, amount | 1.0009 | 1.0305 | 0.9713 | 0.9929 | 1.0223 | +| Taxable IRA distributions, returns | 1.0253 | 1 | 1.0253 | 1.0091 | 1.0161 | +| Taxable pensions, returns | 0.9840 | 1 | 0.9840 | 0.9977 | 1.0139 | +| Unemployment compensation, returns | 0.9934 | 1 | 0.9934 | 0.9829 | 1.0107 | +| Taxable income, amount | 1.0197 | 1.0305 | 0.9895 | 0.9970 | 1.0076 | +| Wages, returns | 1.0016 | 1 | 1.0016 | 0.9947 | 1.0069 | +| Taxable pensions, amount | 1.0224 | 1.0305 | 0.9922 | 0.9857 | 1.0066 | +| Return count | 0.9954 | 1 | 0.9954 | 0.9896 | 1.0060 | +| AGI | 1.0305 | 1.0305 | 1 | 0.9965 | 1.0035 | +| Wages, amount | 1.0478 | 1.0478 | 1 | 0.9969 | 1.0031 | +| Unemployment compensation, amount | 1.0229 | 1.0305 | 0.9926 | 0.9898 | 1.0028 | + +The last column is the vintage factor divided by the TY2022 ratio, or its +inverse. Taxable Social Security grew 15.0% from TY2022 to TY2023, far past +AGI's 3.05%. + +## How the targets combine + +`_rebase_stale_soi_ht2_distributions` runs after the capital-gains rebase and +before the state AGI bands. + +- **Controls.** `_SOI_HT2_NATIONAL_CONTROLS` registers one Pub 1304 table and + column per HT2 column. Only a national all-AGI fact of that table and + column qualifies, and the latest one not after the build period wins. HT2, + HT2's US workbook, the congressional-district file, Table 4.3 (excluding + dependents) and Table 2.1 (itemizers) never qualify. Two different records + at the winning period raise `AmbiguousSoiNationalControlError`, so feed + order never decides. The capital-gains control uses the same chooser. +- **Shares.** Each all-AGI HT2 state row becomes its share of the HT2 US + total, scaled to the control: + `value x control / HT2 US`. The value lands at the control's period and + target aging adds the remaining years, so the control and its states age by + the same factor. Each row records `uprating_factor`, the control's record + id, and `soi_source_concept` / `soi_control_concept` (the Form 1040 line, + equal on both sides). +- **One national level.** Once a quantity's states are shares of the + control, the HT2-lineage national all-AGI rows of that quantity retire: + the HT2 US row and HT2's US workbook row. Matching is by model quantity + (variable, mode, filing status, universe and filters), so the workbook's + EITC row, which shares the `return_count` measure id, stays. Table 1.2's + copies of Table 1.1's count and AGI equal the control and stay. +- **Per-row eligibility.** A row is rebased or retired only when the + control is at or after its own data year. The workbook's data year comes + from its family name (`state_2022`), not its ty2023 stamp. Where HT2 + vintages mix (a state falling back to an older year), the older rows + become shares and a newer HT2 US row keeps its level. +- **No eligible control.** When the control is older than the HT2 row, + after the build period, or missing, or when the HT2 US total is missing, + the HT2 rows keep their levels and the HT2 US row stays. +- **One denominator.** Two different HT2 US records for one measure, filing + status, universe and year raise `AmbiguousSoiNationalControlError`, so + feed order cannot pick the rebase denominator. +- **State AGI bands.** The #940 bands are shares of the state total that + binds. A rebased state total sits at its control's period, and + `_soi_state_agi_band_controls` now reads that landing period + (`uprating_to_period`), so the AGI bands age with the total from TY2023. + Without that, AGI bands would age from TY2022 and overshoot their total by + 3.05%. + +## Effect on the pinned feed + +Compiled at 2024 with target aging and the packaged congressional-district +crosswalk, then narrowed to `national_state`. Before is `main` plus #1036. + +| Quantity | 51 states, before | 51 states, after | Move | States / smaller national, before | after | +|---|---:|---:|---:|---:|---:| +| `taxable_social_security` sum | $509.58B | $571.71B | +12.19% | 0.998 | 0.998 | +| `taxable_social_security` count | 24,397,150 | 25,634,869 | +5.07% | 0.997 | 0.997 | +| `itemized_taxable_income_deductions` sum | $726.41B | $748.63B | +3.06% | 0.997 | 0.997 | +| `ira_distributions` sum | $485.38B | $474.78B | -2.18% | **1.019** | 0.997 | +| `ira_distributions` count | 16,385,750 | 16,649,213 | +1.61% | 0.997 | 0.997 | +| `taxable_pension_income` count | 29,818,930 | 29,410,814 | -1.37% | **1.009** | 0.996 | +| `unemployment_compensation` count | 4,637,960 | 4,687,496 | +1.07% | 0.998 | 0.998 | +| `taxable_income` sum | $12,978.42B | $12,880.42B | -0.76% | 0.999 | 0.992 | +| `employment_income` count | 127,029,870 | 127,910,705 | +0.69% | 0.995 | 0.995 | +| `taxable_pension_income` sum | $1,001.75B | $1,008.37B | +0.66% | 0.995 | 0.995 | +| `count` | 158,710,000 | 159,655,171 | +0.60% | 0.994 | 0.994 | +| `adjusted_gross_income` sum | $16,441.78B | $16,499.02B | +0.35% | 0.993 | 0.993 | +| `employment_income` sum | $10,654.77B | $10,687.80B | +0.31% | 0.992 | 0.992 | +| `unemployment_compensation` sum | $33.43B | $33.52B | +0.28% | 0.997 | 0.997 | + +- Every state row of a family moves by the same factor. After the change, + the 51 states sum to the control times their share of the HT2 US row in + `22in55cmcsv.csv`, exactly (error at most 4e-16). +- 15 national rows retire: the 14 HT2 US rows and the workbook's return + count. No row is added. Compiled targets go from 33,250 to 33,235, and + `national_state` from 6,102 (`882f4764a248`) to 6,087 (`0e899cd7f893`); + the full registry from `f09571684885` to `0bbbad9b22d1`. +- 1,221 values move, each family by one factor: + - the 714 HT2 state rows on `national_state` (14 x 51); + - 99 HT2 rows on `full` only, which stand in for at-large congressional + districts (9 states x 11 measures); + - the 408 #940 state AGI bands, by their state total's factor (AGI + x1.00348, returns x1.00596). Each stays its recorded share of the total + (error at most 6e-16). +- On `national_state`, the 63 compared SOI families each sum to at most the + smaller national target of their quantity, and every compared quantity has + one national value. Before, 14 had two values and two state families summed + past the smaller one. + +The diff and checks are reproduced by `scripts/diff_compiles.py` in the +evidence folder. + +## What this does not change + +- **Share assumption.** TY2022 HT2 state shares stand in for TY2023 shares. + HT2 TY2023 is published (`23in55cmcsv.csv`, already the source of the + #940 bands). If Chronicle packages its state totals under the same record + sets, latest-vintage selection, which strips the period from record-set + ids, keeps them instead, and this pass rebases them onto the TY2023 + controls by the same-year difference alone + (`test_a_control_at_the_hts_own_period_still_rebases`). +- **Congressional-district rows** of these columns stay TY2022 levels on the + `full` surface. They are not on `national_state`; #1040, #1229 and #1231 + reconcile other district families. +- **Columns without an all-returns control** keep their HT2 levels: N04470 + (itemizing returns) and N04800 (returns with taxable income) appear + nationally only in Table 4.3, which excludes dependent filers. +- **A newer HT2 than its control.** If HT2 ever lands a tax year before + Table 1.x, the HT2 rows keep their levels and the HT2 US row stays beside + the older control. The pinned-feed invariant then fails on the re-pin. +- **Wages concept.** Both wage rows bind gross `employment_income`; + microcosm#1191 covers the engine taxing `irs_employment_income`. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 5ba9bb5bc..ffe871493 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -1068,6 +1068,11 @@ def compile_us_fiscal_target_registry( materialized_facts, target_period=target_period, ) + registry = _rebase_stale_soi_ht2_distributions( + registry, + materialized_facts, + target_period=target_period, + ) registry = _rebase_soi_state_agi_bands(registry, materialized_facts) registry = apply_ledger_target_profile( registry, @@ -1623,15 +1628,39 @@ def _soi_capital_gains_active_totals( September re-pin, which re-sorted the feed by content hash, handed the returns control to the congressional-district US row (microcosm#1035). """ + return _latest_unambiguous_soi_controls( + ( + (key, fact) + for fact in facts + if (key := _soi_capital_gains_control_key_from_fact(fact)) is not None + and _soi_capital_gains_concept(fact) == _SOI_CAPITAL_GAINS_MODEL_CONCEPT + ), + target_period=target_period, + error=AmbiguousSoiCapitalGainsControlError, + what="SOI capital-gains control", + issue="microcosm#1035", + ) + + +def _latest_unambiguous_soi_controls( + candidates: Iterable[tuple[tuple[str, str, str], object]], + *, + target_period: int | str, + error: type[ValueError], + what: str, + issue: str, +) -> dict[tuple[str, str, str], _SoiTotalControl]: + """The latest candidate fact per key, not after the build period. + + The choice depends on the fact set alone: two different candidates at the + winning period (different records, or one record with different values + or period labels) raise ``error``, and one record seen twice is one + candidate. A tie at a superseded period does not matter. + """ controls: dict[tuple[str, str, str], _SoiTotalControl] = {} tied: dict[tuple[str, str, str], set[tuple[str, float, str]]] = {} target_period_key = _period_key_from_value(target_period) - for fact in facts: - key = _soi_capital_gains_control_key_from_fact(fact) - if key is None: - continue - if _soi_capital_gains_concept(fact) != _SOI_CAPITAL_GAINS_MODEL_CONCEPT: - continue + for key, fact in candidates: period_key = _period_key(fact) if not _not_after_target_period(period_key, target_period_key): continue @@ -1653,15 +1682,15 @@ def _soi_capital_gains_active_totals( controls[key] = candidate tied[key] = {identity} ambiguous = { - key: sorted(candidates) - for key, candidates in tied.items() - if len(candidates) > 1 + key: sorted(identities) + for key, identities in tied.items() + if len(identities) > 1 } if ambiguous: - raise AmbiguousSoiCapitalGainsControlError( - "SOI capital-gains control is ambiguous: different candidates tie at " - f"the latest period for {ambiguous}. Deduplicate them upstream; " - "feed order must not pick a rebase control (microcosm#1035)." + raise error( + f"{what} is ambiguous: different candidates tie at the latest period " + f"for {ambiguous}. Deduplicate them upstream; feed order must not " + f"pick a rebase control ({issue})." ) return controls @@ -1780,6 +1809,410 @@ def _soi_capital_gains_uprating_index(kind: str) -> str: return "total_net_capital_gains_amount" +@dataclass(frozen=True) +class _SoiHt2NationalControlRule: + """The national control of one Historic Table 2 column. + + ``concept`` names the Form 1040 line both columns report; the control is + the latest ``control_measure_id`` fact of the ``control_family`` table. + """ + + control_family: str + control_measure_id: str + concept: str + + +#: Historic Table 2 columns whose national level comes from a Pub 1304 table +#: (microcosm#1243). Each pair was checked two ways for TY2020-TY2023: +#: +#: - The HT2 documentation guides (``20incmdocguide.doc`` to +#: ``23incmdocguide.doc``) define the HT2 column at the same Form 1040 line +#: as the Table 1.4 column. +#: - The HT2 US row matches Table 1.4's same-year "All returns, total" cell +#: within 2.7%: HT2 / Table 1.4 runs from 0.974 (itemized deductions, +#: TY2022) to 1.009 (IRA distribution returns, TY2022). The cause of that +#: small same-year difference is not established: HT2 is the full +#: population of returns processed in the following calendar year, and +#: Pub 1304 samples the same processing year. The capital-gains columns, +#: by contrast, differ by 2.36x because they count different populations +#: (see ``_SOI_CAPITAL_GAINS_FAMILY_CONCEPTS``). +#: +#: The gap between an aged HT2 US row and its control is therefore vintage +#: times that same-year difference, not a concept difference. HT2 is +#: TY2022 while the controls are TY2023, and HT2 amounts age 2022->2023 by +#: realized AGI growth (wages by realized wage growth), which cannot reproduce +#: each column's own change: taxable Social Security grew 15.0% and taxable +#: IRA distributions 0.1%. As shares of the control, the state rows carry +#: neither component. +#: +#: The feed carries the Table 1.4 itemized-deduction and taxable-income totals +#: only in Table 1.2, which prints the same TY2023 cells ($690.845B and +#: $11,944.447B). Table 1.1 supplies the return count and AGI; Table 1.2 +#: repeats them and stays a national target of equal value. A column absent +#: from this register keeps its HT2 level and its HT2 national row. +_SOI_HT2_NATIONAL_CONTROLS: dict[str, _SoiHt2NationalControlRule] = { + "return_count": _SoiHt2NationalControlRule( + "table_1_1", "return_count", "form_1040_returns" + ), + "adjusted_gross_income": _SoiHt2NationalControlRule( + "table_1_1", "adjusted_gross_income", "form_1040_line_11_agi" + ), + "wages_salaries_amount": _SoiHt2NationalControlRule( + "table_1_4", "wages_salaries_amount", "form_1040_line_1z_wages" + ), + "wages_salaries_returns": _SoiHt2NationalControlRule( + "table_1_4", "wages_salaries_returns", "form_1040_line_1z_wages" + ), + "taxable_ira_distributions_amount": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_ira_distributions_amount", + "form_1040_line_4b_taxable_ira_distributions", + ), + "taxable_ira_distributions_returns": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_ira_distributions_returns", + "form_1040_line_4b_taxable_ira_distributions", + ), + "taxable_pension_income_amount": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_pension_income_amount", + "form_1040_line_5b_taxable_pensions_and_annuities", + ), + "taxable_pension_income_returns": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_pension_income_returns", + "form_1040_line_5b_taxable_pensions_and_annuities", + ), + "unemployment_compensation_amount": _SoiHt2NationalControlRule( + "table_1_4", + "unemployment_compensation_amount", + "schedule_1_line_7_unemployment_compensation", + ), + "unemployment_compensation_returns": _SoiHt2NationalControlRule( + "table_1_4", + "unemployment_compensation_returns", + "schedule_1_line_7_unemployment_compensation", + ), + "taxable_social_security_amount": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_social_security_amount", + "form_1040_line_6b_taxable_social_security", + ), + "taxable_social_security_returns": _SoiHt2NationalControlRule( + "table_1_4", + "taxable_social_security_returns", + "form_1040_line_6b_taxable_social_security", + ), + "itemized_deductions_amount": _SoiHt2NationalControlRule( + "table_1_2", + "total_itemized_deductions_amount", + "form_1040_line_12_itemized_deductions", + ), + "taxable_income_amount": _SoiHt2NationalControlRule( + "table_1_2", "taxable_income_amount", "form_1040_line_15_taxable_income" + ), +} +_SOI_HT2_MEASURE_BY_CONTROL: dict[tuple[str, str], str] = { + (rule.control_family, rule.control_measure_id): measure_id + for measure_id, rule in _SOI_HT2_NATIONAL_CONTROLS.items() +} + + +class AmbiguousSoiNationalControlError(ValueError): + """Two different records tie for one Historic Table 2 national control.""" + + +def _rebase_stale_soi_ht2_distributions( + registry: TargetRegistry, + facts: tuple[object, ...], + *, + target_period: int | str, +) -> TargetRegistry: + """Use Historic Table 2 rows as shares of their national control (#1243). + + Each column in ``_SOI_HT2_NATIONAL_CONTROLS`` counts the same Form 1040 + line as its Pub 1304 control, so an HT2 state row is a valid share of the + nation but a stale level. Each all-AGI HT2 state row ships as its share of + the HT2 US total, scaled to the control of ``_soi_ht2_national_controls`` + at or after its own period. The value lands at the control's period, and + target aging adds the remaining years from there. Before this, 14 + quantities bound a TY2022 HT2 US row beside a TY2023 Table 1.x row, + 0.3% to 12.2% apart, and two state families summed past the newer one. + + Once a quantity's states are shares of the control, the control's row is + its national level, so the HT2 lineage's national all-AGI rows of that + quantity retire if the control is at or after their own data year. Two + rows qualify: the HT2 US row, and the same total from HT2's US workbook + (``22in54us.xlsx``), which the feed packages as + ``irs_soi.ty2023.state_2022.us`` (TY2022 data; its data year comes from + the family name, not the ty2023 stamp). The control's own family may + repeat the total in another table (Table 1.2 prints Table 1.1's count + and AGI); those rows are equal to the control and stay. + + Eligibility is per row. A row whose control is older than the row, or + whose HT2 US total is missing, keeps its HT2 level; a national row newer + than the control stays even when older state rows of its quantity are + rebased (a mixed-vintage fallback). + """ + + controls = _soi_ht2_national_controls(facts, target_period=target_period) + source_totals = _soi_ht2_us_totals(facts) + # Per rebased quantity, the period of the control its rows took. + retired: dict[tuple[object, ...], tuple[int, int, str]] = {} + specs: list[TargetSpec] = [] + for spec in registry.specs: + measure_id = _soi_ht2_rebase_measure(spec) + if measure_id is None: + specs.append(spec) + continue + key = _soi_ht2_control_key_from_spec(spec, measure_id) + source_period = spec.metadata.get("source_period", "") + control = controls.get(key) + source_total = source_totals.get((*key, source_period)) + if ( + control is None + or not _period_not_before( + control.period_key, _period_key_from_value(source_period) + ) + or source_total in (None, 0) + ): + specs.append(spec) + continue + retired[_soi_national_quantity(spec)] = control.period_key + if _is_national_all_agi_spec(spec): + continue + concept = _SOI_HT2_NATIONAL_CONTROLS[measure_id].concept + factor = control.value / source_total + specs.append( + replace( + spec, + value=spec.value * factor, + metadata={ + **dict(spec.metadata), + "uprating_index": f"total_{measure_id}", + "uprating_from_period": source_period, + # Lands at the CONTROL's period; target aging completes + # the remaining links (#488). + "uprating_to_period": control.source_period, + "uprating_index_source_period": control.source_period, + "uprating_index_source_record_id": control.source_record_id, + "uprating_factor": _format_float(factor), + "stale_distribution_rebased_to_active_total": "true", + "soi_source_concept": concept, + "soi_control_concept": concept, + }, + ) + ) + return TargetRegistry( + [ + spec + for spec in specs + if not _is_retired_soi_ht2_national_spec(spec, retired) + ], + country=registry.country, + ) + + +def _soi_ht2_national_controls( + facts: tuple[object, ...], + *, + target_period: int | str, +) -> dict[tuple[str, str, str], _SoiTotalControl]: + """The national control per (HT2 measure, filing status, universe). + + Only a national all-AGI fact of the table and column registered for the + HT2 measure qualifies, so HT2, congressional-district and + excluding-dependents rows never do, whatever their period stamp. The + latest qualifying period not after the build period wins. Two different + candidates at that period (different records, or one record with + different values or period labels) refuse the compile, so feed order never + picks a control (microcosm#1035). + """ + return _latest_unambiguous_soi_controls( + ( + (key, fact) + for fact in facts + if (key := _soi_ht2_control_key_from_fact(fact)) is not None + ), + target_period=target_period, + error=AmbiguousSoiNationalControlError, + what="SOI Historic Table 2 national control", + issue="microcosm#1243", + ) + + +def _soi_ht2_control_key_from_fact(fact: object) -> tuple[str, str, str] | None: + """(HT2 measure, filing status, universe) a control fact anchors, if any.""" + if _source_name(fact) != "irs_soi": + return None + if _geography_level(fact) != "country": + return None + if not _is_all_agi_range_fact(fact): + return None + family = _soi_record_set_family(_str_at(fact, "layout", "record_set_id")) + measure_id = _SOI_HT2_MEASURE_BY_CONTROL.get((family, _measure_id(fact))) + if measure_id is None: + return None + status = _filing_status_label(_dimensions(fact).get("filing_status")) + if status is None: + return None + return (measure_id, status, _soi_return_universe_from_fact(fact)) + + +def _soi_ht2_us_totals( + facts: tuple[object, ...], +) -> dict[tuple[str, str, str, str], float]: + """HT2 US all-AGI totals per (measure, filing status, universe, period). + + The total is a rebase denominator, so feed order must not pick it: two + different candidates for one key (different records, or one record with + different values) raise. One record seen twice is one candidate. + """ + totals: dict[tuple[str, str, str, str], float] = {} + candidates: dict[tuple[str, str, str, str], set[tuple[str, float]]] = {} + for fact in facts: + if _source_name(fact) != "irs_soi": + continue + if _geography_level(fact) != "country": + continue + if not _is_all_agi_range_fact(fact): + continue + record_set_id = _str_at(fact, "layout", "record_set_id") + if _soi_record_set_family(record_set_id) != "historic_table_2": + continue + measure_id = _measure_id(fact) + if measure_id not in _SOI_HT2_NATIONAL_CONTROLS: + continue + status = _filing_status_label(_dimensions(fact).get("filing_status")) + if status is None: + continue + key = ( + measure_id, + status, + _soi_return_universe_from_fact(fact), + str(_period_value(fact)), + ) + value = _numeric_value(fact) + candidates.setdefault(key, set()).add((_source_record_id(fact), value)) + totals[key] = value + ambiguous = { + key: sorted(identities) + for key, identities in candidates.items() + if len(identities) > 1 + } + if ambiguous: + raise AmbiguousSoiNationalControlError( + "SOI Historic Table 2 US total is ambiguous: different candidates " + f"share a key: {ambiguous}. Deduplicate them upstream; feed order " + "must not pick a rebase denominator (microcosm#1243)." + ) + return totals + + +def _soi_ht2_rebase_measure(spec: TargetSpec) -> str | None: + """The registered HT2 measure of an all-AGI HT2 spec, else ``None``. + + AGI-band rows are bounded and stay with ``_rebase_soi_state_agi_bands``, + which scales them to the state total this pass produces. + """ + if spec.family != "irs_soi": + return None + measure_id = spec.metadata.get("source_measure_id", "") + if measure_id not in _SOI_HT2_NATIONAL_CONTROLS: + return None + record_set_id = spec.metadata.get("ledger_layout_record_set_id", "") + if _soi_record_set_family(record_set_id) != "historic_table_2": + return None + if _bounds_from_metadata(spec) != (-float("inf"), float("inf")): + return None + return measure_id + + +def _soi_ht2_control_key_from_spec( + spec: TargetSpec, + measure_id: str, +) -> tuple[str, str, str]: + return ( + measure_id, + spec.metadata.get("filing_status", ""), + spec.metadata.get("soi_return_universe", "all_returns"), + ) + + +def _soi_national_quantity(spec: TargetSpec) -> tuple[object, ...]: + """The model quantity a spec measures, apart from its geography.""" + metadata = spec.metadata + return ( + metadata.get("variable"), + metadata.get("measure_mode"), + metadata.get("agi_lower_bound"), + metadata.get("agi_upper_bound"), + metadata.get("filing_status"), + metadata.get("soi_return_universe", "all_returns"), + metadata.get("itemized_only"), + tuple( + sorted( + (key, value) + for key, value in metadata.items() + if key.startswith("ledger_filter_") + ) + ), + ) + + +def _is_soi_ht2_lineage_record_set(record_set_id: str) -> bool: + """Whether a record set publishes Historic Table 2 data. + + Besides ``historic_table_2`` itself, the feed packages HT2's US workbook + (``22in54us.xlsx``, "Historic Table 2 state data, United States total") as + ``irs_soi..state_.us``. + """ + if _soi_record_set_family(record_set_id) == "historic_table_2": + return True + return _soi_ht2_lineage_data_year(record_set_id) is not None + + +def _is_retired_soi_ht2_national_spec( + spec: TargetSpec, + retired: Mapping[tuple[object, ...], tuple[int, int, str]], +) -> bool: + """Whether an HT2-lineage national row lost its quantity to a control. + + Only when the quantity was rebased and that control is at or after the + row's own data year: a newer HT2 national row keeps its level. + """ + if spec.family != "irs_soi" or not retired: + return False + if not _is_national_all_agi_spec(spec): + return False + record_set_id = spec.metadata.get("ledger_layout_record_set_id", "") + if not _is_soi_ht2_lineage_record_set(record_set_id): + return False + control_period_key = retired.get(_soi_national_quantity(spec)) + if control_period_key is None: + return False + return _period_not_before( + control_period_key, + _period_key_from_value( + _soi_ht2_lineage_data_year(record_set_id) + or spec.metadata.get("source_period", "") + ), + ) + + +def _soi_ht2_lineage_data_year(record_set_id: str) -> str | None: + """The data year a ``state_`` US-workbook record set names. + + The feed stamps ``irs_soi.ty2023.state_2022.us`` ty2023 although it holds + TY2022 data (PolicyEngine/chronicle#292); the family name carries the + data year. Other record sets return ``None``. + """ + prefix, _, year = _soi_record_set_family(record_set_id).partition("_") + if prefix == "state" and year.isdigit() and len(year) == 4: + return year + return None + + #: Control universes admitted for the stale-interest rebase. A stale spec's #: own universe always outranks the bridge, and the ONE admitted bridge is #: all_returns -> returns_excluding_dependents: the Pub 1304 excluding- @@ -2428,6 +2861,11 @@ def _soi_state_agi_band_controls( if metadata.get("soi_return_universe", "all_returns") != "all_returns": continue source_period = metadata.get("source_period", "") + if "uprating_factor" in metadata: + # A total rebased onto its national control (#1243) holds a value + # at the control's period, and target aging ages it from there + # (``_age_spec``); its bands must land at the same period. + source_period = metadata.get("uprating_to_period") or source_period candidate = _SoiTotalControl( value=spec.value, source_period=source_period, diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index aaed3a00a..9f9031f85 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -893,10 +893,13 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No def test_pinned_feed_national_state_surface_restores_the_fences( pinned_feed_national_state_surface, ) -> None: - """The release surface on the pinned feed: 33,250 compiled targets after - Medicaid substitution, 6,102 national_state targets at registry - 882f4764a248, 32 CHIP rows none of them for an M-CHIP state, no - other-income row and no tips return count. microcosm#1035 moved it from + """The release surface on the pinned feed: 33,235 compiled targets after + Medicaid substitution, 6,087 national_state targets at registry + 0e899cd7f893, 32 CHIP rows none of them for an M-CHIP state, no + other-income row and no tips return count. microcosm#1243 took it from + 33,250 / 6,102 at 882f4764a248 by rebasing 14 Historic Table 2 state + families onto their Table 1.x controls and retiring the 14 HT2 US rows + and HT2's US-workbook return count. microcosm#1035 moved it from e1f9906bf0e0 at the same counts by choosing the capital-gains control by concept and period, which returns the 51 Historic Table 2 state return counts to Table 1.4 shares. microcosm#1206 moved the @@ -909,9 +912,9 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77 (docs/us-chronicle-feed-repin.md).""" registry, surface, _ = pinned_feed_national_state_surface - assert len(registry.specs) == 33_250 - assert len(surface.specs) == 6_102 - assert surface.version == "882f4764a248" + assert len(registry.specs) == 33_235 + assert len(surface.specs) == 6_087 + assert surface.version == "0e899cd7f893" chip = [ spec for spec in surface.specs @@ -988,51 +991,162 @@ def test_pinned_feed_no_soi_state_family_sums_past_its_national_target( pinned_feed_national_state_surface, ) -> None: """Invariant: the states partition the nation (HT2 state rows omit only - other areas and Puerto Rico), so on national_state no SOI state family may - sum past every national target of the same model quantity and filters. - Before microcosm#1035 the capital_gains_gross return counts, one of the 63 - compared families, summed to 2.39x theirs. Some quantities still carry - both a stale HT2 US row and a newer Table 1.x row, a few percent apart, so - the bound is the larger of them.""" + other areas and Puerto Rico), so on national_state each SOI state family + sums to at most every national target of the same model quantity and + filters, and those national targets agree on one value. Before + microcosm#1035 the capital_gains_gross return counts, one of the 63 + compared families, summed to 2.39x theirs. Before microcosm#1243, 14 + quantities bound a stale HT2 US row beside a newer Table 1.x row, 0.3% to + 12.2% apart, and two state families (ira_distributions sum, + taxable_pension_income count) summed past the smaller one. Table 1.2 + repeats Table 1.1's return count and AGI at the same value.""" _, surface, _ = pinned_feed_national_state_surface - def quantity(spec) -> tuple[object, ...]: - metadata = spec.metadata - return ( - metadata.get("variable"), - metadata.get("measure_mode"), - metadata.get("agi_lower_bound"), - metadata.get("agi_upper_bound"), - metadata.get("filing_status"), - metadata.get("soi_return_universe", "all_returns"), - metadata.get("itemized_only"), - tuple( - sorted( - (key, value) - for key, value in metadata.items() - if key.startswith("ledger_filter_") - ) - ), - ) - national: dict[tuple[object, ...], list[float]] = {} states: dict[tuple[object, ...], float] = {} for spec in surface.specs: if spec.family != "irs_soi": continue + quantity = fiscal_targets._soi_national_quantity(spec) if spec.metadata.get("state_fips"): - states[quantity(spec)] = states.get(quantity(spec), 0.0) + spec.value + states[quantity] = states.get(quantity, 0.0) + spec.value else: - national.setdefault(quantity(spec), []).append(spec.value) + national.setdefault(quantity, []).append(spec.value) compared = {key: total for key, total in states.items() if key in national} assert len(compared) == 63 assert { - key[:2]: total / max(national[key]) + key[:2]: max(national[key]) / min(national[key]) + for key in compared + if not math.isclose(max(national[key]), min(national[key]), rel_tol=1e-12) + } == {} + assert { + key[:2]: total / min(national[key]) for key, total in compared.items() - if total > max(national[key]) + if total > min(national[key]) } == {} +#: Per Historic Table 2 measure on the pinned feed: the control, the factor +#: control / HT2 US, and the 51 states' share of the control after aging, +#: which equals their share of the HT2 US row in 22in55cmcsv.csv. +_PINNED_HT2_CONTROLS = { + "adjusted_gross_income": ( + "irs_soi.ty2023.table_1_1.all.adjusted_gross_income", + "1.03406226790832", + 0.9927705794863034, + ), + "itemized_deductions_amount": ( + "irs_soi.ty2023.table_1_2.all_returns.all.total_itemized_deductions_amount", + "1.06199716446366", + 0.9967186110188511, + ), + "return_count": ( + "irs_soi.ty2023.table_1_1.all.return_count", + "1.0059553340395", + 0.9941038386589074, + ), + "taxable_income_amount": ( + "irs_soi.ty2023.table_1_2.all_returns.all.taxable_income_amount", + "1.02269332455933", + 0.9918572450214445, + ), + "taxable_ira_distributions_amount": ( + "irs_soi.ty2023.table_1_4.all.taxable_ira_distributions_amount", + "1.00797641266581", + 0.9966807616472636, + ), + "taxable_ira_distributions_returns": ( + "irs_soi.ty2023.table_1_4.all.taxable_ira_distributions_returns", + "1.0160787779413", + 0.9973079699427512, + ), + "taxable_pension_income_amount": ( + "irs_soi.ty2023.table_1_4.all.taxable_pension_income_amount", + "1.03728909060027", + 0.9950132662033392, + ), + "taxable_pension_income_returns": ( + "irs_soi.ty2023.table_1_4.all.taxable_pension_income_returns", + "0.986313541255929", + 0.9955834839393801, + ), + "taxable_social_security_amount": ( + "irs_soi.ty2023.table_1_4.all.taxable_social_security_amount", + "1.15610313337446", + 0.9976736652240566, + ), + "taxable_social_security_returns": ( + "irs_soi.ty2023.table_1_4.all.taxable_social_security_returns", + "1.05073211171031", + 0.9968155377569604, + ), + "unemployment_compensation_amount": ( + "irs_soi.ty2023.table_1_4.all.unemployment_compensation_amount", + "1.03338314589632", + 0.9965852213837815, + ), + "unemployment_compensation_returns": ( + "irs_soi.ty2023.table_1_4.all.unemployment_compensation_returns", + "1.01068061430422", + 0.9978699874781082, + ), + "wages_salaries_amount": ( + "irs_soi.ty2023.table_1_4.all.wages_salaries_amount", + "1.05100939239557", + 0.992058516190872, + ), + "wages_salaries_returns": ( + "irs_soi.ty2023.table_1_4.all.wages_salaries_returns", + "1.00693407716956", + 0.9947092345961839, + ), +} + + +def test_pinned_feed_ht2_state_rows_take_their_national_control_level( + pinned_feed_national_state_surface, +) -> None: + """Each of the 14 registered Historic Table 2 families enters as 51 + shares of its TY2023 Table 1.x control (microcosm#1243): the control and + the states age by the same factor, so after aging the states are exactly + their HT2 share of the control, and no HT2-lineage national row of the + quantity is left beside it.""" + assert set(_PINNED_HT2_CONTROLS) == set(fiscal_targets._SOI_HT2_NATIONAL_CONTROLS) + _, surface, _ = pinned_feed_national_state_surface + by_name = {spec.name: spec for spec in surface.specs} + for measure_id, (control_id, factor, share) in _PINNED_HT2_CONTROLS.items(): + concept = fiscal_targets._SOI_HT2_NATIONAL_CONTROLS[measure_id].concept + rows = [ + spec + for spec in surface.specs + if ".historic_table_2.state_broad." in spec.name + and spec.metadata.get("source_measure_id") == measure_id + ] + assert len(rows) == 51, measure_id + control = by_name[control_id] + for spec in rows: + assert spec.metadata["uprating_index_source_record_id"] == control_id + assert spec.metadata["uprating_factor"] == factor + assert spec.metadata["uprating_to_period"] == "2023" + assert spec.metadata["aging_factor"] == control.metadata["aging_factor"] + assert spec.metadata["soi_source_concept"] == concept + assert spec.metadata["soi_control_concept"] == concept + total = sum(spec.value for spec in rows) + assert math.isclose(total / control.value, share, rel_tol=1e-12), measure_id + assert total < control.value + quantity = fiscal_targets._soi_national_quantity(rows[0]) + assert [ + spec.name + for spec in surface.specs + if spec.family == "irs_soi" + and not spec.metadata.get("state_fips") + and fiscal_targets._soi_national_quantity(spec) == quantity + and fiscal_targets._is_soi_ht2_lineage_record_set( + spec.metadata.get("ledger_layout_record_set_id", "") + ) + ] == [], measure_id + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_soi_ht2_national_controls.py b/packages/microcosm-build/tests/engine_free/us/test_us_soi_ht2_national_controls.py new file mode 100644 index 000000000..64f1cf48f --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/us/test_us_soi_ht2_national_controls.py @@ -0,0 +1,730 @@ +"""Historic Table 2 rows rebased onto their national controls (microcosm#1243). + +Fourteen Historic Table 2 (HT2) columns count the same Form 1040 line as a Pub +1304 Table 1.x column, so an HT2 state row is a share of the nation and the +newer Table 1.x row is the national level. ``_rebase_stale_soi_ht2_distributions`` +rebases the state rows onto that control and retires the HT2-lineage national +rows of each rebased quantity. The synthetic tests below pin the rule and its +invariants; the pinned-feed checks sit beside the release fixture in +``test_us_fiscal_targets.py``. +""" + +from __future__ import annotations + +import math + +import pytest + +from microcosm.build.us_runtime import compile_us_fiscal_target_registry, fiscal_targets +from test_support.microcosm_build.us_fiscal_targets import ( + _dynamic_ledger_fact, + packaged_reference_facts, +) + +REGISTER = fiscal_targets._SOI_HT2_NATIONAL_CONTROLS +STATES = (("ca", "06"), ("ny", "36"), ("tx", "48")) + + +def _soi_fact( + period: object, + *, + record_set_id: str, + measure_id: str, + value: float, + geography_level: str = "country", + geography_id: str = "0100000US", + record_id: str | None = None, +) -> dict[str, object]: + return _dynamic_ledger_fact( + source_record_id=record_id or f"{record_set_id}.all.{measure_id}", + source_name="irs_soi", + measure_id=measure_id, + value=value, + period_value=period, + geography_level=geography_level, + geography_id=geography_id, + dimensions={"income_range": "all", "filing_status": "all"}, + layout_record_set_id=record_set_id, + groupby_dimension="us:statutes/26/62#adjusted_gross_income", + groupby_value_id="all", + ) + + +def _ht2_us_id(measure_id: str, period: int = 2022) -> str: + return f"irs_soi.ty{period}.historic_table_2.us.all.{measure_id}" + + +def _ht2_state_id(measure_id: str, postal: str, period: int = 2022) -> str: + return f"irs_soi.ty{period}.historic_table_2.state_broad.{postal}.all.{measure_id}" + + +def _ht2_facts( + measure_id: str, + *, + period: int = 2022, + us: float = 1_000.0, + states: tuple[float, ...] = (250.0, 150.0), +) -> list[dict[str, object]]: + """An HT2 US row and up to three state rows of one measure.""" + return [ + _soi_fact( + period, + record_set_id=f"irs_soi.ty{period}.historic_table_2.us", + measure_id=measure_id, + value=us, + ), + *( + _soi_fact( + period, + record_set_id=( + f"irs_soi.ty{period}.historic_table_2.state_broad.{postal}" + ), + measure_id=measure_id, + value=value, + geography_level="state", + geography_id=f"0400000US{fips}", + ) + for (postal, fips), value in zip(STATES, states, strict=False) + ), + ] + + +def _control_record_set(family: str, period: object) -> str: + # Table 1.2's all-returns section is its own record set on the feed. + suffix = ".all_returns" if family == "table_1_2" else "" + return f"irs_soi.ty{period}.{family}{suffix}" + + +def _control_fact( + measure_id: str, + *, + period: object = 2023, + value: float = 800.0, + family: str | None = None, + record_id: str | None = None, +) -> dict[str, object]: + rule = REGISTER[measure_id] + return _soi_fact( + period, + record_set_id=_control_record_set(family or rule.control_family, period), + measure_id=rule.control_measure_id, + value=value, + record_id=record_id, + ) + + +def _state_2022_us_fact( + measure_id: str, value: float, *, data_year: int = 2022 +) -> dict[str, object]: + """HT2's US workbook (22in54us.xlsx) as the feed packages it: data for + ``data_year`` stamped the following year under ``state_`` + (PolicyEngine/chronicle#292).""" + stamp = data_year + 1 + record_set_id = f"irs_soi.ty{stamp}.state_{data_year}.us.{measure_id}" + return _soi_fact( + stamp, + record_set_id=record_set_id, + measure_id=measure_id, + value=value, + record_id=f"{record_set_id}.all_returns.{measure_id}", + ) + + +def _compile(facts, *, target_period: int = 2024): + return compile_us_fiscal_target_registry( + [*packaged_reference_facts(), *facts], + target_period=target_period, + allow_unaged_dollar_targets=True, + ) + + +def _by_record(registry) -> dict[str, object]: + return {spec.metadata["ledger_source_record_id"]: spec for spec in registry.specs} + + +def _controls(facts, *, target_period: int = 2024): + return fiscal_targets._soi_ht2_national_controls( + tuple(facts), target_period=target_period + ) + + +# -------------------------------------------------------------------------- +# The register + + +def test_each_registered_column_and_its_control_bind_the_same_model_quantity() -> None: + """A control can stand in for an HT2 column only if the compile maps both + to the same model variable and measure mode; the register is one-to-one + and leaves the interest and capital-gains families to their own passes.""" + + def binding(measure_id: str) -> tuple[str, str] | None: + amount = fiscal_targets.SOI_AMOUNT_MEASURE_VARIABLES.get(measure_id) + if amount is not None: + return ("sum", amount) + count = fiscal_targets.SOI_RETURN_MEASURE_VARIABLES.get(measure_id) + return None if count is None else ("indicator_sum", count) + + assert len(REGISTER) == 14 + for measure_id, rule in REGISTER.items(): + assert binding(measure_id) is not None, measure_id + assert binding(measure_id) == binding(rule.control_measure_id), measure_id + assert rule.concept + assert rule.control_family in {"table_1_1", "table_1_2", "table_1_4"} + assert len(fiscal_targets._SOI_HT2_MEASURE_BY_CONTROL) == len(REGISTER) + assert not set(REGISTER) & { + *fiscal_targets._SOI_TOTAL_UPRATED_AMOUNT_MEASURES, + *fiscal_targets._SOI_TOTAL_UPRATED_RETURN_MEASURES, + "net_capital_gains_amount", + "net_capital_gains_returns", + } + + +@pytest.mark.parametrize( + ("record_set_id", "lineage"), + [ + ("irs_soi.ty2022.historic_table_2.us", True), + ("irs_soi.ty2022.historic_table_2.state_broad.ca", True), + ("irs_soi.ty2023.state_2022.us.return_count", True), + ("irs_soi.ty2024.state_2023.us", True), + ("irs_soi.ty2023.state_agi", False), + ("irs_soi.ty2023.table_1_1", False), + ("irs_soi.ty2023.congressional_district_2022.all_returns", False), + ("irs_soi.ty2023.state_data", False), + ("", False), + ], +) +def test_ht2_lineage_record_sets(record_set_id: str, lineage: bool) -> None: + assert fiscal_targets._is_soi_ht2_lineage_record_set(record_set_id) is lineage + + +# -------------------------------------------------------------------------- +# The rebase + + +@pytest.mark.parametrize("measure_id", sorted(REGISTER)) +def test_ht2_state_rows_become_shares_of_their_national_control( + measure_id: str, +) -> None: + """CA 250 and NY 150 of an HT2 US 1,000 become 200 and 120 of an 800 + control; the HT2 US row retires and the control's own row is the only + national level of the quantity.""" + rule = REGISTER[measure_id] + control_id = f"{_control_record_set(rule.control_family, 2023)}.all.{ + rule.control_measure_id + }" + registry = _compile([*_ht2_facts(measure_id), _control_fact(measure_id)]) + specs = _by_record(registry) + + assert _ht2_us_id(measure_id) not in specs + assert specs[control_id].value == 800.0 + ca = specs[_ht2_state_id(measure_id, "ca")] + ny = specs[_ht2_state_id(measure_id, "ny")] + assert (ca.value, ny.value) == (200.0, 120.0) + assert ca.metadata["uprating_factor"] == "0.8" + assert ca.metadata["uprating_index"] == f"total_{measure_id}" + assert ca.metadata["uprating_from_period"] == "2022" + # Lands at the control's period; aging owns the rest (#488). + assert ca.metadata["uprating_to_period"] == "2023" + assert ca.metadata["uprating_index_source_record_id"] == control_id + assert ca.metadata["stale_distribution_rebased_to_active_total"] == "true" + assert ca.metadata["soi_source_concept"] == rule.concept + assert ca.metadata["soi_control_concept"] == rule.concept + national = [ + spec + for spec in registry.specs + if spec.family == "irs_soi" + and not spec.metadata.get("state_fips") + and fiscal_targets._soi_national_quantity(spec) + == fiscal_targets._soi_national_quantity(ca) + ] + assert [spec.metadata["ledger_source_record_id"] for spec in national] == [ + control_id + ] + + +def test_hts_us_workbook_row_retires_with_its_quantity_only() -> None: + """HT2's US workbook packaged as ``state_2022`` is the HT2 US total again. + It retires when its quantity's states are rebased, and stays when only + another quantity is (here AGI is rebased and the return count is not).""" + workbook = _state_2022_us_fact("return_count", 1_000.0) + workbook_id = workbook["lineage"]["source_record_id"] + rebased_count = _compile( + [ + *_ht2_facts("return_count"), + workbook, + _control_fact("return_count", value=1_010.0), + ] + ) + assert workbook_id not in _by_record(rebased_count) + + agi_only = _compile( + [ + *_ht2_facts("return_count"), + *_ht2_facts("adjusted_gross_income"), + workbook, + _control_fact("adjusted_gross_income", value=1_100.0), + ] + ) + specs = _by_record(agi_only) + assert specs[workbook_id].value == 1_000.0 + assert specs[_ht2_us_id("return_count")].value == 1_000.0 + assert _ht2_us_id("adjusted_gross_income") not in specs + + +@pytest.mark.parametrize( + "control", + [ + None, + # Older than the HT2 row: the rebase would run backwards. + {"period": 2021}, + # After the build period: not yet a level for a 2024 build. + {"period": 2025}, + ], + ids=["no-control", "older-control", "future-control"], +) +def test_ht2_rows_keep_their_level_without_an_eligible_control(control) -> None: + measure_id = "taxable_social_security_amount" + facts = [*_ht2_facts(measure_id)] + if control is not None: + facts.append(_control_fact(measure_id, **control)) + specs = _by_record(_compile(facts)) + + assert specs[_ht2_us_id(measure_id)].value == 1_000.0 + ca = specs[_ht2_state_id(measure_id, "ca")] + assert ca.value == 250.0 + assert "uprating_factor" not in ca.metadata + + +def test_the_ht2_us_row_retires_even_without_state_rows() -> None: + """The control owns the national quantity whether or not any HT2 state + row of it compiles.""" + measure_id = "unemployment_compensation_amount" + specs = _by_record( + _compile([*_ht2_facts(measure_id, states=()), _control_fact(measure_id)]) + ) + assert _ht2_us_id(measure_id) not in specs + + +def test_ht2_rows_keep_their_level_without_an_ht2_us_total() -> None: + measure_id = "taxable_pension_income_returns" + facts = [*_ht2_facts(measure_id)[1:], _control_fact(measure_id)] + specs = _by_record(_compile(facts)) + assert specs[_ht2_state_id(measure_id, "ca")].value == 250.0 + + +def test_a_control_at_the_hts_own_period_still_rebases() -> None: + """When HT2 and its control share a tax year, the factor is the + same-year HT2 / Pub 1304 difference alone.""" + measure_id = "return_count" + specs = _by_record( + _compile( + [ + *_ht2_facts(measure_id, period=2023), + _control_fact(measure_id, period=2023, value=1_004.0), + ] + ) + ) + ca = specs[_ht2_state_id(measure_id, "ca", period=2023)] + assert ca.value == pytest.approx(251.0, rel=1e-12) + assert ca.metadata["uprating_to_period"] == "2023" + assert _ht2_us_id(measure_id, period=2023) not in specs + + +def test_a_newer_hts_national_row_survives_an_older_control() -> None: + """Mixed vintages (microcosm#1243 review): CA has TY2023 HT2 data, NY only + TY2022, and the control is TY2022. NY becomes a share of the control; CA + and the TY2023 HT2 US row keep their levels, because the control does + not cover their year.""" + measure_id = "taxable_social_security_amount" + facts = [ + *_ht2_facts(measure_id, period=2023, us=1_100.0, states=(500.0,)), + _soi_fact( + 2022, + record_set_id="irs_soi.ty2022.historic_table_2.us", + measure_id=measure_id, + value=1_000.0, + ), + _soi_fact( + 2022, + record_set_id="irs_soi.ty2022.historic_table_2.state_broad.ny", + measure_id=measure_id, + value=400.0, + geography_level="state", + geography_id="0400000US36", + ), + _control_fact(measure_id, period=2022, value=800.0), + ] + specs = _by_record(_compile(facts)) + assert specs[_ht2_state_id(measure_id, "ny")].value == 320.0 + assert specs[_ht2_state_id(measure_id, "ca", period=2023)].value == 500.0 + assert specs[_ht2_us_id(measure_id, period=2023)].value == 1_100.0 + + +@pytest.mark.parametrize( + ("data_year", "retired"), [(2022, True), (2023, False)], ids=["ty2022", "ty2023"] +) +def test_the_us_workbook_retires_by_its_data_year(data_year, retired) -> None: + """The workbook's period is stamped a year late, so its data year comes + from ``state_``: a TY2022 control retires the TY2022 workbook + (stamped ty2023) but not a TY2023 one.""" + measure_id = "return_count" + workbook = _state_2022_us_fact(measure_id, 1_000.0, data_year=data_year) + specs = _by_record( + _compile( + [ + *_ht2_facts(measure_id), + workbook, + _control_fact(measure_id, period=2022, value=1_010.0), + ] + ) + ) + assert (workbook["lineage"]["source_record_id"] not in specs) is retired + + +def test_conflicting_hts_us_totals_refuse_the_compile() -> None: + """Two HT2 US records for one key would let feed order pick the rebase + denominator (microcosm#1243 review); one record seen twice is fine.""" + measure_id = "taxable_ira_distributions_amount" + us, *states = _ht2_facts(measure_id, states=(600.0, 400.0)) + rival = _soi_fact( + 2022, + record_set_id="irs_soi.ty2022.historic_table_2.us", + measure_id=measure_id, + value=500.0, + record_id=f"irs_soi.ty2022.historic_table_2.us.v2.{measure_id}", + ) + for ordered in ((us, rival), (rival, us)): + with pytest.raises( + fiscal_targets.AmbiguousSoiNationalControlError, + match="Historic Table 2 US total", + ): + fiscal_targets._soi_ht2_us_totals(ordered) + assert fiscal_targets._soi_ht2_us_totals((us, us)) == { + (measure_id, "All", "all_returns", "2022"): 1_000.0 + } + with pytest.raises(fiscal_targets.AmbiguousSoiNationalControlError): + _compile([us, rival, *states, _control_fact(measure_id)]) + + +def test_retirement_follows_each_rows_own_year() -> None: + """Property over compiles: with HT2 states at one vintage, the newest + HT2 US row at that vintage or later, an optional US workbook and any + control vintage, (a) the states rebase iff the control is eligible for + their year, (b) the newest US row retires iff the control is eligible + for its year, and (c) the workbook retires iff the states' quantity was + rebased and the control is at or after the workbook's data year.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + @settings(max_examples=40, deadline=None) + @given( + st.sampled_from(sorted(REGISTER)), + st.integers(min_value=2021, max_value=2023), + st.integers(min_value=0, max_value=1), + st.one_of(st.none(), st.integers(min_value=2020, max_value=2025)), + st.one_of(st.none(), st.integers(min_value=2021, max_value=2023)), + ) + def check(measure_id, state_year, us_lead, control_year, workbook_year) -> None: + us_year = state_year + us_lead + facts = _ht2_facts(measure_id, period=state_year, states=(250.0,)) + if us_lead: + facts += _ht2_facts(measure_id, period=us_year, us=1_050.0, states=()) + if control_year is not None: + facts.append(_control_fact(measure_id, period=control_year)) + workbook = None + if workbook_year is not None: + workbook = _state_2022_us_fact(measure_id, 1_000.0, data_year=workbook_year) + facts.append(workbook) + specs = _by_record(_compile(facts)) + + def eligible(year: int) -> bool: + return control_year is not None and year <= control_year <= 2024 + + state = specs[_ht2_state_id(measure_id, "ca", period=state_year)] + assert ("uprating_factor" in state.metadata) is eligible(state_year) + assert (_ht2_us_id(measure_id, period=us_year) not in specs) is eligible( + us_year + ) + if workbook is not None: + assert (workbook["lineage"]["source_record_id"] not in specs) is ( + eligible(state_year) and control_year >= workbook_year + ) + + check() + + +# -------------------------------------------------------------------------- +# The control chooser + + +def test_no_other_family_is_ever_a_control() -> None: + """HT2, HT2's US workbook, the congressional-district file and the + excluding-dependents and itemized tables all carry these measures, some + stamped later than Table 1.4; none is registered, so none anchors.""" + measure_id = "taxable_ira_distributions_amount" + decoys = [ + _soi_fact( + 2024, + record_set_id="irs_soi.ty2024.historic_table_2.us", + measure_id=measure_id, + value=1.0, + ), + _soi_fact( + 2023, + record_set_id="irs_soi.ty2023.congressional_district_2022.all_returns", + measure_id=measure_id, + value=2.0, + record_id=( + "irs_soi.ty2023.congressional_district_2022.all_returns.us." + f"{measure_id}" + ), + ), + _state_2022_us_fact(measure_id, 3.0), + _soi_fact( + 2023, + record_set_id="irs_soi.ty2023.table_4_3.all_returns_excluding_dependents", + measure_id=measure_id, + value=4.0, + ), + _soi_fact( + 2023, + record_set_id="irs_soi.ty2023.table_2_1.itemized_all_returns", + measure_id=measure_id, + value=5.0, + ), + # Right table, wrong column: Table 1.4's IRA count is another key. + _control_fact("taxable_ira_distributions_returns", period=2024, value=6.0), + ] + assert measure_id not in {key[0] for key in _controls(decoys)} + control = _control_fact(measure_id, period=2022, value=7.0) + (chosen,) = ( + value + for key, value in _controls([*decoys, control]).items() + if key[0] == measure_id + ) + assert (chosen.source_period, chosen.value) == ("2022", 7.0) + + +def test_the_control_refuses_a_tie_and_accepts_one_record_twice() -> None: + measure_id = "return_count" + control = _control_fact(measure_id, value=10.0) + rival = _control_fact( + measure_id, + value=10.0, + record_id="irs_soi.ty2023.table_1_1.v2.return_count", + ) + for ordered in ((control, rival), (rival, control)): + with pytest.raises( + fiscal_targets.AmbiguousSoiNationalControlError, + match="table_1_1.v2.return_count", + ): + _controls(ordered) + assert len(_controls((control, control))) == 1 + # A later control supersedes an older tie. + older = ( + _control_fact(measure_id, period=2022, value=1.0), + _control_fact( + measure_id, + period=2022, + value=2.0, + record_id="irs_soi.ty2022.table_1_1.b.return_count", + ), + ) + (chosen,) = _controls((*older, control)).values() + assert chosen.value == 10.0 + + +def test_the_control_is_order_free_latest_and_registered() -> None: + """Property: for any national facts across the registered and decoy + families and any build period, the control per HT2 measure is the latest + fact of its registered table and column not after the build period, + whatever the feed order; with none, there is no control.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + families = ( + "table_1_1", + "table_1_2", + "table_1_4", + "historic_table_2", + "congressional_district_2022", + "state_2022", + "table_4_3", + ) + columns = sorted( + {*REGISTER, *(rule.control_measure_id for rule in REGISTER.values())} + ) + candidate = st.tuples( + st.sampled_from(families), + st.integers(min_value=2018, max_value=2027), + st.sampled_from(columns), + st.floats(min_value=1.0, max_value=1e13, allow_nan=False), + ) + + def record_set(family: str, period: int) -> str: + if family == "table_1_2": + return f"irs_soi.ty{period}.table_1_2.all_returns" + if family == "historic_table_2": + return f"irs_soi.ty{period}.historic_table_2.us" + if family == "congressional_district_2022": + return f"irs_soi.ty{period}.congressional_district_2022.all_returns" + if family == "state_2022": + return f"irs_soi.ty{period}.state_2022.us" + return f"irs_soi.ty{period}.{family}" + + @settings(max_examples=300, deadline=None) + @given( + st.lists(candidate, max_size=20, unique_by=lambda item: item[:3]), + st.integers(min_value=2019, max_value=2026), + st.randoms(use_true_random=False), + ) + def check(candidates, target_period, rng) -> None: + facts = [ + _soi_fact( + period, + record_set_id=record_set(family, period), + measure_id=column, + value=value, + ) + for family, period, column, value in candidates + ] + shuffled = list(facts) + rng.shuffle(shuffled) + controls = _controls(facts, target_period=target_period) + assert controls == _controls(shuffled, target_period=target_period) + chosen = {key[0]: control for key, control in controls.items()} + for measure_id, rule in REGISTER.items(): + eligible = [ + (period, value) + for family, period, column, value in candidates + if family == rule.control_family + and column == rule.control_measure_id + and period <= target_period + ] + if not eligible: + assert measure_id not in chosen + continue + period, value = max(eligible) + control = chosen[measure_id] + assert (control.source_period, control.value) == (str(period), value) + assert control.source_record_id == ( + f"{record_set(rule.control_family, period)}.all." + f"{rule.control_measure_id}" + ) + assert set(chosen) <= set(REGISTER) + + check() + + +# -------------------------------------------------------------------------- +# Invariants over compiles + + +def test_rebased_states_conserve_shares_and_one_national_level_remains() -> None: + """Property over compiles, for every registered column, any HT2 vintage, + any control vintage (or none) and any feed order: + + * **Shares conserved.** With an eligible control (not after the build + period, not before the HT2 row), each state row is its HT2 value times + control / HT2 US, so the states keep their HT2 shares and sum to the + control times the states' share of the HT2 nation. + * **States never exceed the nation.** When the HT2 states sum to at most + the HT2 US total, the rebased states sum to at most the control. + * **One national level.** The control's row is the only national + all-AGI row of the quantity: the HT2 US row and HT2's US workbook row + retire. + * **No eligible control, no change.** Otherwise every HT2 row keeps its + level and the HT2 US row stays. + * **Order-free.** Shuffling the facts compiles the same registry. + """ + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + @settings(max_examples=40, deadline=None) + @given( + st.sampled_from(sorted(REGISTER)), + st.integers(min_value=2021, max_value=2024), + st.one_of(st.none(), st.integers(min_value=2020, max_value=2026)), + st.lists( + st.integers(min_value=1, max_value=10_000_000), min_size=1, max_size=3 + ), + st.integers(min_value=0, max_value=5_000_000), + st.integers(min_value=1, max_value=50_000_000), + st.booleans(), + st.randoms(use_true_random=False), + ) + def check( + measure_id, + ht2_period, + control_period, + state_values, + other_areas, + control_value, + with_workbook, + rng, + ) -> None: + national = float(sum(state_values) + other_areas) + facts = _ht2_facts( + measure_id, + period=ht2_period, + us=national, + states=tuple(float(value) for value in state_values), + ) + if with_workbook: + # HT2's US workbook for the same data year, stamped a year late. + facts.append( + _state_2022_us_fact(measure_id, national, data_year=ht2_period) + ) + if control_period is not None: + facts.append( + _control_fact( + measure_id, period=control_period, value=float(control_value) + ) + ) + shuffled = list(facts) + rng.shuffle(shuffled) + registry = _compile(facts) + assert sorted( + (spec.name, spec.value, tuple(sorted(spec.metadata.items()))) + for spec in registry.specs + ) == sorted( + (spec.name, spec.value, tuple(sorted(spec.metadata.items()))) + for spec in _compile(shuffled).specs + ) + + specs = _by_record(registry) + states = [ + specs[_ht2_state_id(measure_id, postal, period=ht2_period)] + for (postal, _), _ in zip(STATES, state_values, strict=False) + ] + quantity = fiscal_targets._soi_national_quantity(states[0]) + national_rows = [ + spec + for spec in registry.specs + if spec.family == "irs_soi" + and not spec.metadata.get("state_fips") + and fiscal_targets._soi_national_quantity(spec) == quantity + ] + eligible = control_period is not None and ht2_period <= control_period <= 2024 + if eligible: + factor = control_value / national + for spec, value in zip(states, state_values, strict=False): + assert math.isclose(spec.value, value * factor, rel_tol=1e-12) + total = sum(spec.value for spec in states) + assert math.isclose( + total, + control_value * sum(state_values) / national, + rel_tol=1e-12, + ) + assert total <= control_value * (1 + 1e-12) + assert [spec.value for spec in national_rows] == [float(control_value)] + else: + assert [spec.value for spec in states] == [ + float(value) for value in state_values + ] + assert specs[_ht2_us_id(measure_id, period=ht2_period)].value == national + + check() diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_soi_state_agi_bands.py b/packages/microcosm-build/tests/engine_free/us/test_us_soi_state_agi_bands.py index 6edeb8e2e..2058e7151 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_soi_state_agi_bands.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_soi_state_agi_bands.py @@ -605,6 +605,89 @@ def test_rebased_bands_age_with_the_state_total() -> None: ) +def _national_fact(record_set_id: str, measure_id: str, period: int, value: float): + return _fact( + source_record_id=f"{record_set_id}.all.{measure_id}", + record_set_id=record_set_id, + measure_id=measure_id, + value=value, + period=period, + geography_level="country", + geography_id="0100000US", + dimensions={"filing_status": "all", "income_range": "all"}, + groupby_dimension=AGI, + groupby_value_id="all", + ) + + +def test_bands_follow_a_state_total_rebased_onto_its_national_control() -> None: + """When the HT2 state total is itself a share of a newer Table 1.1 + control (microcosm#1243), it sits at the control's period, and the bands + land there with it: every band keeps its exact share of the aged total, + and AGI bands take the total's 2023->2024 link, not 2022->2024.""" + us_returns, us_agi = 160_000_000.0, 14_800_000_000_000.0 + control_returns, control_agi = 161_000_000.0, 15_300_000_000_000.0 + registry = _compile( + [ + *_partition("CO", 2023, CO_2023), + *_totals("CO", 2022, CO_2022_TOTAL_RETURNS, CO_2022_TOTAL_AGI), + _national_fact( + "irs_soi.ty2022.historic_table_2.us", "return_count", 2022, us_returns + ), + _national_fact( + "irs_soi.ty2022.historic_table_2.us", + "adjusted_gross_income", + 2022, + us_agi, + ), + _national_fact( + "irs_soi.ty2023.table_1_1", "return_count", 2023, control_returns + ), + _national_fact( + "irs_soi.ty2023.table_1_1", + "adjusted_gross_income", + 2023, + control_agi, + ), + ] + ) + cbo = ( + _cbo_agi_fact(2022, 14_000_000_000_000), + _cbo_agi_fact(2023, 15_000_000_000_000), + _cbo_agi_fact(2024, 16_000_000_000_000), + ) + aged = { + spec.name: spec + for spec in age_us_dollar_targets(registry, cbo, target_period=2024).specs + } + total_agi = aged[ + "irs_soi.ty2022.historic_table_2.state_broad.co.all.adjusted_gross_income" + ] + total_returns = aged[ + "irs_soi.ty2022.historic_table_2.state_broad.co.all.return_count" + ] + link = 16 / 15 + + assert total_agi.metadata["uprating_to_period"] == "2023" + assert total_agi.value == pytest.approx( + CO_2022_TOTAL_AGI * control_agi / us_agi * link, rel=1e-12 + ) + assert total_returns.value == pytest.approx( + CO_2022_TOTAL_RETURNS * control_returns / us_returns, rel=1e-12 + ) + for band in BINDING_BANDS: + agi = aged[_co_name(band, "adjusted_gross_income")] + returns = aged[_co_name(band, "return_count")] + assert agi.metadata["uprating_to_period"] == "2023" + assert float(agi.metadata["aging_factor"]) == pytest.approx(link, rel=1e-12) + assert agi.value / total_agi.value == pytest.approx( + float(agi.metadata["state_agi_band_share"]), rel=1e-12 + ) + assert returns.value / total_returns.value == pytest.approx( + float(returns.metadata["state_agi_band_share"]), rel=1e-12 + ) + + def test_release_coverage_requires_both_top_tail_rows_in_each_state() -> None: """The release requires a $1M+ AGI row and a $1M+ count row for every one of the 51 states: dropping one state's row fails exactly that state's @@ -1173,17 +1256,29 @@ def test_pinned_feed_binds_every_state_band_from_ty2023(pinned_feed_registry) -> float(spec.metadata["state_agi_band_share"]), rel=1e-12 ) + # Colorado's TY2022 totals are themselves shares of Table 1.1's TY2023 + # totals (microcosm#1243), so the $1M+ rows are the published TY2023 + # share of that rebased total, aged from 2023. + co_fips = POSTAL_TO_FIPS["CO"] top = bands[_co_name("1m_plus", "adjusted_gross_income")] share = 47_266_883_000 / sum(agi for _, agi in CO_2023.values()) assert float(top.metadata["state_agi_band_share"]) == pytest.approx( share, rel=1e-12 ) + total_agi = totals[(co_fips, "adjusted_gross_income")] + assert top.metadata["uprating_to_period"] == "2023" + assert top.metadata["aging_factor"] == total_agi.metadata["aging_factor"] assert top.value == pytest.approx( - CO_2022_TOTAL_AGI * share * float(top.metadata["aging_factor"]), rel=1e-12 + CO_2022_TOTAL_AGI + * float(total_agi.metadata["uprating_factor"]) + * share + * float(top.metadata["aging_factor"]), + rel=1e-12, ) top_returns = bands[_co_name("1m_plus", "return_count")] assert top_returns.value == pytest.approx( CO_2022_TOTAL_RETURNS + * float(totals[(co_fips, "return_count")].metadata["uprating_factor"]) * 15_610 / sum(returns for returns, _ in CO_2023.values()), rel=1e-12, @@ -1211,9 +1306,9 @@ def test_pinned_feed_state_top_tail_agrees_with_table_1_1(pinned_feed_registry) the states, the aged $500k-$1M and $1M+ bands come to 97%-100% of SOI Table 1.1's TY2023 classes aged to 2024 the same way. Table 1.1 also counts returns filed from other areas and Puerto Rico, which no state - holds (1.2%-1.7% of these classes in the raw HT2 files), and the states' - TY2022 totals grew like the nation's to TY2023 (AGI x1.0305 in HT2 and in - Table 1.1; returns +0.19%), so the expected ratio is 0.98-0.99.""" + holds (1.2%-1.7% of these classes in the raw HT2 files), and each state + total is its HT2 share of Table 1.1's TY2023 total (microcosm#1243), so + the expected ratio is 0.98-0.99.""" facts, registry = pinned_feed_registry bands = _band_specs(registry) table_1_1: dict[tuple[str, float, float], float] = {}