diff --git a/changelog.d/1228-soi-net-income-less-loss.fixed.md b/changelog.d/1228-soi-net-income-less-loss.fixed.md new file mode 100644 index 0000000000..21dccfcc20 --- /dev/null +++ b/changelog.d/1228-soi-net-income-less-loss.fixed.md @@ -0,0 +1 @@ +Bind the Historic Table 2 and congressional-district business (N/A00900), partnership/S-corp (N/A26270) and rental/royalty (N/A25870) targets to signed "net income (less loss)" concepts that sum the signed tax-unit total and count every unit with a nonzero total (#1228). Business and partnership amounts had been positive parts, all three counts had counted positive totals only, business had left out SSTB Schedule C income, and rental had included Form 4835 farm rent. diff --git a/docs/us-soi-net-income-less-loss-concepts.md b/docs/us-soi-net-income-less-loss-concepts.md new file mode 100644 index 0000000000..018cddcce1 --- /dev/null +++ b/docs/us-soi-net-income-less-loss-concepts.md @@ -0,0 +1,231 @@ +# SOI business, partnership and rental "net income (less loss)" concepts + +Historic Table 2 (HT2) and the congressional-district file publish business, +partnership/S-corp and rental/royalty income as one signed column each. Until +microcosm#1228 the target compiler bound those columns to one-sided model +concepts. This page records what each IRS column counts, what the model +measures, and how `fiscal_targets.py` binds them now. + +## What each column counts + +| Measure ids | Column (HT2, district file) | Definition (2022 and 2023 doc guides) | Form line | +|---|---|---|---| +| `schedule_c_income_{returns,amount}` | N/A00900 | "business or professional net income (less loss)" | Schedule 1 line 3 | +| `partnership_scorp_income_{returns,amount}` | N/A26270 | "partnership/S-corp net income (less loss)" | Schedule E line 32 | +| `rental_royalty_income_{returns,amount}` | N/A25870 | "total rental and royalty net income (less loss)" | Schedule E line 26 | + +The amount nets losses against income. The count is every return with an +entry on the line, net income or net loss. Table 1.4 splits each line into a +"net income" column and a "net loss" column, and the HT2 columns are their +union. Publication 1304 prints this convention for its own "(less loss)" +lines: in the TY2021 report's summary table, TY2020 business "net income +(less loss)" returns are 27,727 thousand = 19,977 net income + 7,750 net +loss, and partnership and S corporation returns are 9,002 = 5,902 + 3,100. + +**Business income.** Table 1.4 "Business or profession" net income and net +loss: + +| Tax year | HT2 N00900 / T1.4 (NI + NL returns) | HT2 A00900 / T1.4 (NI − NL) | T1.4 NI returns / HT2 N00900 | T1.4 NI amount / HT2 A00900 | +|---|---:|---:|---:|---:| +| 2020 | 1.0059 | 1.0225 | 0.716 | 1.266 | +| 2021 | 1.0064 | 1.0195 | 0.732 | 1.233 | +| 2022 | 0.9999 | 1.0272 | 0.724 | 1.289 | +| 2023 | 1.0070 | 1.0334 | 0.698 | 1.362 | + +**Partnership and S corporation income.** TY2020 is the last year Table 1.4 +published combined partnership-and-S-corp columns. That year HT2 N26270 +(9,011,930) equals the combined NI + NL returns (9,001,513) within 0.12%, +and only 65.5% of them are net-income returns. In TY2021–TY2023 Table 1.4 +splits partnership from S corporation, so no combined net-income count +exists. HT2 A26270 equals the net within 0.8% in TY2020 and the summed +partnership and S-corp nets within 0.2–0.35% in TY2021–TY2023. + +**Rental and royalty income.** N25870/A25870 first appear in TY2022. Both +doc guides tie them to Schedule E line 26 alone. Publication 1304 defines +Table 1.4's "Total rental and royalty net income or loss" as "line 26 + +lines 39 and 40, Schedule E", so the Table 1.4 total also carries REMIC +income (line 39) and Form 4835 farm rental (line 40). The amounts agree: +in TY2022 HT2 A25870 is 0.919 of the Table 1.4 total net, and 0.983 once +farm rental is removed; in TY2023 the ratios are 0.919 and 0.992. HT2 +N25870 is 0.97 and 0.96 of the Table 1.4 NI + NL returns. The gap is +consistent with returns whose only entries are on lines 39 or 40, or with +the population-versus-sample difference between the files; it is not +resolved here. + +The workbooks are the feed-pinned `20in14ar.xls` … `23in14ar.xls` and +`20in55cmcsv.csv` … `23in55cmcsv.csv`. The script and its output are in +`~/PolicyEngine/_reviews/soi-net-income-less-loss-20261012/` +(`scripts/identities.py`, `out/identities.out`). + +## What the model measured before #1228 + +`tools/build_us_fiscal_refresh_release.py` `_signed_component` / +`_soi_component_row` sum each SOI concept per tax unit: + +- `business_net_profits`, mapped to `self_employment_income`, took the + **positive part** and counted units with a **positive** total. +- `partnership_and_s_corp_income`, mapped to + `tax_unit_partnership_s_corp_income`, did the same. +- `rent_and_royalty_net_income`, mapped to `rental_income + + farm_rent_income`, kept the sign but counted a **positive** total. + +Two further gaps sat on the same rows: + +- **SSTB businesses.** The frame moves a specified-service business's whole + Schedule C amount from `self_employment_income_before_lsr` to + `sstb_self_employment_income_before_lsr` (`us_runtime/qbi_inputs.py`). + Schedule 1 line 3 is every Schedule C, which is PE-US + `total_self_employment_income`. The business target never saw SSTB income. +- **Farm rental.** The rental concept added Form 4835 farm rental, which + line 26 leaves out. + +## How the targets bind now + +`SOI_NET_INCOME_LESS_LOSS_VARIABLES` registers three signed concepts: + +| Concept | PE-US column (tax-unit total) | Amount | Count | `soi_source_concept` | +|---|---|---|---|---| +| `business_net_income_less_loss` | `total_self_employment_income` | signed sum | total ≠ 0 | `form_1040_schedule_1_line_3_net_income_less_loss` | +| `partnership_and_s_corp_net_income_less_loss` | `tax_unit_partnership_s_corp_income` | signed sum | total ≠ 0 | `schedule_e_line_32_net_income_less_loss` | +| `rent_and_royalty_net_income_less_loss` | `rental_income` | signed sum | total ≠ 0 | `schedule_e_line_26_net_income_less_loss` | + +These are the frame's closest columns, with two caveats from the donor +lineage: + +- **Partnership and S corporation income is narrower than line 32 in this + frame.** The processed PUF donor builds `partnership_s_corp_income` from + the nonpassive fields only. policyengine-us-data `datasets/puf/puf.py` + computes `(E26190 − E26180) + (E25980 − E25960)`, commented "active" + S-corp and partnership income, and microcosm's fallback in + `us_runtime/puf_support.py` reads the same fields. It does not use E26270, + the line-32 total that includes passive income. The Route A frame starts + at $315B net ($411B positive part) against a $1,158B target, so this + binding asks calibration to roughly triple the carriers' mass; the release + binding asked the same of the positive part. The column is still the right + one to bind: PE-US documents `passive_partnership_s_corp_income` as a + subset of `partnership_s_corp_income`, so the model column is meant to be + line 32. The donor gap is a frame defect. +- **Rental's donor is `E25850 − E25860`** (Schedule E rent and royalty net + income less net loss). Line 26 is after the passive-loss limitation; that + these PUF fields are too is not verified here. The re-solve lands the + line-26 net at $98.4B against $95.9B, so the levels are compatible. + +- The six measure ids bind these concepts. +- Only HT2 and the district file are reviewed to publish them + (`_SOI_NET_INCOME_LESS_LOSS_FAMILIES`). The same id from any other family + raises `UnregisteredSoiNetIncomeLessLossFamilyError`, because Table 1.4 + publishes these lines as one-sided columns. +- The one-sided concepts (`business_net_profits` / `business_net_losses`, + `partnership_and_s_corp_income` / `_losses`, `rent_and_royalty_net_income` / + `_losses`) are reserved for Table 1.4's split columns. No pinned fact uses + them. Two of them changed so a future leg binds correctly: + - the business legs now read `total_self_employment_income`; + - `rent_and_royalty_net_income` is now a positive part, like every other + income leg. It keeps farm rental, as Table 1.4's total does. + + PolicyEngine/chronicle#346 asks Chronicle for those legs. + +On the pinned feed (`consumer_facts_us_51aa40f`, compile at 2024 with aging, +the packaged district crosswalk and Medicaid substitution) the change: + +- rebinds 3,294 specs. 312 are the Historic Table 2 rows on + `national_state` (6 ids × 51 states + the US row). The other 2,982 sit on + `full` only: 2,928 district-file rows (each id's US, 51 state and 436 + district rows) and 54 Historic Table 2 rows of the nine single-district + states, which `full` classes as district rows; +- changes no target value, and no spec enters or leaves either surface; +- moves `national_state` `e1f9906bf0e0` → `38c1db8880da` and `full` + `f5bd921d6b21` → `9f6c65fe8734`. + +## Invariants + +- **Leg conservation.** For every signed tax-unit vector and nonnegative + weights: + - a less-loss count equals the net-income leg's count plus the net-loss + leg's count; + - a less-loss amount equals the net-income leg's amount minus the net-loss + leg's amount. + + For business this is the published identity in every year (N00900 = + Table 1.4 NI + NL returns, A00900 = NI − NL), and for partnership and S + corporation in TY2020's combined columns. Table 1.4's rental legs report a + different line (Schedule E lines 26 + 39 + 40, which is why the model's + rental legs add farm rent), and its TY2021+ partnership and S-corp legs are + per entity type. For those two the property checks the model's own + decomposition of one signed column, not a published identity. Hypothesis + property: `test_net_income_less_loss_rows_are_the_sum_of_their_table_1_4_legs`. +- **End to end.** `test_net_income_less_loss_rows_materialize_signed_totals_and_nonzero_counts` + materializes the six concepts through the SOI slice on a fake engine: a + unit whose members net to a loss, an SSTB-only unit and a farm-rent-only + unit, hand-computed. +- **Bounds and sign symmetry.** The less-loss count is at least the + net-income count, and the amount at most the net-income amount. Negating + every value keeps the count and negates the amount. Same property. +- **Family-only binding.** For any id, tax year and record-set shape, a fact + binds its line's signed concept exactly when its family is HT2 or the + district file, and refuses otherwise. Property + `test_net_income_less_loss_binding_depends_only_on_the_family`. +- **Pinned feed.** `test_pinned_feed_net_income_less_loss_rows_bind_signed_concepts` + checks: + - every business, partnership and rental row binds its line's concept; + - only those rows bind a signed concept; + - no row binds a one-sided leg; + - `national_state` carries 312 of them; + - each family's 51 state rows sum to at most the US row. This is a data + pin, not a law: the US row includes territories and other areas, which + could be net negative for a signed amount. + +## The method choice + +These rows are on `national_state`, the surface the certified Route A +release calibrates, so changing what they bind is a ruling. Three options +were measured on the release's own calibration checkpoint (see "Effect on +the Route A calibration" below), with a fourth arm that keeps option 1 but +drops the Schedule C counts: + +1. **Concept-exact (this PR).** Bind each column to the signed tax-unit + total of its line. This needs no new facts and no share assumption. +2. **Rebase onto a one-sided Table 1.4 control**, as #1036 does for capital + gains: HT2 state rows as shares of a national net-income leg. This needs + PolicyEngine/chronicle#346. There is no combined partnership/S-corp + net-income column after TY2020, and state shares of a net amount stand in + for state shares of profits. +3. **Drop** the rows whose concept differs. That is all but the rental + amount: 260 on `national_state`. + +**The frame limits option 1.** The `puf_tax_detail` stage lists +`self_employment_income_before_lsr` under `nonnegative_outputs` +(`us/spec/sources.yaml`), although the donor's `self_employment_income` is +the signed E00900. As a result: + +- the Route A frame carries 0.39M weighted Schedule C loss units before + calibration (0.65M after), against 8.39M loss returns in Table 1.4 + TY2022; +- a nonzero Schedule C count can still be met only by adding profit + carriers until the frame carries losses; +- partnership and rental (line 26) do carry losses: 1.68M and 3.66M units before + calibration. + +## Effect on the Route A calibration + +`experiments/1228-net-income-less-loss/` re-solves the certified Route A release's `national_state` calibration from its own checkpoint (5,694 targets, 352,932 households, the release's solver settings), once per binding. Only the 312 rows differ between arms. The replay of the release's own binding reproduces it: loss 0.028420 against the recorded 0.028426, weight correlation 0.9999. + +| 2024 national total | IRS (Table 1.4 TY2022 aged ×1.12; HT2 for nets and counts) | Release binding | Concept-exact (this PR) | Rebase onto Table 1.4 net income | Drop the 260 rows | +|---|---:|---:|---:|---:|---:| +| Schedule C net, incl. SSTB | $460B (HT2 $472B) | $821B | $507B | $613B | $1,321B | +| Schedule C profits | $609B | $837B | $532B | $628B | $1,333B | +| Schedule C profit units | 21.97M | 30.64M | 28.08M | 21.89M | 19.03M | +| Schedule C units with an entry | 30.35M | 31.29M | 29.37M | 22.49M | 19.55M | +| Partnership/S-corp net | $1,158B | $1,021B | $1,120B | $1,119B | $811B | +| Partnership/S-corp profit units | ≤ 7.34M | 9.48M | 7.26M | 7.27M | 5.13M | +| Rental and royalty net (line 26) | $95.9B | $248.6B | $98.4B | $118.2B | $95.1B | +| Rental and royalty units with an entry (line 26) | 9.40M | 14.09M | 9.32M | 10.67M | 14.25M | +| The 5,382 other targets within 10% | | 93.96% | 93.96% | 94.13% | 94.43% | +| Kish ESS | | 77,794 | 77,306 | 81,036 | 84,834 | + +- The concept-exact binding is right on what the IRS columns measure, and costs nothing elsewhere. +- Its Schedule C profit units (1.28×) and profits (0.87×) are off because the frame has few Schedule C losses. +- The rebase gets profits right and the net 1.3× high, for the same reason. It also needs PolicyEngine/chronicle#346 and has no partnership counterpart. +- Dropping the rows lets Schedule C reach $1.3T. + +The experiment's README has the arm definitions, the checks, per-measure state fit and the full totals. diff --git a/experiments/1228-net-income-less-loss/README.md b/experiments/1228-net-income-less-loss/README.md new file mode 100644 index 0000000000..57280cc968 --- /dev/null +++ b/experiments/1228-net-income-less-loss/README.md @@ -0,0 +1,123 @@ +# #1228: re-solving Route A with the business, partnership and rental rows bound five ways + +This experiment re-solves the certified Route A release's `national_state` calibration from its own checkpoint, once per way of binding the 312 Historic Table 2 rows of microcosm#1228: + +- the release: `populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z`; +- the surface: 5,694 targets and 352,932 households. + +The rows are business (N/A00900), partnership/S-corp (N/A26270) and rental/royalty (N/A25870). Everything else stays fixed: + +- the other 5,382 targets; +- the initial weights; +- the solver settings. + +So the arms differ only in what those rows ask of the weights. It is research evidence: nothing was published, and no default changed. + +## Arms + +| Arm | The 312 rows | Loss weights | +|---|---|---| +| `release` | As Route A bound them. Business and partnership take the positive part and count units with a positive total; business reads `self_employment_income`, without SSTB. Rental is signed, includes farm rent, and counts a positive total. | Release's, reproduced exactly | +| `concept_exact` | #1228 (this PR). Signed sums, nonzero counts, Schedule C including SSTB, rental without farm rent. | Identical to the release's: on `national_state` every row is its own concept group and no value changes | +| `concept_exact_no_business_count` | As `concept_exact`, without the 52 Schedule C return counts. The frame has almost no Schedule C loss carriers. | Recomputed (`target_loss_weights`) | +| `drop` | The 260 rows whose release concept differs from the IRS column are dropped. The rental amount keeps its release concept. | Recomputed | +| `rebase_positive_part` | The capital-gains approach of #1036. Business and rental keep one-sided concepts (Schedule C including SSTB; rental plus farm rent), and each HT2 state row becomes its share of a national Table 1.4 TY2022 net-income control. Partnership rows stay concept-exact, because Table 1.4 has no combined net-income column after TY2020. | Recomputed (208 values change) | + +The Table 1.4 controls are not in the pinned feed (PolicyEngine/chronicle#346). They were read from the feed-pinned `22in14ar.xls`: + +| Control | Returns | Amount | +|---|---:|---:| +| "Business or profession: Net income" | 21,969,832 | $543,564,980 thousand | +| "Total rental and royalty: Net income" | 6,380,131 | $148,879,455 thousand | + +Amounts are aged by the rows' own factor, 1.12034597479376; counts are not aged, as in the release. + +## Method + +- **`prepare.py`** streams the checkpoint's 5,694 target columns into a CSR matrix. It also builds the 312 replacement rows, the arms' loss weights and household-level concept totals. Checks (`MANIFEST.json`): + - `A @ initial` and `A @ published` reproduce every target's recorded initial and published estimate exactly (max relative gap 0). + - Rebuilding the 312 rows from person inputs on the release's concepts matches the checkpoint's materialized columns to 5.4e-8, its float32 storage precision. + - The concept-exact rows come from the same code with the #1228 rules. + - `target_loss_weights` reproduces the release's recorded loss weights exactly. +- **`prepare_rebase.py`** builds the `rebase_positive_part` rows and values from the same concept totals. +- **`resolve.py`** calls `microcosm.calibrate.calibrate` with the release's options from `calibration_diagnostics.json`: Adam, 1,500 epochs, learning rate 0.02, `mass="conserve"`, `max_weight_ratio=5`, `target_loss_cap=1`, no L1/L2, seed 0. The release arm's epoch-1 loss, 0.202372, equals the release's recorded initial loss, 0.2023716. +- **`modal_resolve.py`** runs the five arms on Modal: 16 CPUs each, torch 2.12, the kernel packages from this branch's HEAD. +- **`analyze.py`** scores each arm and writes `results/summary.{json,md}`. + +Inputs live in `/Users/maxghenis/PolicyEngine/_build_artifacts/1228-net-income-resolve/inputs` and runs in `.../runs`. The scripts hard-code the Route A run directory (`_recovered/scratch-backup/893/overnight-20260923/route-a/run-4b57d15a287c`), as research harnesses here do. The arms' recorded metrics predate the inputs-digest and `resolve.py`-digest fields that `modal_resolve.py` now checks before reusing a finished arm. + +## Results + +`results/summary.md` and `results/summary.json`, written by `analyze.py`. Each arm ran on Modal with 16 CPUs in 20–26 minutes, at kernel HEAD 5146a96b0. + +**The replay reproduces the release.** The `release` arm ends at loss 0.028420; the release recorded 0.028426. Its weights correlate with the published ones at 0.9999, the per-household ratio stays within 0.79–1.25, and every model total below matches the published weights' to within 0.1%. The other arms' differences are therefore the binding's, not the solver's. + +### Fit + +| Arm | Own loss | Own rows within 10% | The 312 rows within 10% (own binding) | The 5,382 shared rows within 10% | Shared rows, mean capped error | Kish ESS | +|---|---:|---:|---:|---:|---:|---:| +| release | 0.02842 | 93.41% | 84.0% | 93.96% | 0.03064 | 77,794 | +| concept_exact | 0.02641 | 93.89% | 92.6% | 93.96% | 0.03074 | 77,306 | +| concept_exact_no_business_count | 0.02583 | 94.04% | 93.1% (260 rows) | 94.09% | 0.03042 | 82,769 | +| drop | 0.02414 | 94.33% | 84.6% (52 rows) | 94.43% | 0.02848 | 84,834 | +| rebase_positive_part | 0.02552 | 94.06% | 92.9% | 94.13% | 0.03011 | 81,036 | + +No arm worsens any other target family's mean capped error by more than 0.0025 against the release arm. Under `concept_exact`, AGI's error falls from 0.0138 to 0.0109; the largest rise is itemized deductions', from 0.0439 to 0.0465. Two arms improve one family more: dropping the rows cuts real estate taxes' error by 0.018, and the rebase cuts capital gains' by 0.014. + +### Model totals: what each binding does to the income it targets + +These are 2024 totals under each arm's weights, summed over tax units from the frame's inputs. "IRS" is Table 1.4 TY2022 aged ×1.12035, the rows' own factor; counts are not aged. + +**Schedule C (`self_employment_income + sstb_self_employment_income`)** + +| | Positive part | Profit units | Negative part | Loss units | Net | Nonzero units | +|---|---:|---:|---:|---:|---:|---:| +| IRS | $609.0B | 21.97M | −$149.2B | 8.39M | $459.8B (HT2 net: $472.3B) | 30.36M (HT2: 30.35M) | +| release | $836.9B | 30.64M | −$15.9B | 0.65M | $821.0B | 31.29M | +| concept_exact | $532.2B | 28.08M | −$25.3B | 1.29M | $506.9B | 29.37M | +| concept_exact_no_business_count | $513.0B | 13.65M | −$19.5B | 0.75M | $493.5B | 14.40M | +| drop | $1,332.9B | 19.03M | −$12.1B | 0.52M | $1,320.7B | 19.55M | +| rebase_positive_part | $627.7B | 21.89M | −$14.4B | 0.60M | $613.3B | 22.49M | + +- **SSTB Schedule C**, which the release's business concept leaves out, totals $347.9B under the release, $95.6B under `concept_exact` and $103.3B under the rebase. +- **The frame carries few Schedule C losses** in any arm: 0.5–1.3M units against the IRS's 8.39M. This is the `puf_tax_detail` stage's `nonnegative_outputs`. So no binding gets both the net and the profits right: + - `concept_exact` hits the net and the nonzero count, with too many, smaller profit units (1.28×) and profits at 0.87×; + - the rebase hits profits and profit units, with the net at 1.30–1.33×. +- **Dropping the rows lets Schedule C float to $1.3T**, so other targets do not pin it. + +**Partnership and S corporation (`tax_unit_partnership_s_corp_income`)** + +| | Positive part | Profit units | Negative part | Loss units | Net | Nonzero units | +|---|---:|---:|---:|---:|---:|---:| +| IRS | ≤ $1,456.2B (legs' sum) | ≤ 7.34M (legs' sum) | | | $1,155.3B (HT2: $1,157.6B) | HT2: 9.55M | +| release | $1,129.0B | 9.48M | −$108.0B | 2.01M | $1,021.0B | 11.49M | +| concept_exact | $1,248.7B | 7.26M | −$128.9B | 2.26M | $1,119.8B | 9.52M | +| drop | $986.9B | 5.13M | −$175.8B | 2.32M | $811.1B | 7.45M | + +The release's 9.48M positive units exceed the 7.34M upper bound. Under `concept_exact` both sides land inside the bounds, and the net and nonzero count land near HT2's. + +**Rental and royalty** + +| | Line 26 net (`rental_income`) | Line 26 nonzero units | With farm rent: positive part | With farm rent: profit units | With farm rent: net | +|---|---:|---:|---:|---:|---:| +| IRS | HT2: $95.9B | HT2: 9.40M | $166.8B | 6.38M | $104.4B | +| release | $248.6B | 14.09M | $345.3B | 9.32M | $253.1B | +| concept_exact | $98.4B | 9.32M | $199.5B | 6.15M | $111.3B | +| drop | $95.1B | 14.25M | $216.1B | 8.95M | $96.2B | +| rebase_positive_part | $118.2B | 10.67M | $192.9B | 6.35M | $119.8B | + +The release's positive-only rental count, pinned at HT2's profit-or-loss count, held 9.3M positive units against the net target. That left the rental net at 2.6× its target. `concept_exact` brings both the line-26 net and count within 3% of HT2. + +### Reading + +- **`concept_exact`** is the only arm that is right on every quantity the HT2 columns measure: + - the business, partnership and rental nets and nonzero counts land within 0.3–7.3% of HT2 nationally; + - 82–100% of each measure's state rows land within 10%; + - fit elsewhere and ESS are unchanged. + + Its remaining error, Schedule C profit units at 1.28× and profits at 0.87×, is the frame's missing Schedule C losses. +- **`rebase_positive_part`** gets Schedule C profits right and the net 30% high, for the same reason. It needs the Table 1.4 legs in Chronicle (PolicyEngine/chronicle#346) and a share assumption. It has no partnership counterpart. +- **Dropping the Schedule C count** (`concept_exact_no_business_count`) halves the Schedule C units (14.4M against 30.4M nonzero returns) and buys almost nothing. +- **Dropping all 260 rows** lets Schedule C reach $1.3T. + +Once the frame carries Schedule C losses and Chronicle carries the legs, `concept_exact` plus national Table 1.4 profit and loss targets can get both sides. diff --git a/experiments/1228-net-income-less-loss/analyze.py b/experiments/1228-net-income-less-loss/analyze.py new file mode 100644 index 0000000000..13b1b6b070 --- /dev/null +++ b/experiments/1228-net-income-less-loss/analyze.py @@ -0,0 +1,287 @@ +"""Compare the #1228 re-solve arms of the Route A release. + +Reads ``prepare.py``'s outputs (``--inputs``) and each arm's ``metrics.json`` +and ``weights.npz`` (``--runs``), and writes ``results/summary.json`` and +``results/summary.md``: + +* **Reproduction.** The release arm against the published weights: final + loss, weight correlation and the largest per-household ratio. +* **Fit.** Each arm's own-surface loss, and, on the 5,382 rows every arm + shares (everything but the 312 business, partnership and rental rows), the + share within 10% and the capped error, overall and by SOI concept. +* **The 312 rows on both concepts.** Each arm's estimates against the + published values, read as the release concepts and as the #1228 concepts. +* **Model totals.** National positive part, negative part, net, and counts of + each income concept under each arm's weights, beside the IRS Table 1.4 + TY2022 legs aged by the rows' own factor (1.12035). +* **Concentration.** Kish ESS, and the chi-square distance from the initial + weights. + +Run: ``uv run --no-sync python experiments/1228-net-income-less-loss/analyze.py --inputs DIR --runs DIR`` +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy import sparse + +HERE = Path(__file__).resolve().parent +ARMS = ( + "release", + "concept_exact", + "concept_exact_no_business_count", + "drop", + "rebase_positive_part", +) +#: Table 1.4 TY2022 all-returns legs (22in14ar.xls; amounts in dollars), +#: aged by the HT2 rows' factor so they sit beside the 2024 targets. +AGING = 1.12034597479376 +IRS_TY2022 = { + "schedule_c_total": { + "pos_count": 21_969_832, + "neg_count": 8_386_569, + "pos_sum": 543_564_980e3 * AGING, + "neg_sum": -133_172_174e3 * AGING, + }, + "partnership_s_corp": { + # Separate partnership and S-corp legs; their union bounds the + # combined count, and the summed amounts bound the combined parts. + "pos_count_upper": 3_311_926 + 4_027_461, + "pos_sum_upper": (487_094_509e3 + 812_741_318e3) * AGING, + "net": (487_094_509e3 - 175_602_300e3 + 812_741_318e3 - 93_025_607e3) * AGING, + }, + "rental_plus_farm_rent": { + "pos_count": 6_380_131, + "neg_count": 3_313_901, + "pos_sum": 148_879_455e3 * AGING, + "neg_sum": -55_702_877e3 * AGING, + }, +} +CONCEPTS = ( + "schedule_c_total", + "schedule_c_nonsstb", + "sstb_only", + "partnership_s_corp", + "rental_only", + "rental_plus_farm_rent", + "farm_rent_only", +) + + +def capped_error(estimates: np.ndarray, targets: np.ndarray) -> np.ndarray: + scale = np.maximum(np.abs(targets), 1.0) + return np.minimum(np.abs(estimates - targets) / scale, 1.0) + + +def kish(weights: np.ndarray) -> float: + return float(weights.sum() ** 2 / np.square(weights).sum()) + + +def fit(estimates: np.ndarray, targets: np.ndarray) -> dict: + error = capped_error(estimates, targets) + return { + "n": int(len(targets)), + "within_10pct": float((error <= 0.10).mean()) if len(error) else None, + "mean_capped_error": float(error.mean()) if len(error) else None, + } + + +def concept_totals(totals: dict[str, np.ndarray], weights: np.ndarray) -> dict: + out = {} + for concept in CONCEPTS: + pos = float(totals[f"{concept}__pos_sum"] @ weights) + neg = float(totals[f"{concept}__neg_sum"] @ weights) + pos_count = float(totals[f"{concept}__pos_count"] @ weights) + neg_count = float(totals[f"{concept}__neg_count"] @ weights) + out[concept] = { + "pos_sum": pos, + "neg_sum": neg, + "net": pos + neg, + "pos_count": pos_count, + "neg_count": neg_count, + "nonzero_count": pos_count + neg_count, + } + out["tax_units"] = float(totals["tax_units"] @ weights) + return out + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--inputs", type=Path, required=True) + parser.add_argument("--runs", type=Path, required=True) + parser.add_argument("--results", type=Path, default=HERE / "results") + args = parser.parse_args() + + meta = pd.read_parquet(args.inputs / "targets_meta.parquet") + matrix = sparse.load_npz(args.inputs / "target_matrix.npz").tocsr() + with np.load(args.inputs / "concept_rows.npz") as payload: + concept_rows = payload["rows"] + concept_matrix = sparse.csr_array( + (payload["data"], payload["indices"], payload["indptr"]), + shape=tuple(payload["shape"]), + ) + weights_payload = dict(np.load(args.inputs / "weights.npz")) + totals = dict(np.load(args.inputs / "concept_totals.npz")) + initial = weights_payload["initial"] + published = weights_payload["published"] + values = meta["value"].to_numpy() + common = ~meta["net_income_concept"].astype(bool).to_numpy() + concept_values = values[concept_rows] + concept_meta = meta.iloc[concept_rows].reset_index(drop=True) + + arms = {"published": {"weights": published, "metrics": None}} + for arm in ARMS: + path = args.runs / arm + if not (path / "weights.npz").exists(): + print(f"missing arm {arm}") + continue + with np.load(path / "weights.npz") as payload: + weights = payload["weights"] + arms[arm] = { + "weights": weights, + "metrics": json.loads((path / "metrics.json").read_text()), + } + + summary: dict = {"arms": {}} + for arm, payload in arms.items(): + weights = payload["weights"] + release_estimates = matrix @ weights + concept_estimates = concept_matrix @ weights + by_concept = {} + for variable, group in meta[common].groupby("variable"): + rows = group["row"].to_numpy() + by_concept[str(variable)] = fit(release_estimates[rows], values[rows]) + rows_312 = {} + for stem in ( + "schedule_c_income", + "partnership_scorp_income", + "rental_royalty_income", + ): + for kind in ("returns", "amount"): + measure = f"{stem}_{kind}" + mask = (concept_meta["measure_id"] == measure).to_numpy() + us = mask & (concept_meta["geography_level"] == "country").to_numpy() + states = mask & (concept_meta["geography_level"] == "state").to_numpy() + rows_312[measure] = { + "us_target": float(concept_values[us][0]), + "us_release_concept": float(release_estimates[concept_rows[us]][0]), + "us_concept_exact": float(concept_estimates[us][0]), + "states_release_concept": fit( + release_estimates[concept_rows[states]], + concept_values[states], + ), + "states_concept_exact": fit( + concept_estimates[states], concept_values[states] + ), + } + own = None + if payload["metrics"]: + metrics = payload["metrics"] + own_rows = np.asarray(metrics["own_rows"], dtype=np.int64) + own_values = np.asarray(metrics["own_values"]) + own_estimates = np.asarray(metrics["own_estimates"]) + own_weights = np.asarray(metrics["own_loss_weights"]) + error = capped_error(own_estimates, own_values) + in_312 = np.isin(own_rows, concept_rows) + own = { + **fit(own_estimates, own_values), + "weighted_capped_error": float( + np.sum(own_weights * error) / np.sum(own_weights) + ), + "rows_312": fit(own_estimates[in_312], own_values[in_312]), + "rows_312_by_measure": { + measure: fit( + own_estimates[in_312 & (own_measure == measure)], + own_values[in_312 & (own_measure == measure)], + ) + for own_measure in [meta["measure_id"].to_numpy()[own_rows]] + for measure in sorted(set(own_measure[in_312])) + }, + } + record = { + "own_surface": own, + "own_final_loss": ( + payload["metrics"]["final_loss"] if payload["metrics"] else None + ), + "seconds": payload["metrics"]["seconds"] if payload["metrics"] else None, + "common_rows": fit(release_estimates[common], values[common]), + "common_rows_by_variable": by_concept, + "rows_312": rows_312, + "model_totals": concept_totals(totals, weights), + "kish_ess": kish(weights), + "chi_square_from_initial": float( + np.sum(np.square(weights - initial) / initial) / initial.sum() + ), + "max_ratio_to_published": float(np.max(weights / published)), + "min_ratio_to_published": float(np.min(weights / published)), + "correlation_with_published": float(np.corrcoef(weights, published)[0, 1]), + } + summary["arms"][arm] = record + summary["irs_ty2022_aged"] = IRS_TY2022 + args.results.mkdir(parents=True, exist_ok=True) + (args.results / "summary.json").write_text(json.dumps(summary, indent=1)) + (args.results / "summary.md").write_text(render(summary)) + print(render(summary)) + + +def render(summary: dict) -> str: + arms = summary["arms"] + lines = ["# #1228 re-solve summary", ""] + lines += [ + "| Arm | Own loss | Own rows within 10% | Own 312 rows within 10% | Common rows within 10% | Common mean capped error | Kish ESS | Corr. with published |", + "|---|---:|---:|---:|---:|---:|---:|---:|", + ] + for arm, record in arms.items(): + loss = record["own_final_loss"] + own = record["own_surface"] + own_within = "" if own is None else f"{own['within_10pct']:.4f}" + own_312 = ( + "" + if own is None or own["rows_312"]["within_10pct"] is None + else f"{own['rows_312']['within_10pct']:.3f}" + ) + lines.append( + f"| {arm} | {'' if loss is None else f'{loss:.6f}'} | {own_within} | " + f"{own_312} | " + f"{record['common_rows']['within_10pct']:.4f} | " + f"{record['common_rows']['mean_capped_error']:.5f} | " + f"{record['kish_ess']:,.0f} | {record['correlation_with_published']:.4f} |" + ) + lines += ["", "## Model totals (2024 weights; $B and millions)", ""] + lines += [ + "| Arm | Concept | Positive $B | Negative $B | Net $B | Positive M | Negative M | Nonzero M |", + "|---|---|---:|---:|---:|---:|---:|---:|", + ] + for arm, record in arms.items(): + for concept, totals in record["model_totals"].items(): + if concept == "tax_units": + continue + lines.append( + f"| {arm} | {concept} | {totals['pos_sum'] / 1e9:,.1f} | " + f"{totals['neg_sum'] / 1e9:,.1f} | {totals['net'] / 1e9:,.1f} | " + f"{totals['pos_count'] / 1e6:.2f} | {totals['neg_count'] / 1e6:.2f} | " + f"{totals['nonzero_count'] / 1e6:.2f} |" + ) + lines += ["", "## The 312 rows: US value and state fit on each concept", ""] + lines += [ + "| Arm | Measure | US target | US, release concept | US, #1228 concept | States within 10%, release concept | States within 10%, #1228 concept |", + "|---|---|---:|---:|---:|---:|---:|", + ] + for arm, record in arms.items(): + for measure, row in record["rows_312"].items(): + lines.append( + f"| {arm} | {measure} | {row['us_target']:,.0f} | " + f"{row['us_release_concept']:,.0f} | {row['us_concept_exact']:,.0f} | " + f"{row['states_release_concept']['within_10pct']:.2f} | " + f"{row['states_concept_exact']['within_10pct']:.2f} |" + ) + return "\n".join(lines) + "\n" + + +if __name__ == "__main__": + main() diff --git a/experiments/1228-net-income-less-loss/modal_resolve.py b/experiments/1228-net-income-less-loss/modal_resolve.py new file mode 100644 index 0000000000..35c5a33aa1 --- /dev/null +++ b/experiments/1228-net-income-less-loss/modal_resolve.py @@ -0,0 +1,198 @@ +"""Run the five #1228 arms of the Route A re-solve on Modal. + +Each arm is one ``resolve.py`` calibration of the release's 5,694-target +national_state surface (352,932 households) in its own container. The kernel +packages are installed from this worktree's committed tree; the launch refuses +a tree whose kernel sources or ``resolve.py`` differ from HEAD, and every run +records that HEAD. A finished arm is reused only when its HEAD, its inputs' +``MANIFEST.json`` digest and its ``resolve.py`` digest all match. + + uv run --no-sync modal run --detach experiments/1228-net-income-less-loss/modal_resolve.py::launch --inputs DIR + uv run --no-sync modal run experiments/1228-net-income-less-loss/modal_resolve.py::collect --into DIR + +``launch`` uploads ``prepare.py``'s outputs once and fans out the arms. +``collect`` copies each finished arm's ``metrics.json`` and ``weights.npz``. +""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +import modal + +HERE = Path(__file__).resolve().parent +REPO = ( + HERE.parents[1] + if len(HERE.parents) > 1 and (HERE.parents[1] / ".git").exists() + else None +) +UPLOAD = ( + "MANIFEST.json", + "target_matrix.npz", + "concept_rows.npz", + "targets_meta.parquet", + "weights.npz", + "rebase_rows.npz", + "rebase_meta.parquet", +) +KERNEL = ( + "microcosm-frame", + "microcosm-graph", + "microcosm-diagnostics", + "microcosm-calibrate", +) +ARMS = ( + "release", + "concept_exact", + "concept_exact_no_business_count", + "drop", + "rebase_positive_part", +) +VOLUME_ROOT = "/resolve" +CPUS = 16 + + +def _head() -> str: + return subprocess.run( + ["git", "-C", str(REPO), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + + +def _kernel_clean() -> bool: + status = subprocess.run( + ["git", "-C", str(REPO), "status", "--porcelain", "--"] + + [f"packages/{name}/src" for name in KERNEL] + + [str(HERE.relative_to(REPO) / "resolve.py")], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + return not status + + +GIT_SHA = _head() if REPO is not None else "container" + +image = ( + modal.Image.debian_slim(python_version="3.13") + .uv_pip_install( + "torch==2.12.0", + index_url="https://download.pytorch.org/whl/cpu", + ) + .uv_pip_install( + "numpy==2.4.6", + "scipy==1.17.1", + "pandas==3.0.3", + "pyarrow==25.0.0", + "pydantic>=2.10,<3", + ) + .env( + { + "OMP_NUM_THREADS": str(CPUS), + "MKL_NUM_THREADS": str(CPUS), + "OPENBLAS_NUM_THREADS": str(CPUS), + "MICROCOSM_GIT_SHA": GIT_SHA, + "MICROCOSM_REPO": "/opt/none", + } + ) +) +if REPO is not None: + for name in KERNEL: + image = image.add_local_dir( + str(REPO / "packages" / name), f"/opt/kernel/{name}", copy=True + ) + image = image.run_commands( + "python -m pip install --no-deps " + + " ".join(f"/opt/kernel/{name}" for name in KERNEL) + ).add_local_file(str(HERE / "resolve.py"), "/opt/resolve/resolve.py") + +app = modal.App("microcosm-1228-net-income-resolve", image=image) +volume = modal.Volume.from_name( + "microcosm-1228-net-income-resolve", create_if_missing=True +) + + +@app.function( + cpu=CPUS, + memory=24576, + timeout=6 * 3600, + volumes={VOLUME_ROOT: volume}, + retries=1, +) +def run_arm(arm: str) -> dict: + import sys + + import torch + + torch.set_num_threads(CPUS) + volume.reload() + out_root = Path(VOLUME_ROOT) / "runs" + finished = out_root / arm / "metrics.json" + if finished.exists(): + import hashlib + + payload = json.loads(finished.read_text()) + identity = { + "git_sha": GIT_SHA, + "inputs_manifest_sha256": hashlib.sha256( + Path(f"{VOLUME_ROOT}/inputs/MANIFEST.json").read_bytes() + ).hexdigest(), + "resolve_sha256": hashlib.sha256( + Path("/opt/resolve/resolve.py").read_bytes() + ).hexdigest(), + } + if all(payload.get(key) == value for key, value in identity.items()): + return {"arm": arm, "cached": True, "final_loss": payload["final_loss"]} + sys.argv = [ + "resolve.py", + "--inputs", + f"{VOLUME_ROOT}/inputs", + "--out", + str(out_root), + "--arm", + arm, + ] + sys.path.insert(0, "/opt/resolve") + import resolve + + resolve.main() + volume.commit() + payload = json.loads(finished.read_text()) + return { + "arm": arm, + "final_loss": payload["final_loss"], + "seconds": payload["seconds"], + } + + +@app.local_entrypoint() +def launch(inputs: str) -> None: + if not _kernel_clean(): + raise SystemExit("kernel sources differ from HEAD; commit before launching") + print(f"kernel HEAD {GIT_SHA}") + with volume.batch_upload(force=True) as batch: + for name in UPLOAD: + batch.put_file(str(Path(inputs) / name), f"/inputs/{name}") + for result in run_arm.map(ARMS, return_exceptions=True): + print(json.dumps(result, default=str), flush=True) + + +@app.local_entrypoint() +def collect(into: str) -> None: + target = Path(into) + for arm in ARMS: + for name in ("metrics.json", "weights.npz"): + remote = f"/runs/{arm}/{name}" + try: + data = b"".join(volume.read_file(remote)) + except FileNotFoundError: + print(f"missing {remote}") + continue + path = target / arm / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + print(f"wrote {path} ({len(data)} bytes)") diff --git a/experiments/1228-net-income-less-loss/prepare.py b/experiments/1228-net-income-less-loss/prepare.py new file mode 100644 index 0000000000..662cf598c5 --- /dev/null +++ b/experiments/1228-net-income-less-loss/prepare.py @@ -0,0 +1,476 @@ +"""Build the #1228 re-solve inputs from the Route A release's own checkpoint. + +Inputs (read only): + +* ``release-checkpoints/target_frame_checkpoint.h5``: the release's + materialized target frame, one household column per target (5,694) plus the + person and tax-unit input tables. +* ``calibration_diagnostics.json``: each target's name, measure column, + compiled value, loss weight, initial and final estimates, and metadata. +* ``populace_us_2024_calibration.npz``: the initial and published weights. + +Outputs (under ``--out``): + +* ``target_matrix.npz``: the release's 5,694 target rows as CSR (float64), + in the diagnostics' order. Checked: ``A @ initial`` and ``A @ final`` + reproduce every target's recorded initial and final estimate. +* ``concept_rows.npz``: the 312 business, partnership and rental rows rebuilt + on the signed net-income-(less-loss) concepts of #1228, as CSR, with their + diagnostics row numbers. The script first rebuilds the same 312 rows on the + release's own concepts and requires them to equal the checkpoint's columns, + so both sets come from one verified materialization. +* ``targets_meta.parquet``: one row per target: name, value, family, SOI + measure id, geography, release loss weight, and the loss weights of every + arm (recomputed with ``target_loss_weights`` over the arm's specs; the + release arm must reproduce the recorded weights). +* ``concept_totals.npz``: household-level tax-unit sums (positive part, + negative part, positive count, negative count) of each income concept, so + any weight vector's model totals can be read off without a simulation. +* ``weights.npz``: ``initial`` and ``published``. +* ``MANIFEST.json``: sha256 of each output, the input paths and digests, the + structural checks, and this repository's HEAD. + +Run: ``uv run --no-sync python experiments/1228-net-income-less-loss/prepare.py --out DIR`` +""" + +from __future__ import annotations + +import argparse +import hashlib +import io +import json +import resource +import subprocess +import time +import zipfile +from pathlib import Path +from types import SimpleNamespace + +import h5py +import numpy as np +import pandas as pd +from scipy import sparse + +HERE = Path(__file__).resolve().parent +REPO = HERE.parents[1] +RELEASE_ID = "populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z" +RUN = Path( + "/Users/maxghenis/PolicyEngine/_recovered/scratch-backup/893/" + "overnight-20260923/route-a/run-4b57d15a287c" +) +CHECKPOINT = RUN / "release-checkpoints" / "target_frame_checkpoint.h5" +RELEASE = RUN / "release-out" / RELEASE_ID +DIAGNOSTICS = RELEASE / "releases" / RELEASE_ID / "calibration_diagnostics.json" +CALIBRATION_NPZ = RELEASE / "artifacts" / "populace_us_2024_calibration.npz" + +#: The six HT2 measure ids (#1228), their release concept and their +#: net-income-(less-loss) concept. +MEASURES = { + "schedule_c_income": ("schedule_c_nonsstb", "schedule_c_total"), + "partnership_scorp_income": ("partnership_s_corp", "partnership_s_corp"), + "rental_royalty_income": ("rental_plus_farm_rent", "rental_only"), +} +#: Person inputs summed to the tax unit for each concept. +CONCEPT_INPUTS = { + "schedule_c_nonsstb": ("self_employment_income_before_lsr",), + "schedule_c_total": ( + "self_employment_income_before_lsr", + "sstb_self_employment_income_before_lsr", + ), + "sstb_only": ("sstb_self_employment_income_before_lsr",), + "partnership_s_corp": ("partnership_income", "s_corp_income"), + "rental_only": ("rental_income",), + "rental_plus_farm_rent": ("rental_income", "farm_rent_income"), + "farm_rent_only": ("farm_rent_income",), +} +#: Arms: which of the 312 rows each keeps, and on which concept. +#: ``release``: all on the release concepts. ``concept_exact``: all on the +#: #1228 concepts. ``concept_exact_no_business_count``: the same without the +#: 52 Schedule C return counts (the frame has almost no Schedule C loss +#: carriers). ``drop``: the 260 rows whose release concept differs from the +#: IRS column are dropped; the rental amount keeps its release concept. +ARMS = ( + "release", + "concept_exact", + "concept_exact_no_business_count", + "drop", +) + + +def log(message: str, started: float) -> None: + rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e9 + print(f"[{time.time() - started:7.1f}s rss={rss:.2f}GB] {message}", flush=True) + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for block in iter(lambda: handle.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +class CheckpointTable: + """One table of the checkpoint: columns stored one group per column.""" + + def __init__(self, handle: h5py.File, table: str): + self.group = handle[f"tables/{table}"] + self.names = json.loads(self.group.attrs["columns_json"]) + self.index = {name: position for position, name in enumerate(self.names)} + + def column(self, name: str) -> np.ndarray: + key = f"{self.index[name]:05d}" + node = self.group["columns"][key] + assert node.attrs["name"] == name, (node.attrs["name"], name) + return node["values"][()] + + +def npz_array(path: Path, name: str) -> np.ndarray: + with zipfile.ZipFile(path) as archive: + return np.lib.format.read_array( + io.BytesIO(archive.read(f"{name}.npy")), allow_pickle=False + ) + + +def measure_id(entry: dict) -> str: + return str(entry["metadata"].get("source_measure_id", "")) + + +def concept_stem(entry: dict) -> str | None: + measure = measure_id(entry) + for stem in MEASURES: + if measure in {f"{stem}_amount", f"{stem}_returns"}: + return stem + return None + + +def tax_unit_concepts(person: CheckpointTable, household_ids: np.ndarray): + """Per-tax-unit totals of each concept, and each tax unit's household row.""" + + tax_unit_of_person = person.column("person_tax_unit_id").astype(np.int64) + household_of_person = person.column("person_household_id").astype(np.int64) + units, unit_position = np.unique(tax_unit_of_person, return_inverse=True) + household_row = pd.Series(np.arange(len(household_ids)), index=household_ids) + unit_household = np.full(len(units), -1, dtype=np.int64) + unit_household[unit_position] = household_row.reindex( + household_of_person + ).to_numpy() + assert (unit_household >= 0).all(), "a person's household is not in the frame" + # Every person of a tax unit shares its household. + check = np.full(len(units), -1, dtype=np.int64) + np.maximum.at( + check, unit_position, household_row.reindex(household_of_person).to_numpy() + ) + assert np.array_equal(check, unit_household), "a tax unit spans households" + inputs = { + column: person.column(column).astype(np.float64) + for columns in CONCEPT_INPUTS.values() + for column in columns + } + totals = {} + for concept, columns in CONCEPT_INPUTS.items(): + person_total = np.zeros(len(tax_unit_of_person), dtype=np.float64) + for column in columns: + person_total += inputs[column] + unit_total = np.zeros(len(units), dtype=np.float64) + np.add.at(unit_total, unit_position, person_total) + totals[concept] = unit_total + return totals, unit_household + + +def household_row( + unit_values: np.ndarray, unit_household: np.ndarray, n_households: int +) -> np.ndarray: + out = np.zeros(n_households, dtype=np.float64) + np.add.at(out, unit_household, unit_values) + return out + + +def materialize( + entry: dict, + *, + concept_totals: dict[str, np.ndarray], + unit_household: np.ndarray, + household_state: np.ndarray, + exact: bool, +) -> np.ndarray: + """One target's household column on the release or the #1228 concept. + + Mirrors tools/build_us_fiscal_refresh_release.py ``_soi_component_row``: + the release takes the positive part of business and partnership income + and counts a positive total; rental keeps the sign but counts a positive + total. The #1228 concepts keep the sign and count a nonzero total. + """ + + stem = concept_stem(entry) + release_concept, exact_concept = MEASURES[stem] + values = concept_totals[exact_concept if exact else release_concept] + returns = measure_id(entry).endswith("_returns") + if exact: + unit = (values != 0).astype(np.float64) if returns else values + elif returns: + unit = (values > 0).astype(np.float64) + elif stem == "rental_royalty_income": + unit = values + else: + unit = np.maximum(values, 0.0) + metadata = entry["metadata"] + assert metadata.get("agi_lower_bound") == "-inf", metadata.get("agi_lower_bound") + assert metadata.get("agi_upper_bound") == "inf", metadata.get("agi_upper_bound") + assert metadata.get("filing_status") == "All", metadata.get("filing_status") + assert "itemized_only" not in metadata and "taxable_only" not in metadata + column = household_row(unit, unit_household, len(household_state)) + state_fips = metadata.get("state_fips") + if state_fips: + column = np.where(household_state == int(state_fips), column, 0.0) + return column + + +def arm_rows(entries: list[dict], arm: str) -> np.ndarray: + keep = [] + for row, entry in enumerate(entries): + measure = measure_id(entry) + if arm == "concept_exact_no_business_count" and ( + measure == "schedule_c_income_returns" + ): + continue + if ( + arm == "drop" + and concept_stem(entry) is not None + and (measure != "rental_royalty_income_amount") + ): + continue + keep.append(row) + return np.asarray(keep, dtype=np.int64) + + +def arm_specs(entries: list[dict], rows: np.ndarray, *, exact: bool) -> list: + """Spec stand-ins for ``target_loss_weights``: what the formula reads.""" + + concepts = { + "schedule_c_income": "business_net_income_less_loss", + "partnership_scorp_income": "partnership_and_s_corp_net_income_less_loss", + "rental_royalty_income": "rent_and_royalty_net_income_less_loss", + } + specs = [] + for row in rows: + entry = entries[int(row)] + metadata = dict(entry["metadata"]) + stem = concept_stem(entry) + if exact and stem is not None: + metadata["source_variable"] = concepts[stem] + metadata["variable"] = concepts[stem] + specs.append( + SimpleNamespace( + name=entry["target_name"], + value=float(entry["compiled_target"]), + family=entry["registry"]["family"], + entity=entry["entity"], + period=entry["period"], + filter=entry["filter"], + metadata=metadata, + ) + ) + return specs + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + started = time.time() + out: Path = args.out + out.mkdir(parents=True, exist_ok=True) + + diagnostics = json.loads(DIAGNOSTICS.read_text()) + entries: list[dict] = diagnostics["targets"] + log(f"{len(entries)} targets from diagnostics", started) + initial = npz_array(CALIBRATION_NPZ, "initial_household_weight") + published = npz_array(CALIBRATION_NPZ, "household_weight") + + with h5py.File(CHECKPOINT, "r") as handle: + household = CheckpointTable(handle, "household") + person = CheckpointTable(handle, "person") + household_ids = household.column("household_id").astype(np.int64) + household_state = household.column("state_fips").astype(np.int64) + n_households = len(household_ids) + assert n_households == len(initial) == len(published) + checkpoint_weights = handle["weights/household/values"][()] + assert np.allclose(checkpoint_weights, initial, rtol=0, atol=1e-9), ( + "checkpoint weights are not the release's initial weights" + ) + + indptr = [0] + indices: list[np.ndarray] = [] + data: list[np.ndarray] = [] + release_columns: dict[int, np.ndarray] = {} + for row, entry in enumerate(entries): + column = household.column(entry["measure"]["name"]).astype(np.float64) + nonzero = np.flatnonzero(column) + indices.append(nonzero.astype(np.int32)) + data.append(column[nonzero]) + indptr.append(indptr[-1] + len(nonzero)) + if concept_stem(entry) is not None: + release_columns[row] = column + if row % 500 == 0: + log(f"read row {row}", started) + matrix = sparse.csr_array( + (np.concatenate(data), np.concatenate(indices), np.asarray(indptr)), + shape=(len(entries), n_households), + ) + log(f"matrix {matrix.shape} nnz={matrix.nnz}", started) + + concept_totals, unit_household = tax_unit_concepts(person, household_ids) + log(f"{len(unit_household)} tax units", started) + + checks: dict[str, object] = {} + for label, weights, key in ( + ("initial", initial, "initial_estimate"), + ("published", published, "final_estimate"), + ): + estimates = matrix @ weights + recorded = np.asarray([entry[key] for entry in entries], dtype=np.float64) + gap = np.abs(estimates - recorded) / np.maximum(np.abs(recorded), 1.0) + checks[f"{label}_estimates_max_relative_gap"] = float(gap.max()) + assert gap.max() < 1e-6, (label, float(gap.max())) + + exact_rows = sorted(release_columns) + assert len(exact_rows) == 312, len(exact_rows) + rebuild_gap = 0.0 + exact_columns = [] + for row in exact_rows: + entry = entries[row] + kwargs = dict( + concept_totals=concept_totals, + unit_household=unit_household, + household_state=household_state, + ) + rebuilt = materialize(entry, exact=False, **kwargs) + reference = release_columns[row] + scale = max(float(np.abs(reference).max()), 1.0) + rebuild_gap = max(rebuild_gap, float(np.abs(rebuilt - reference).max()) / scale) + exact_columns.append(materialize(entry, exact=True, **kwargs)) + checks["release_concept_rebuild_max_relative_gap"] = rebuild_gap + assert rebuild_gap < 1e-6, rebuild_gap + exact_matrix = sparse.csr_array(np.vstack(exact_columns)) + log( + f"rebuilt 312 release rows (max gap {rebuild_gap:.2e}); exact rows nnz={exact_matrix.nnz}", + started, + ) + + from microcosm.build.us_runtime.target_loss_weights import target_loss_weights + + recorded_weights = np.asarray( + [entry["target_loss_weight"] for entry in entries], dtype=np.float64 + ) + arm_weights: dict[str, np.ndarray] = {} + arm_row_sets: dict[str, np.ndarray] = {} + for arm in ARMS: + rows = arm_rows(entries, arm) + weights = target_loss_weights( + arm_specs(entries, rows, exact=arm.startswith("concept_exact")) + ) + full = np.full(len(entries), np.nan) + full[rows] = weights + arm_weights[arm] = full + arm_row_sets[arm] = rows + log(f"arm {arm}: {len(rows)} targets", started) + release_gap = float( + np.max(np.abs(arm_weights["release"] - recorded_weights) / recorded_weights) + ) + checks["release_loss_weights_max_relative_gap"] = release_gap + assert release_gap < 1e-9, release_gap + exact_gap = float( + np.max( + np.abs(arm_weights["concept_exact"] - recorded_weights) / recorded_weights + ) + ) + checks["concept_exact_loss_weights_max_relative_gap"] = exact_gap + + meta = pd.DataFrame( + { + "row": np.arange(len(entries)), + "name": [entry["target_name"] for entry in entries], + "value": [float(entry["compiled_target"]) for entry in entries], + "family": [entry["registry"]["family"] for entry in entries], + "measure_id": [measure_id(entry) for entry in entries], + "variable": [entry["metadata"].get("variable", "") for entry in entries], + "geography_level": [ + entry["hierarchy"]["geography"]["level"] for entry in entries + ], + "state_fips": [ + entry["metadata"].get("state_fips", "") for entry in entries + ], + "net_income_concept": [concept_stem(entry) or "" for entry in entries], + "initial_estimate": [entry["initial_estimate"] for entry in entries], + "published_estimate": [entry["final_estimate"] for entry in entries], + **{f"loss_weight_{arm}": arm_weights[arm] for arm in ARMS}, + } + ) + meta.to_parquet(out / "targets_meta.parquet", index=False) + sparse.save_npz(out / "target_matrix.npz", matrix, compressed=True) + np.savez_compressed( + out / "concept_rows.npz", + rows=np.asarray(exact_rows, dtype=np.int64), + data=exact_matrix.data, + indices=exact_matrix.indices, + indptr=exact_matrix.indptr, + shape=np.asarray(exact_matrix.shape), + ) + household_totals = {} + for concept, values in concept_totals.items(): + for part, unit in ( + ("pos_sum", np.maximum(values, 0.0)), + ("neg_sum", np.minimum(values, 0.0)), + ("pos_count", (values > 0).astype(np.float64)), + ("neg_count", (values < 0).astype(np.float64)), + ): + household_totals[f"{concept}__{part}"] = household_row( + unit, unit_household, n_households + ) + household_totals["tax_units"] = household_row( + np.ones(len(unit_household)), unit_household, n_households + ) + np.savez_compressed( + out / "concept_totals.npz", state_fips=household_state, **household_totals + ) + np.savez_compressed( + out / "weights.npz", + initial=initial, + published=published, + **{f"rows_{arm}": arm_row_sets[arm] for arm in ARMS}, + ) + head = subprocess.run( + ["git", "-C", str(REPO), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + manifest = { + "release_id": RELEASE_ID, + "inputs": { + str(path): sha256(path) + for path in (CHECKPOINT, DIAGNOSTICS, CALIBRATION_NPZ) + }, + "outputs": { + name: sha256(out / name) + for name in ( + "target_matrix.npz", + "concept_rows.npz", + "targets_meta.parquet", + "concept_totals.npz", + "weights.npz", + ) + }, + "checks": checks, + "arms": {arm: int(len(arm_row_sets[arm])) for arm in ARMS}, + "release_options": diagnostics["options"], + "release_final_loss": diagnostics["final_loss"], + "repository_head": head, + } + (out / "MANIFEST.json").write_text(json.dumps(manifest, indent=1, sort_keys=True)) + log(f"wrote {out}", started) + + +if __name__ == "__main__": + main() diff --git a/experiments/1228-net-income-less-loss/prepare_rebase.py b/experiments/1228-net-income-less-loss/prepare_rebase.py new file mode 100644 index 0000000000..d8b02fb75c --- /dev/null +++ b/experiments/1228-net-income-less-loss/prepare_rebase.py @@ -0,0 +1,171 @@ +"""Add the "rebase onto a one-sided Table 1.4 control" arm (#1228 option 2). + +The capital-gains fix (#1036) keeps the model's one-sided concept and binds +each HT2 state row as its share of a national Table 1.4 control. This builds +the same for business and rental income, from ``prepare.py``'s outputs, so +the option can be measured before Chronicle carries the legs +(PolicyEngine/chronicle#346): + +* **Business.** The model concept is the positive part of + ``total_self_employment_income``, counted when positive. The control is + Table 1.4 TY2022 "Business or profession: Net income": 21,969,832 returns + and $543,564,980 thousand. +* **Rental.** The model concept is the positive part of ``rental_income + + farm_rent_income``, counted when positive. The control is Table 1.4 TY2022 + "Total rental and royalty: Net income", which includes farm rental: + 6,380,131 returns and $148,879,455 thousand. +* **Partnership and S corporation.** Table 1.4 has had no combined net-income + column since TY2020, so there is no control. These rows keep the + concept-exact binding. + +A state row's value is its HT2 value / the HT2 US value × the control; the US +row's value is the control. Amounts are aged by the rows' own factor, +1.12034597479376, and counts are not aged, as in the release. + +Writes ``rebase_rows.npz`` (the 312 rows on this arm's concepts, CSR, with +their values) and ``rebase_meta.parquet`` (each target's value and loss weight +on this arm, with the weights recomputed by ``target_loss_weights``). + +Run: ``uv run --no-sync python experiments/1228-net-income-less-loss/prepare_rebase.py --inputs DIR`` +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pandas as pd +from prepare import DIAGNOSTICS, concept_stem, measure_id +from scipy import sparse + +AGING = 1.12034597479376 +CONTROLS = { + "schedule_c_income_returns": ("schedule_c_total", "pos_count", 21_969_832.0), + "schedule_c_income_amount": ( + "schedule_c_total", + "pos_sum", + 543_564_980e3 * AGING, + ), + "rental_royalty_income_returns": ( + "rental_plus_farm_rent", + "pos_count", + 6_380_131.0, + ), + "rental_royalty_income_amount": ( + "rental_plus_farm_rent", + "pos_sum", + 148_879_455e3 * AGING, + ), +} +#: Model concept ids for the loss-weight stand-in specs. +ONE_SIDED = { + "schedule_c_income": "business_net_profits", + "rental_royalty_income": "rent_and_royalty_net_income", + "partnership_scorp_income": "partnership_and_s_corp_net_income_less_loss", +} + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--inputs", type=Path, required=True) + args = parser.parse_args() + inputs: Path = args.inputs + + entries = json.loads(DIAGNOSTICS.read_text())["targets"] + meta = pd.read_parquet(inputs / "targets_meta.parquet") + totals = dict(np.load(inputs / "concept_totals.npz")) + state = totals["state_fips"] + with np.load(inputs / "concept_rows.npz") as payload: + concept_rows = payload["rows"] + exact = sparse.csr_array( + (payload["data"], payload["indices"], payload["indptr"]), + shape=tuple(payload["shape"]), + ) + values = meta["value"].to_numpy().copy() + us_value = { + measure: float( + meta.loc[ + (meta["measure_id"] == measure) + & (meta["geography_level"] == "country"), + "value", + ].iloc[0] + ) + for measure in CONTROLS + } + rows = [] + for position, row in enumerate(concept_rows): + entry = entries[int(row)] + measure = measure_id(entry) + if measure not in CONTROLS: + rows.append(exact[[position]].toarray()[0]) + continue + concept, part, control = CONTROLS[measure] + column = totals[f"{concept}__{part}"].astype(np.float64) + state_fips = entry["metadata"].get("state_fips") + if state_fips: + column = np.where(state == int(state_fips), column, 0.0) + values[int(row)] = ( + float(entry["compiled_target"]) / us_value[measure] * control + ) + else: + values[int(row)] = control + rows.append(column) + rebase = sparse.csr_array(np.vstack(rows)) + + from microcosm.build.us_runtime.target_loss_weights import target_loss_weights + + specs = [] + for row, entry in enumerate(entries): + metadata = dict(entry["metadata"]) + stem = concept_stem(entry) + if stem is not None: + metadata["source_variable"] = ONE_SIDED[stem] + metadata["variable"] = ONE_SIDED[stem] + specs.append( + SimpleNamespace( + name=entry["target_name"], + value=float(values[row]), + family=entry["registry"]["family"], + entity=entry["entity"], + period=entry["period"], + filter=entry["filter"], + metadata=metadata, + ) + ) + weights = target_loss_weights(specs) + pd.DataFrame( + { + "row": np.arange(len(entries)), + "value_rebase_positive_part": values, + "loss_weight_rebase_positive_part": weights, + } + ).to_parquet(inputs / "rebase_meta.parquet", index=False) + np.savez_compressed( + inputs / "rebase_rows.npz", + rows=np.asarray(concept_rows, dtype=np.int64), + data=rebase.data, + indices=rebase.indices, + indptr=rebase.indptr, + shape=np.asarray(rebase.shape), + ) + manifest_path = inputs / "MANIFEST.json" + manifest = json.loads(manifest_path.read_text()) + for name in ("rebase_rows.npz", "rebase_meta.parquet"): + manifest["outputs"][name] = hashlib.sha256( + (inputs / name).read_bytes() + ).hexdigest() + manifest["rebase_positive_part"] = { + "controls": {measure: control for measure, (_, _, control) in CONTROLS.items()}, + "us_values": us_value, + "changed_values": int(np.sum(values != meta["value"].to_numpy())), + } + manifest_path.write_text(json.dumps(manifest, indent=1, sort_keys=True)) + print(json.dumps(manifest["rebase_positive_part"], indent=1)) + + +if __name__ == "__main__": + main() diff --git a/experiments/1228-net-income-less-loss/resolve.py b/experiments/1228-net-income-less-loss/resolve.py new file mode 100644 index 0000000000..485513b10a --- /dev/null +++ b/experiments/1228-net-income-less-loss/resolve.py @@ -0,0 +1,241 @@ +"""Re-solve one #1228 arm of the Route A release's national_state calibration. + +Reads ``prepare.py``'s outputs from ``--inputs``, swaps in the arm's rows and +loss weights, and calls ``microcosm.calibrate.calibrate`` with the release's +settings (``calibration_diagnostics.json`` options): Adam, 1,500 epochs, +learning rate 0.02, ``mass="conserve"``, ``max_weight_ratio=5``, +``target_loss_cap=1``, no L1/L2, seed 0, from the release's initial weights. + +Writes ``//weights.npz`` and ``//metrics.json``: the final +and initial loss, epochs, seconds, and the arm's estimates of every release +row and every #1228 row, so each arm can be scored on either concept. + +Run: ``python resolve.py --inputs DIR --out DIR --arm ARM [--epochs N]`` +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import platform +import resource +import time +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy import sparse + +RELEASE_SETTINGS = { + "weight_entity": "household", + "method": "adam", + "epochs": 1500, + "learning_rate": 0.02, + "mass": "conserve", + "max_weight_ratio": 5.0, + "target_loss_cap": 1.0, + "l2_lambda": 0.0, + "seed": 0, +} +ARMS = ( + "release", + "concept_exact", + "concept_exact_no_business_count", + "drop", + "rebase_positive_part", +) + + +def load_rows(path: Path) -> tuple[np.ndarray, sparse.csr_array]: + with np.load(path) as payload: + return payload["rows"], sparse.csr_array( + (payload["data"], payload["indices"], payload["indptr"]), + shape=tuple(payload["shape"]), + ) + + +def load(inputs: Path): + matrix = sparse.load_npz(inputs / "target_matrix.npz").tocsr() + meta = pd.read_parquet(inputs / "targets_meta.parquet") + concept_rows, concept_matrix = load_rows(inputs / "concept_rows.npz") + rebase_rows, rebase_matrix = load_rows(inputs / "rebase_rows.npz") + assert np.array_equal(concept_rows, rebase_rows) + rebase_meta = pd.read_parquet(inputs / "rebase_meta.parquet") + assert np.array_equal(rebase_meta["row"].to_numpy(), meta["row"].to_numpy()) + meta["value_rebase_positive_part"] = rebase_meta["value_rebase_positive_part"] + meta["loss_weight_rebase_positive_part"] = rebase_meta[ + "loss_weight_rebase_positive_part" + ] + weights = dict(np.load(inputs / "weights.npz")) + weights["rows_rebase_positive_part"] = np.arange(matrix.shape[0], dtype=np.int64) + return matrix, meta, concept_rows, concept_matrix, rebase_matrix, weights + + +def swap_rows( + matrix: sparse.csr_array, + swapped_rows: np.ndarray, + replacement: sparse.csr_array, + rows: np.ndarray, +) -> sparse.csr_array: + """``matrix[rows]`` with ``swapped_rows`` taken from ``replacement``.""" + + n_release = matrix.shape[0] + combined = sparse.csr_array(sparse.vstack([matrix, replacement]).tocsr()) + source = np.arange(n_release, dtype=np.int64) + source[swapped_rows] = n_release + np.arange(len(swapped_rows), dtype=np.int64) + return sparse.csr_array(combined[source[rows]]) + + +def csr_row_measure(matrix: sparse.csr_array, row: int, n: int): + start, stop = int(matrix.indptr[row]), int(matrix.indptr[row + 1]) + indices = matrix.indices[start:stop] + data = matrix.data[start:stop] + + def measure(frame) -> np.ndarray: + values = np.zeros(n, dtype=np.float64) + values[indices] = data + return values + + measure.__qualname__ = f"target_matrix_row[{row}]" + return measure + + +def build_frame(n_households: int, weights: np.ndarray): + from microcosm.frame import EntitySchema, Frame, WeightKind, Weights + + household_id = np.arange(n_households, dtype=np.int64) + tables = { + "household": pd.DataFrame({"household_id": household_id}), + "person": pd.DataFrame( + {"person_id": household_id, "person_household_id": household_id} + ), + } + return Frame( + tables, + EntitySchema(group_entities=("household",)), + {"household": Weights(np.asarray(weights, np.float64), WeightKind.DESIGN)}, + ) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--inputs", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--arm", choices=ARMS, required=True) + parser.add_argument("--epochs", type=int, default=RELEASE_SETTINGS["epochs"]) + args = parser.parse_args() + + from microcosm.calibrate import Target, TargetSet, calibrate + + started = time.time() + matrix, meta, concept_rows, concept_matrix, rebase_matrix, weights = load( + args.inputs + ) + rows = weights[f"rows_{args.arm}"].astype(np.int64) + loss_weights = meta[f"loss_weight_{args.arm}"].to_numpy()[rows] + assert np.isfinite(loss_weights).all() + if args.arm.startswith("concept_exact"): + solve_matrix = swap_rows(matrix, concept_rows, concept_matrix, rows) + elif args.arm == "rebase_positive_part": + solve_matrix = swap_rows(matrix, concept_rows, rebase_matrix, rows) + else: + solve_matrix = sparse.csr_array(matrix[rows]) + n_households = matrix.shape[1] + names = meta["name"].to_numpy()[rows] + value_column = ( + "value_rebase_positive_part" if args.arm == "rebase_positive_part" else "value" + ) + values = meta[value_column].to_numpy()[rows] + targets = TargetSet( + Target( + name=str(names[position]), + entity="household", + measure=csr_row_measure(solve_matrix, position, n_households), + value=float(values[position]), + period=2024, + source="route_a_checkpoint", + ) + for position in range(len(rows)) + ) + frame = build_frame(n_households, weights["initial"]) + trajectory: list[dict] = [] + + def on_progress(event: dict) -> None: + if event.get("kind") != "calibration_epoch": + return + epoch = int(event["epoch"]) + if epoch == 1 or epoch % 100 == 0 or epoch == args.epochs: + trajectory.append({"epoch": epoch, "loss": float(event["loss"])}) + elapsed = time.time() - started + print( + f"[{args.arm}] epoch {epoch} loss {float(event['loss']):.6f} " + f"{elapsed:.0f}s", + flush=True, + ) + + result = calibrate( + frame, + targets, + weight_entity=RELEASE_SETTINGS["weight_entity"], + method=RELEASE_SETTINGS["method"], + epochs=args.epochs, + learning_rate=RELEASE_SETTINGS["learning_rate"], + mass=RELEASE_SETTINGS["mass"], + max_weight_ratio=RELEASE_SETTINGS["max_weight_ratio"], + l2_lambda=RELEASE_SETTINGS["l2_lambda"], + seed=RELEASE_SETTINGS["seed"], + target_loss_weights=loss_weights, + target_loss_cap=RELEASE_SETTINGS["target_loss_cap"], + progress_callback=on_progress, + ) + if result.problem.skipped: + raise SystemExit(f"calibration skipped {len(result.problem.skipped)} targets") + compiled = [str(name).removesuffix("@2024") for name in result.problem.names] + if compiled != [str(name) for name in names]: + raise SystemExit( + "compiled target rows differ from the arm's rows: " + f"{compiled[:2]} vs {list(names[:2])}" + ) + final = np.asarray(result.weights, dtype=np.float64) + out = args.out / args.arm + out.mkdir(parents=True, exist_ok=True) + np.savez_compressed(out / "weights.npz", weights=final, rows=rows) + release_estimates = matrix @ final + concept_estimates = concept_matrix @ final + metrics = { + "arm": args.arm, + "n_targets": int(len(rows)), + "epochs": int(args.epochs), + "settings": RELEASE_SETTINGS, + "initial_loss": float(result.initial_loss), + "final_loss": float(result.final_loss), + "seconds": time.time() - started, + "peak_rss_gb": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e9, + "platform": platform.platform(), + "cpus": os.cpu_count(), + "git_sha": os.environ.get("MICROCOSM_GIT_SHA", ""), + "inputs_manifest_sha256": hashlib.sha256( + (args.inputs / "MANIFEST.json").read_bytes() + ).hexdigest(), + "resolve_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "trajectory": trajectory, + "release_row_estimates": release_estimates.tolist(), + "concept_row_estimates": concept_estimates.tolist(), + "rebase_row_estimates": (rebase_matrix @ final).tolist(), + "own_rows": rows.tolist(), + "own_values": values.tolist(), + "own_estimates": (solve_matrix @ final).tolist(), + "own_loss_weights": loss_weights.tolist(), + } + (out / "metrics.json").write_text(json.dumps(metrics)) + print( + f"[{args.arm}] done: loss {result.initial_loss:.6f} -> " + f"{result.final_loss:.6f} in {metrics['seconds']:.0f}s", + flush=True, + ) + + +if __name__ == "__main__": + main() diff --git a/experiments/1228-net-income-less-loss/results/summary.json b/experiments/1228-net-income-less-loss/results/summary.json new file mode 100644 index 0000000000..db8d560d00 --- /dev/null +++ b/experiments/1228-net-income-less-loss/results/summary.json @@ -0,0 +1,2141 @@ +{ + "arms": { + "published": { + "own_surface": null, + "own_final_loss": null, + "seconds": null, + "common_rows": { + "n": 5382, + "within_10pct": 0.9392419175027871, + "mean_capped_error": 0.030671954752352796 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.9447066756574511, + "mean_capped_error": 0.02611096140291406 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.013708336438015325 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.004786171988310605 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9326923076923077, + "mean_capped_error": 0.03556621885364327 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09587898316943377 + }, + "count": { + "n": 56, + "within_10pct": 0.7321428571428571, + "mean_capped_error": 0.0869336563476515 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.02634991203200058 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0226157203827145 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8950064020486556, + "mean_capped_error": 0.04963768466675568 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.005154815167880631 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0009358205197866781 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5003016468374404 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.010861502280506776 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.013310321133514092 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.021600962328985254 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0012514028465502194 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.9090909090909091, + "mean_capped_error": 0.0439132617667858 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.0052195006170793 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0003712451388514287 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9444444444444444, + "mean_capped_error": 0.015595675212413949 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.017877472854352136 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.7924528301886793, + "mean_capped_error": 0.08391890450650809 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.004349847818168548 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9622641509433962, + "mean_capped_error": 0.023849333564630443 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.013464319164800426 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.0019586491206874143 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9161849710982659, + "mean_capped_error": 0.04811431349605853 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0010277452255966256 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.0030915283103182946 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 0.0005263360639137434 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.0013767768666579788 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 26300786.91539395, + "us_concept_exact": 31295989.887002863, + "states_release_concept": { + "n": 51, + "within_10pct": 0.7647058823529411, + "mean_capped_error": 0.0607635921486313 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.19607843137254902, + "mean_capped_error": 0.15645942058342163 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 488894776128.99677, + "us_concept_exact": 820989668707.7206, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9411764705882353, + "mean_capped_error": 0.05982328106114371 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.6356146216496612 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 9479193.161301691, + "us_concept_exact": 11489273.647078848, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9803921568627451, + "mean_capped_error": 0.008239930285000788 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0784313725490196, + "mean_capped_error": 0.21008219822300894 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 1128517678095.899, + "us_concept_exact": 1020542634029.1309, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9019607843137255, + "mean_capped_error": 0.02318281918623664 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.5686274509803921, + "mean_capped_error": 0.12766310243791923 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 9321528.97427554, + "us_concept_exact": 14090442.84911722, + "states_release_concept": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0009848472970694563 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.5085784363190154 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 253318942421.4058, + "us_concept_exact": 248787751422.50922, + "states_release_concept": { + "n": 51, + "within_10pct": 0.47058823529411764, + "mean_capped_error": 0.5307151541117274 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.3333333333333333, + "mean_capped_error": 0.580705596340832 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 836923406141.0552, + "neg_sum": -15933737433.32917, + "net": 820989668707.726, + "pos_count": 30643372.032014083, + "neg_count": 652617.8549889551, + "nonzero_count": 31295989.887003038 + }, + "schedule_c_nonsstb": { + "pos_sum": 488894776164.28094, + "neg_sum": -4303569740.482876, + "net": 484591206423.7981, + "pos_count": 26300786.915394105, + "neg_count": 311333.74241678946, + "nonzero_count": 26612120.657810893 + }, + "sstb_only": { + "pos_sum": 348028629976.77423, + "neg_sum": -11630167692.84629, + "net": 336398462283.9279, + "pos_count": 4420640.526723596, + "neg_count": 365923.7217944156, + "nonzero_count": 4786564.2485180115 + }, + "partnership_s_corp": { + "pos_sum": 1128517679151.5771, + "neg_sum": -107975045122.44385, + "net": 1020542634029.1333, + "pos_count": 9479193.161301674, + "neg_count": 2010080.4857771557, + "nonzero_count": 11489273.64707883 + }, + "rental_only": { + "pos_sum": 335168916158.93835, + "neg_sum": -86381164736.42978, + "net": 248787751422.50858, + "pos_count": 9070089.715544755, + "neg_count": 5020353.133572421, + "nonzero_count": 14090442.849117175 + }, + "rental_plus_farm_rent": { + "pos_sum": 345450943406.025, + "neg_sum": -92132001343.94482, + "net": 253318942062.0802, + "pos_count": 9321528.974275539, + "neg_count": 5182645.849439174, + "nonzero_count": 14504174.823714713 + }, + "farm_rent_only": { + "pos_sum": 13225380301.14615, + "neg_sum": -8694189661.574564, + "net": 4531190639.571587, + "pos_count": 442237.19948787196, + "neg_count": 290542.75199212704, + "nonzero_count": 732779.951479999 + }, + "tax_units": 172034458.48638445 + }, + "kish_ess": 77758.10900441968, + "chi_square_from_initial": 2.1881298165428027, + "max_ratio_to_published": 1.0, + "min_ratio_to_published": 1.0, + "correlation_with_published": 1.0 + }, + "release": { + "own_surface": { + "n": 5694, + "within_10pct": 0.934141201264489, + "mean_capped_error": 0.03529936550385233, + "weighted_capped_error": 0.028419576730484935, + "rows_312": { + "n": 312, + "within_10pct": 0.8397435897435898, + "mean_capped_error": 0.11564262716297395 + }, + "rows_312_by_measure": { + "partnership_scorp_income_amount": { + "n": 52, + "within_10pct": 0.9038461538461539, + "mean_capped_error": 0.023320138248756637 + }, + "partnership_scorp_income_returns": { + "n": 52, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.008221509483320824 + }, + "rental_royalty_income_amount": { + "n": 52, + "within_10pct": 0.46153846153846156, + "mean_capped_error": 0.5394571640411362 + }, + "rental_royalty_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.0010888826136124061 + }, + "schedule_c_income_amount": { + "n": 52, + "within_10pct": 0.9423076923076923, + "mean_capped_error": 0.05959167522265514 + }, + "schedule_c_income_returns": { + "n": 52, + "within_10pct": 0.75, + "mean_capped_error": 0.06217639336836244 + } + } + }, + "own_final_loss": 0.028419576730484935, + "seconds": 1314.747535943985, + "common_rows": { + "n": 5382, + "within_10pct": 0.9396135265700483, + "mean_capped_error": 0.03064178511781629 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.9447066756574511, + "mean_capped_error": 0.02607499254475389 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.013769658324178547 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.004593915130734745 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9326923076923077, + "mean_capped_error": 0.035465935544292046 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09592778936820183 + }, + "count": { + "n": 56, + "within_10pct": 0.7321428571428571, + "mean_capped_error": 0.08705073993276376 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.026298896614688365 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.022744707404218863 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8962868117797695, + "mean_capped_error": 0.049592415099011015 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.0051731378611796835 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0012008217416993015 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5000652883687807 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.010911652262527898 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.013254987869856218 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.02150793794214733 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0012726777407143868 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.9090909090909091, + "mean_capped_error": 0.04393657867856458 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.005510149531800581 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0006312685635856593 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9444444444444444, + "mean_capped_error": 0.015562940587337261 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.017812678478596577 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.7924528301886793, + "mean_capped_error": 0.08398954409997555 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.00422327290136127 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9716981132075472, + "mean_capped_error": 0.02384864423929114 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.01348791640248878 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.0019690755270115475 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9161849710982659, + "mean_capped_error": 0.04801447186391042 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0010066147968133762 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.0032495664292109252 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 0.00010769990955436877 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.0015443807110568134 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 26295799.159897976, + "us_concept_exact": 31289968.970477466, + "states_release_concept": { + "n": 51, + "within_10pct": 0.7647058823529411, + "mean_capped_error": 0.060773672488153394 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.19607843137254902, + "mean_capped_error": 0.15649278382802412 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 488979977388.8304, + "us_concept_exact": 820954710357.5177, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9411764705882353, + "mean_capped_error": 0.06006721407271851 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.635341723231795 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 9476975.60119726, + "us_concept_exact": 11485125.694225589, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9803921568627451, + "mean_capped_error": 0.008229177847167563 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0784313725490196, + "mean_capped_error": 0.2097752029874944 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 1128981889247.4282, + "us_concept_exact": 1021022818600.7543, + "states_release_concept": { + "n": 51, + "within_10pct": 0.9019607843137255, + "mean_capped_error": 0.02329261446345788 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.5686274509803921, + "mean_capped_error": 0.1272839113459558 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 9318329.851936102, + "us_concept_exact": 14087963.156447668, + "states_release_concept": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0009439899904904554 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.5084520879200642 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 253129223408.97986, + "us_concept_exact": 248630372192.89108, + "states_release_concept": { + "n": 51, + "within_10pct": 0.47058823529411764, + "mean_capped_error": 0.5304269123556683 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.3333333333333333, + "mean_capped_error": 0.5804471959768585 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 836897564747.8885, + "neg_sum": -15942854390.369133, + "net": 820954710357.5194, + "pos_count": 30637295.53793346, + "neg_count": 652673.432543968, + "nonzero_count": 31289968.97047743 + }, + "schedule_c_nonsstb": { + "pos_sum": 488979977425.3806, + "neg_sum": -4301564361.853214, + "net": 484678413063.5274, + "pos_count": 26295799.159898017, + "neg_count": 311378.7877498479, + "nonzero_count": 26607177.947647866 + }, + "sstb_only": { + "pos_sum": 347917587322.50793, + "neg_sum": -11641290028.515919, + "net": 336276297293.992, + "pos_count": 4419462.965977609, + "neg_count": 365861.0144020738, + "nonzero_count": 4785323.980379683 + }, + "partnership_s_corp": { + "pos_sum": 1128981890303.2515, + "neg_sum": -107959071702.4982, + "net": 1021022818600.7533, + "pos_count": 9476975.601197299, + "neg_count": 2008150.093028321, + "nonzero_count": 11485125.69422562 + }, + "rental_only": { + "pos_sum": 335035227671.0874, + "neg_sum": -86404855478.19797, + "net": 248630372192.88943, + "pos_count": 9067391.242437936, + "neg_count": 5020571.914009664, + "nonzero_count": 14087963.1564476 + }, + "rental_plus_farm_rent": { + "pos_sum": 345291078153.60815, + "neg_sum": -92161855104.38545, + "net": 253129223049.22272, + "pos_count": 9318329.851936135, + "neg_count": 5182856.221886316, + "nonzero_count": 14501186.073822452 + }, + "farm_rent_only": { + "pos_sum": 13188683648.979633, + "neg_sum": -8689832792.646439, + "net": 4498850856.333195, + "pos_count": 441794.9868180575, + "neg_count": 290154.4428066949, + "nonzero_count": 731949.4296247524 + }, + "tax_units": 172041973.64516407 + }, + "kish_ess": 77794.03493503302, + "chi_square_from_initial": 2.186732258128342, + "max_ratio_to_published": 1.2504528492253517, + "min_ratio_to_published": 0.7855713373068746, + "correlation_with_published": 0.9998780972433269 + }, + "concept_exact": { + "own_surface": { + "n": 5694, + "within_10pct": 0.9388830347734457, + "mean_capped_error": 0.03164366258004792, + "weighted_capped_error": 0.0264063957697224, + "rows_312": { + "n": 312, + "within_10pct": 0.9262820512820513, + "mean_capped_error": 0.04720747594386884 + }, + "rows_312_by_measure": { + "partnership_scorp_income_amount": { + "n": 52, + "within_10pct": 0.8653846153846154, + "mean_capped_error": 0.028883996442526278 + }, + "partnership_scorp_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.0006390508073137742 + }, + "rental_royalty_income_amount": { + "n": 52, + "within_10pct": 0.8269230769230769, + "mean_capped_error": 0.1603230193528389 + }, + "rental_royalty_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.0012057894819960864 + }, + "schedule_c_income_amount": { + "n": 52, + "within_10pct": 0.9230769230769231, + "mean_capped_error": 0.0796010739995202 + }, + "schedule_c_income_returns": { + "n": 52, + "within_10pct": 0.9423076923076923, + "mean_capped_error": 0.01259192557901783 + } + } + }, + "own_final_loss": 0.0264063957697224, + "seconds": 1558.8121206760406, + "common_rows": { + "n": 5382, + "within_10pct": 0.9396135265700483, + "mean_capped_error": 0.030741412529971347 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.9440323668240054, + "mean_capped_error": 0.02607568753445054 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.010904197587764846 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0036052334092457083 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.034226396595080685 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09109925234109437 + }, + "count": { + "n": 56, + "within_10pct": 0.7321428571428571, + "mean_capped_error": 0.08418714100797045 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.023674542364517125 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.018840278291496468 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8975672215108835, + "mean_capped_error": 0.05005965123013537 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.005204105689222395 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0009279681696786606 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5000646071918048 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.009827796120916967 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.01362232148396899 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.019371396875947505 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0012899603468806559 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.8909090909090909, + "mean_capped_error": 0.04645260572676645 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.004868626949900442 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.00035132536074768226 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9444444444444444, + "mean_capped_error": 0.016097656987760754 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.01677932581115465 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.8018867924528302, + "mean_capped_error": 0.08083348356126543 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0032387306139560106 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9622641509433962, + "mean_capped_error": 0.024149769427090677 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9903846153846154, + "mean_capped_error": 0.010599831815436804 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.0024466417052489346 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9142581888246628, + "mean_capped_error": 0.049668974340546054 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0011221610304182265 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.0031690911297172874 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 0.00045554322728548507 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.001559968378187311 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 21097308.06794628, + "us_concept_exact": 29368663.80894041, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.28878027173319204 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.9411764705882353, + "mean_capped_error": 0.012201901204235446 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 436541144438.9505, + "us_concept_exact": 506922149127.2931, + "states_release_concept": { + "n": 51, + "within_10pct": 0.21568627450980393, + "mean_capped_error": 0.20976920148774575 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.9215686274509803, + "mean_capped_error": 0.07972405668147982 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 7262592.469218501, + "us_concept_exact": 9522292.71397138, + "states_release_concept": { + "n": 51, + "within_10pct": 0.09803921568627451, + "mean_capped_error": 0.23745014602284834 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.000591070336106078 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 1248728293657.6807, + "us_concept_exact": 1119829818707.648, + "states_release_concept": { + "n": 51, + "within_10pct": 0.5098039215686274, + "mean_capped_error": 0.12091337897545473 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.8627450980392157, + "mean_capped_error": 0.028810547174107477 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 6146224.56082022, + "us_concept_exact": 9320362.054195859, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.3258843907753155 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0010674291015424786 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 111266691574.44617, + "us_concept_exact": 98375781508.66736, + "states_release_concept": { + "n": 51, + "within_10pct": 0.3333333333333333, + "mean_capped_error": 0.3613558796914522 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.8235294117647058, + "mean_capped_error": 0.16297079722510247 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 532174277507.26306, + "neg_sum": -25252128379.97207, + "net": 506922149127.291, + "pos_count": 28075258.461739875, + "neg_count": 1293405.3472004451, + "nonzero_count": 29368663.80894032 + }, + "schedule_c_nonsstb": { + "pos_sum": 436541144279.8536, + "neg_sum": -9372107430.03218, + "net": 427169036849.8214, + "pos_count": 21097308.06794647, + "neg_count": 855021.9108168985, + "nonzero_count": 21952329.978763368 + }, + "sstb_only": { + "pos_sum": 95633133227.40947, + "neg_sum": -15880020949.939894, + "net": 79753112277.46957, + "pos_count": 7037689.125085464, + "neg_count": 473861.43453633785, + "nonzero_count": 7511550.559621802 + }, + "partnership_s_corp": { + "pos_sum": 1248728295308.59, + "neg_sum": -128898476600.9437, + "net": 1119829818707.6465, + "pos_count": 7262592.469218479, + "neg_count": 2259700.244752976, + "nonzero_count": 9522292.713971455 + }, + "rental_only": { + "pos_sum": 182663737698.69913, + "neg_sum": -84287956190.0322, + "net": 98375781508.66693, + "pos_count": 5750779.802142654, + "neg_count": 3569582.2520532925, + "nonzero_count": 9320362.054195946 + }, + "rental_plus_farm_rent": { + "pos_sum": 199502863673.60962, + "neg_sum": -88236172932.60324, + "net": 111266690741.00638, + "pos_count": 6146224.560820239, + "neg_count": 3703174.3814952965, + "nonzero_count": 9849398.942315536 + }, + "farm_rent_only": { + "pos_sum": 20079006757.558445, + "neg_sum": -7188097525.218986, + "net": 12890909232.339458, + "pos_count": 548511.1120325762, + "neg_count": 252812.77444826922, + "nonzero_count": 801323.8864808455 + }, + "tax_units": 171798780.41966832 + }, + "kish_ess": 77306.34727637228, + "chi_square_from_initial": 2.1989173208342634, + "max_ratio_to_published": 187.12452832058665, + "min_ratio_to_published": 0.0027996439820645188, + "correlation_with_published": 0.9173741192580671 + }, + "concept_exact_no_business_count": { + "own_surface": { + "n": 5642, + "within_10pct": 0.9404466501240695, + "mean_capped_error": 0.031323200173138596, + "weighted_capped_error": 0.025832973648824337, + "rows_312": { + "n": 260, + "within_10pct": 0.9307692307692308, + "mean_capped_error": 0.050001055259384634 + }, + "rows_312_by_measure": { + "partnership_scorp_income_amount": { + "n": 52, + "within_10pct": 0.8653846153846154, + "mean_capped_error": 0.028530542772320446 + }, + "partnership_scorp_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.000684900228991228 + }, + "rental_royalty_income_amount": { + "n": 52, + "within_10pct": 0.8461538461538461, + "mean_capped_error": 0.1593774574958563 + }, + "rental_royalty_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.0014704569866811333 + }, + "schedule_c_income_amount": { + "n": 52, + "within_10pct": 0.9423076923076923, + "mean_capped_error": 0.059941918813074056 + } + } + }, + "own_final_loss": 0.025832973648824337, + "seconds": 1200.0367929935455, + "common_rows": { + "n": 5382, + "within_10pct": 0.9409141583054627, + "mean_capped_error": 0.03042088833322333 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.9447066756574511, + "mean_capped_error": 0.026529874659650177 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.010574763973047635 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.003085691130915091 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.03168180587198839 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09178930934524338 + }, + "count": { + "n": 56, + "within_10pct": 0.75, + "mean_capped_error": 0.0831646723093811 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.022714403445755794 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.017400626020309076 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8975672215108835, + "mean_capped_error": 0.049082028571479516 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.004713594982796336 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.001987146973107481 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5002390646810833 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.010808419058344136 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.012586838072405852 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.019279363433851986 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0013007420016721324 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.9090909090909091, + "mean_capped_error": 0.04449095632812949 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.004917012080573806 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0006176716667074328 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9444444444444444, + "mean_capped_error": 0.016587755295978918 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.016275365231536054 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.8018867924528302, + "mean_capped_error": 0.080191155286596 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.003283691205816094 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9716981132075472, + "mean_capped_error": 0.023536825658785326 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9903846153846154, + "mean_capped_error": 0.011485702657938988 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.0019159759978313123 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9171483622350675, + "mean_capped_error": 0.04882282902706604 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.001193572563453933 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.0032011161553566 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 0.0004261590438591849 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.0013449864823105277 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 10446931.684720758, + "us_concept_exact": 14401263.986489767, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.6270138550401406 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.4855405722321898 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 422542007444.54694, + "us_concept_exact": 493450521143.3887, + "states_release_concept": { + "n": 51, + "within_10pct": 0.27450980392156865, + "mean_capped_error": 0.1892787054302032 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.9411764705882353, + "mean_capped_error": 0.06023872356878862 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 7247623.202793523, + "us_concept_exact": 9523957.756858235, + "states_release_concept": { + "n": 51, + "within_10pct": 0.11764705882352941, + "mean_capped_error": 0.23844306238491975 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0006412367608047584 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 1252526622019.3538, + "us_concept_exact": 1120216282645.756, + "states_release_concept": { + "n": 51, + "within_10pct": 0.47058823529411764, + "mean_capped_error": 0.12354065050484782 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.8627450980392157, + "mean_capped_error": 0.02845670909142064 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 6140638.852181552, + "us_concept_exact": 9323595.021850869, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.3260537135587229 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0013440313721226156 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 113142552295.48412, + "us_concept_exact": 99319784558.6815, + "states_release_concept": { + "n": 51, + "within_10pct": 0.3137254901960784, + "mean_capped_error": 0.3551583848245998 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.8431372549019608, + "mean_capped_error": 0.16181378251448247 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 513000301045.8628, + "neg_sum": -19549779902.473724, + "net": 493450521143.38904, + "pos_count": 13650600.58147629, + "neg_count": 750663.4050134177, + "nonzero_count": 14401263.986489708 + }, + "schedule_c_nonsstb": { + "pos_sum": 422542007032.52625, + "neg_sum": -5772391174.62831, + "net": 416769615857.89795, + "pos_count": 10446931.68472077, + "neg_count": 389819.73932549707, + "nonzero_count": 10836751.424046267 + }, + "sstb_only": { + "pos_sum": 90458294013.33652, + "neg_sum": -13777388727.845417, + "net": 76680905285.4911, + "pos_count": 3238309.7180729257, + "neg_count": 390560.49617484875, + "nonzero_count": 3628870.2142477743 + }, + "partnership_s_corp": { + "pos_sum": 1252526623709.004, + "neg_sum": -132310341063.25182, + "net": 1120216282645.7522, + "pos_count": 7247623.202793542, + "neg_count": 2276334.5540646818, + "nonzero_count": 9523957.756858224 + }, + "rental_only": { + "pos_sum": 185437626581.64838, + "neg_sum": -86117842022.96698, + "net": 99319784558.6814, + "pos_count": 5753234.4943641145, + "neg_count": 3570360.5274867145, + "nonzero_count": 9323595.021850828 + }, + "rental_plus_farm_rent": { + "pos_sum": 202601658081.3424, + "neg_sum": -89459106543.11453, + "net": 113142551538.22787, + "pos_count": 6140638.852181494, + "neg_count": 3704029.2692447496, + "nonzero_count": 9844668.121426243 + }, + "farm_rent_only": { + "pos_sum": 20513634304.56611, + "neg_sum": -6690867325.019619, + "net": 13822766979.54649, + "pos_count": 538518.496693682, + "neg_count": 258571.61566575454, + "nonzero_count": 797090.1123594366 + }, + "tax_units": 171453411.49112204 + }, + "kish_ess": 82769.38427576036, + "chi_square_from_initial": 1.9969664973829373, + "max_ratio_to_published": 186.25197277796858, + "min_ratio_to_published": 0.0020738144761323915, + "correlation_with_published": 0.8495061593350868 + }, + "drop": { + "own_surface": { + "n": 5434, + "within_10pct": 0.9433198380566802, + "mean_capped_error": 0.029588555266496584, + "weighted_capped_error": 0.02414134084794914, + "rows_312": { + "n": 52, + "within_10pct": 0.8461538461538461, + "mean_capped_error": 0.14401425021136027 + }, + "rows_312_by_measure": { + "rental_royalty_income_amount": { + "n": 52, + "within_10pct": 0.8461538461538461, + "mean_capped_error": 0.14401425021136027 + } + } + }, + "own_final_loss": 0.02414134084794914, + "seconds": 1327.6631488800049, + "common_rows": { + "n": 5382, + "within_10pct": 0.9442586399108138, + "mean_capped_error": 0.0284829929964979 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.9501011463250169, + "mean_capped_error": 0.02473938074825016 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.011291184144557436 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0031072439607380147 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.03050141917914439 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09301644230735817 + }, + "count": { + "n": 56, + "within_10pct": 0.7678571428571429, + "mean_capped_error": 0.07969413734883017 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.02163188704911867 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.020514266626268585 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8950064020486556, + "mean_capped_error": 0.05034426166635014 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.0030517663934444272 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0015511247300519912 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5004930092510557 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.012167779249265043 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.011184960830172093 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.02280387142337408 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0012212386144266822 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.9090909090909091, + "mean_capped_error": 0.039132420333477 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.004343737060606912 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.00016722962812168495 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9537037037037037, + "mean_capped_error": 0.014512006728752636 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.015950702266462176 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.839622641509434, + "mean_capped_error": 0.065558611555918 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0023703100230219915 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9811320754716981, + "mean_capped_error": 0.022688841278055212 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.012103793322479297 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.0017192305259913546 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9229287090558767, + "mean_capped_error": 0.043133486881561364 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.001058140387657035 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.003109099500562069 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 4.577865852951706e-05 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.001689537783649304 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 14705482.92440955, + "us_concept_exact": 19545853.955323044, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.4854461020585187 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0392156862745098, + "mean_capped_error": 0.316764832844319 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 1013049234569.7578, + "us_concept_exact": 1320738618653.3716, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.8280560883734137 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.9117405017268697 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 5125504.830486556, + "us_concept_exact": 7448437.801826627, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.48844006848619353 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.17647058823529413, + "mean_capped_error": 0.2585143607948064 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 986909097087.566, + "us_concept_exact": 811085770783.4054, + "states_release_concept": { + "n": 51, + "within_10pct": 0.058823529411764705, + "mean_capped_error": 0.3601264397789437 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.09803921568627451, + "mean_capped_error": 0.45242757009616796 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 8952775.496664645, + "us_concept_exact": 14248242.492186503, + "states_release_concept": { + "n": 51, + "within_10pct": 0.5098039215686274, + "mean_capped_error": 0.1164169947667931 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0784313725490196, + "mean_capped_error": 0.511207759456351 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 96244798433.67328, + "us_concept_exact": 95051311739.32632, + "states_release_concept": { + "n": 51, + "within_10pct": 0.8431372549019608, + "mean_capped_error": 0.1467777268818779 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.49019607843137253, + "mean_capped_error": 0.2668533600129238 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 1332854434606.9531, + "neg_sum": -12115815953.579754, + "net": 1320738618653.3733, + "pos_count": 19027805.645274416, + "neg_count": 518048.3100485289, + "nonzero_count": 19545853.955322944 + }, + "schedule_c_nonsstb": { + "pos_sum": 1013049234094.7592, + "neg_sum": -3823314218.207496, + "net": 1009225919876.5516, + "pos_count": 14705482.924409505, + "neg_count": 313331.05016851786, + "nonzero_count": 15018813.974578023 + }, + "sstb_only": { + "pos_sum": 319805200512.19366, + "neg_sum": -8292501735.372258, + "net": 311512698776.8214, + "pos_count": 4361342.790689465, + "neg_count": 219729.21551480342, + "nonzero_count": 4581072.006204268 + }, + "partnership_s_corp": { + "pos_sum": 986909098527.1526, + "neg_sum": -175823327743.75406, + "net": 811085770783.3986, + "pos_count": 5125504.830486545, + "neg_count": 2322932.9713401077, + "nonzero_count": 7448437.801826653 + }, + "rental_only": { + "pos_sum": 208807231655.78845, + "neg_sum": -113755919916.461, + "net": 95051311739.32745, + "pos_count": 8707768.69404723, + "neg_count": 5540473.79813936, + "nonzero_count": 14248242.49218659 + }, + "rental_plus_farm_rent": { + "pos_sum": 216072084141.34116, + "neg_sum": -119827286324.72166, + "net": 96244797816.61949, + "pos_count": 8952775.496664695, + "neg_count": 5713390.034520076, + "nonzero_count": 14666165.53118477 + }, + "farm_rent_only": { + "pos_sum": 10085180185.960823, + "neg_sum": -8891694108.668758, + "net": 1193486077.2920647, + "pos_count": 416675.3343875678, + "neg_count": 298485.2804042252, + "nonzero_count": 715160.614791793 + }, + "tax_units": 170803381.19051546 + }, + "kish_ess": 84834.25503378721, + "chi_square_from_initial": 1.9164335353108426, + "max_ratio_to_published": 168.73463300089801, + "min_ratio_to_published": 0.0044356778480312276, + "correlation_with_published": 0.8530574344564628 + }, + "rebase_positive_part": { + "own_surface": { + "n": 5694, + "within_10pct": 0.9406392694063926, + "mean_capped_error": 0.031124895924437537, + "weighted_capped_error": 0.025520896184999003, + "rows_312": { + "n": 312, + "within_10pct": 0.9294871794871795, + "mean_capped_error": 0.048663839443980575 + }, + "rows_312_by_measure": { + "partnership_scorp_income_amount": { + "n": 52, + "within_10pct": 0.8653846153846154, + "mean_capped_error": 0.029121361073805672 + }, + "partnership_scorp_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.0007174959400305368 + }, + "rental_royalty_income_amount": { + "n": 52, + "within_10pct": 0.7692307692307693, + "mean_capped_error": 0.20061369170231247 + }, + "rental_royalty_income_returns": { + "n": 52, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.020449154676961213 + }, + "schedule_c_income_amount": { + "n": 52, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.04022053983444267 + }, + "schedule_c_income_returns": { + "n": 52, + "within_10pct": 1.0, + "mean_capped_error": 0.000860793436330879 + } + } + }, + "own_final_loss": 0.025520896184999003, + "seconds": 1360.8871195316315, + "common_rows": { + "n": 5382, + "within_10pct": 0.9412857673727238, + "mean_capped_error": 0.03010814557547852 + }, + "common_rows_by_variable": { + "": { + "n": 1483, + "within_10pct": 0.947403910991234, + "mean_capped_error": 0.025680003134721487 + }, + "adjusted_gross_income": { + "n": 57, + "within_10pct": 1.0, + "mean_capped_error": 0.010869756310194949 + }, + "assigned_aca_ptc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0031830344921850028 + }, + "capital_gains_gross": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.021545397499507925 + }, + "charitable_deduction": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.09389530298475145 + }, + "count": { + "n": 56, + "within_10pct": 0.75, + "mean_capped_error": 0.08117755116568282 + }, + "ctc": { + "n": 104, + "within_10pct": 0.9615384615384616, + "mean_capped_error": 0.022104357248142234 + }, + "deductible_mortgage_interest": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.015159476349723526 + }, + "eitc": { + "n": 781, + "within_10pct": 0.8962868117797695, + "mean_capped_error": 0.04905632374082366 + }, + "employment_income": { + "n": 106, + "within_10pct": 0.9905660377358491, + "mean_capped_error": 0.0045833346660445985 + }, + "estate_income": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.0002535638551836992 + }, + "estate_losses": { + "n": 2, + "within_10pct": 0.5, + "mean_capped_error": 0.5004057178816248 + }, + "income_tax": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.010053325465220626 + }, + "income_tax_before_credits": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.013096228500835659 + }, + "interest_deduction": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.018109815462155773 + }, + "ira_distributions": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0013584711903109092 + }, + "itemized_taxable_income_deductions": { + "n": 55, + "within_10pct": 0.8909090909090909, + "mean_capped_error": 0.04459104451985205 + }, + "medical_expense_deduction": { + "n": 104, + "within_10pct": 0.9807692307692307, + "mean_capped_error": 0.004945892430593925 + }, + "non_sch_d_capital_gains": { + "n": 2, + "within_10pct": 1.0, + "mean_capped_error": 0.00016985252287565246 + }, + "ordinary_dividend_income": { + "n": 108, + "within_10pct": 0.9537037037037037, + "mean_capped_error": 0.016346793255598208 + }, + "qualified_dividends": { + "n": 104, + "within_10pct": 0.9519230769230769, + "mean_capped_error": 0.016374502025037138 + }, + "real_estate_taxes": { + "n": 106, + "within_10pct": 0.7924528301886793, + "mean_capped_error": 0.08014987554314529 + }, + "refundable_ctc": { + "n": 104, + "within_10pct": 1.0, + "mean_capped_error": 0.0029265620560569375 + }, + "salt_deduction": { + "n": 106, + "within_10pct": 0.9622641509433962, + "mean_capped_error": 0.02346409117959429 + }, + "tax_exempt_interest_income": { + "n": 104, + "within_10pct": 0.9903846153846154, + "mean_capped_error": 0.010888368956829044 + }, + "taxable_income": { + "n": 107, + "within_10pct": 1.0, + "mean_capped_error": 0.002506899924440365 + }, + "taxable_interest_income": { + "n": 1038, + "within_10pct": 0.9171483622350675, + "mean_capped_error": 0.049660796430057455 + }, + "taxable_pension_income": { + "n": 108, + "within_10pct": 1.0, + "mean_capped_error": 0.0011393456822805243 + }, + "taxable_social_security": { + "n": 108, + "within_10pct": 0.9907407407407407, + "mean_capped_error": 0.0032063208613954198 + }, + "tip_income": { + "n": 1, + "within_10pct": 1.0, + "mean_capped_error": 0.0005644290296060586 + }, + "unemployment_compensation": { + "n": 106, + "within_10pct": 1.0, + "mean_capped_error": 0.001624817581294069 + } + }, + "rows_312": { + "schedule_c_income_returns": { + "us_target": 30354680.0, + "us_release_concept": 16765414.239372674, + "us_concept_exact": 22493042.079613097, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.44619656914319605 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.255240628173601 + } + }, + "schedule_c_income_amount": { + "us_target": 472289642161.9491, + "us_release_concept": 524427962672.9276, + "us_concept_exact": 613289949110.9703, + "states_release_concept": { + "n": 51, + "within_10pct": 0.49019607843137253, + "mean_capped_error": 0.13453075480353566 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.2910165220758344 + } + }, + "partnership_scorp_income_returns": { + "us_target": 9551770.0, + "us_release_concept": 7268318.530816951, + "us_concept_exact": 9523299.47472488, + "states_release_concept": { + "n": 51, + "within_10pct": 0.11764705882352941, + "mean_capped_error": 0.23816200449975694 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 1.0, + "mean_capped_error": 0.0006731202839592062 + } + }, + "partnership_scorp_income_amount": { + "us_target": 1157602275777.119, + "us_release_concept": 1247921767076.7927, + "us_concept_exact": 1119446019405.733, + "states_release_concept": { + "n": 51, + "within_10pct": 0.5098039215686274, + "mean_capped_error": 0.11793068670235751 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.8627450980392157, + "mean_capped_error": 0.029046065096675455 + } + }, + "rental_royalty_income_returns": { + "us_target": 9398010.0, + "us_release_concept": 6349815.02144129, + "us_concept_exact": 10669611.298720777, + "states_release_concept": { + "n": 51, + "within_10pct": 0.0, + "mean_capped_error": 0.3283768515217562 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.43137254901960786, + "mean_capped_error": 0.1733655485579856 + } + }, + "rental_royalty_income_amount": { + "us_target": 95949567370.41321, + "us_release_concept": 119793997239.833, + "us_concept_exact": 118216927758.00688, + "states_release_concept": { + "n": 51, + "within_10pct": 0.13725490196078433, + "mean_capped_error": 0.5175850176851029 + }, + "states_concept_exact": { + "n": 51, + "within_10pct": 0.1568627450980392, + "mean_capped_error": 0.5054651608700602 + } + } + }, + "model_totals": { + "schedule_c_total": { + "pos_sum": 627729978529.0084, + "neg_sum": -14440029418.037537, + "net": 613289949110.971, + "pos_count": 21893495.858489927, + "neg_count": 599546.2211232035, + "nonzero_count": 22493042.07961313 + }, + "schedule_c_nonsstb": { + "pos_sum": 524427962388.01794, + "neg_sum": -3459685483.7158675, + "net": 520968276904.30206, + "pos_count": 16765414.239372676, + "neg_count": 306384.8612301914, + "nonzero_count": 17071799.10060287 + }, + "sstb_only": { + "pos_sum": 103302016140.99051, + "neg_sum": -10980343934.32167, + "net": 92321672206.66884, + "pos_count": 5173156.727650797, + "neg_count": 314370.05891128955, + "nonzero_count": 5487526.786562087 + }, + "partnership_s_corp": { + "pos_sum": 1247921768743.3425, + "neg_sum": -128475749337.61453, + "net": 1119446019405.728, + "pos_count": 7268318.530816973, + "neg_count": 2254980.943907943, + "nonzero_count": 9523299.474724917 + }, + "rental_only": { + "pos_sum": 186504895565.0938, + "neg_sum": -68287967807.08555, + "net": 118216927758.00827, + "pos_count": 6185587.588570689, + "neg_count": 4484023.710150097, + "nonzero_count": 10669611.298720784 + }, + "rental_plus_farm_rent": { + "pos_sum": 192889949831.28137, + "neg_sum": -73095953101.79054, + "net": 119793996729.49083, + "pos_count": 6349815.021441329, + "neg_count": 4657982.993235243, + "nonzero_count": 11007798.01467657 + }, + "farm_rent_only": { + "pos_sum": 8840170100.950193, + "neg_sum": -7263101129.467646, + "net": 1577068971.4825478, + "pos_count": 305834.6418223487, + "neg_count": 265774.60906043684, + "nonzero_count": 571609.2508827855 + }, + "tax_units": 171169825.5187798 + }, + "kish_ess": 81036.17721739199, + "chi_square_from_initial": 2.052086504679212, + "max_ratio_to_published": 186.84931341249253, + "min_ratio_to_published": 0.002040609286800876, + "correlation_with_published": 0.9056528913504068 + } + }, + "irs_ty2022_aged": { + "schedule_c_total": { + "pos_count": 21969832, + "neg_count": 8386569, + "pos_sum": 608980837381.8507, + "neg_sum": -149198909095.43423 + }, + "partnership_s_corp": { + "pos_count_upper": 7339387, + "pos_sum_upper": 1456265836672.1682, + "net": 1155309642347.4458 + }, + "rental_plus_farm_rent": { + "pos_count": 6380131, + "neg_count": 3313901, + "pos_sum": 166796498138.73874, + "neg_sum": -62406494031.38192 + } + } +} \ No newline at end of file diff --git a/experiments/1228-net-income-less-loss/results/summary.md b/experiments/1228-net-income-less-loss/results/summary.md new file mode 100644 index 0000000000..c659321af3 --- /dev/null +++ b/experiments/1228-net-income-less-loss/results/summary.md @@ -0,0 +1,98 @@ +# #1228 re-solve summary + +| Arm | Own loss | Own rows within 10% | Own 312 rows within 10% | Common rows within 10% | Common mean capped error | Kish ESS | Corr. with published | +|---|---:|---:|---:|---:|---:|---:|---:| +| published | | | | 0.9392 | 0.03067 | 77,758 | 1.0000 | +| release | 0.028420 | 0.9341 | 0.840 | 0.9396 | 0.03064 | 77,794 | 0.9999 | +| concept_exact | 0.026406 | 0.9389 | 0.926 | 0.9396 | 0.03074 | 77,306 | 0.9174 | +| concept_exact_no_business_count | 0.025833 | 0.9404 | 0.931 | 0.9409 | 0.03042 | 82,769 | 0.8495 | +| drop | 0.024141 | 0.9433 | 0.846 | 0.9443 | 0.02848 | 84,834 | 0.8531 | +| rebase_positive_part | 0.025521 | 0.9406 | 0.929 | 0.9413 | 0.03011 | 81,036 | 0.9057 | + +## Model totals (2024 weights; $B and millions) + +| Arm | Concept | Positive $B | Negative $B | Net $B | Positive M | Negative M | Nonzero M | +|---|---|---:|---:|---:|---:|---:|---:| +| published | schedule_c_total | 836.9 | -15.9 | 821.0 | 30.64 | 0.65 | 31.30 | +| published | schedule_c_nonsstb | 488.9 | -4.3 | 484.6 | 26.30 | 0.31 | 26.61 | +| published | sstb_only | 348.0 | -11.6 | 336.4 | 4.42 | 0.37 | 4.79 | +| published | partnership_s_corp | 1,128.5 | -108.0 | 1,020.5 | 9.48 | 2.01 | 11.49 | +| published | rental_only | 335.2 | -86.4 | 248.8 | 9.07 | 5.02 | 14.09 | +| published | rental_plus_farm_rent | 345.5 | -92.1 | 253.3 | 9.32 | 5.18 | 14.50 | +| published | farm_rent_only | 13.2 | -8.7 | 4.5 | 0.44 | 0.29 | 0.73 | +| release | schedule_c_total | 836.9 | -15.9 | 821.0 | 30.64 | 0.65 | 31.29 | +| release | schedule_c_nonsstb | 489.0 | -4.3 | 484.7 | 26.30 | 0.31 | 26.61 | +| release | sstb_only | 347.9 | -11.6 | 336.3 | 4.42 | 0.37 | 4.79 | +| release | partnership_s_corp | 1,129.0 | -108.0 | 1,021.0 | 9.48 | 2.01 | 11.49 | +| release | rental_only | 335.0 | -86.4 | 248.6 | 9.07 | 5.02 | 14.09 | +| release | rental_plus_farm_rent | 345.3 | -92.2 | 253.1 | 9.32 | 5.18 | 14.50 | +| release | farm_rent_only | 13.2 | -8.7 | 4.5 | 0.44 | 0.29 | 0.73 | +| concept_exact | schedule_c_total | 532.2 | -25.3 | 506.9 | 28.08 | 1.29 | 29.37 | +| concept_exact | schedule_c_nonsstb | 436.5 | -9.4 | 427.2 | 21.10 | 0.86 | 21.95 | +| concept_exact | sstb_only | 95.6 | -15.9 | 79.8 | 7.04 | 0.47 | 7.51 | +| concept_exact | partnership_s_corp | 1,248.7 | -128.9 | 1,119.8 | 7.26 | 2.26 | 9.52 | +| concept_exact | rental_only | 182.7 | -84.3 | 98.4 | 5.75 | 3.57 | 9.32 | +| concept_exact | rental_plus_farm_rent | 199.5 | -88.2 | 111.3 | 6.15 | 3.70 | 9.85 | +| concept_exact | farm_rent_only | 20.1 | -7.2 | 12.9 | 0.55 | 0.25 | 0.80 | +| concept_exact_no_business_count | schedule_c_total | 513.0 | -19.5 | 493.5 | 13.65 | 0.75 | 14.40 | +| concept_exact_no_business_count | schedule_c_nonsstb | 422.5 | -5.8 | 416.8 | 10.45 | 0.39 | 10.84 | +| concept_exact_no_business_count | sstb_only | 90.5 | -13.8 | 76.7 | 3.24 | 0.39 | 3.63 | +| concept_exact_no_business_count | partnership_s_corp | 1,252.5 | -132.3 | 1,120.2 | 7.25 | 2.28 | 9.52 | +| concept_exact_no_business_count | rental_only | 185.4 | -86.1 | 99.3 | 5.75 | 3.57 | 9.32 | +| concept_exact_no_business_count | rental_plus_farm_rent | 202.6 | -89.5 | 113.1 | 6.14 | 3.70 | 9.84 | +| concept_exact_no_business_count | farm_rent_only | 20.5 | -6.7 | 13.8 | 0.54 | 0.26 | 0.80 | +| drop | schedule_c_total | 1,332.9 | -12.1 | 1,320.7 | 19.03 | 0.52 | 19.55 | +| drop | schedule_c_nonsstb | 1,013.0 | -3.8 | 1,009.2 | 14.71 | 0.31 | 15.02 | +| drop | sstb_only | 319.8 | -8.3 | 311.5 | 4.36 | 0.22 | 4.58 | +| drop | partnership_s_corp | 986.9 | -175.8 | 811.1 | 5.13 | 2.32 | 7.45 | +| drop | rental_only | 208.8 | -113.8 | 95.1 | 8.71 | 5.54 | 14.25 | +| drop | rental_plus_farm_rent | 216.1 | -119.8 | 96.2 | 8.95 | 5.71 | 14.67 | +| drop | farm_rent_only | 10.1 | -8.9 | 1.2 | 0.42 | 0.30 | 0.72 | +| rebase_positive_part | schedule_c_total | 627.7 | -14.4 | 613.3 | 21.89 | 0.60 | 22.49 | +| rebase_positive_part | schedule_c_nonsstb | 524.4 | -3.5 | 521.0 | 16.77 | 0.31 | 17.07 | +| rebase_positive_part | sstb_only | 103.3 | -11.0 | 92.3 | 5.17 | 0.31 | 5.49 | +| rebase_positive_part | partnership_s_corp | 1,247.9 | -128.5 | 1,119.4 | 7.27 | 2.25 | 9.52 | +| rebase_positive_part | rental_only | 186.5 | -68.3 | 118.2 | 6.19 | 4.48 | 10.67 | +| rebase_positive_part | rental_plus_farm_rent | 192.9 | -73.1 | 119.8 | 6.35 | 4.66 | 11.01 | +| rebase_positive_part | farm_rent_only | 8.8 | -7.3 | 1.6 | 0.31 | 0.27 | 0.57 | + +## The 312 rows: US value and state fit on each concept + +| Arm | Measure | US target | US, release concept | US, #1228 concept | States within 10%, release concept | States within 10%, #1228 concept | +|---|---|---:|---:|---:|---:|---:| +| published | schedule_c_income_returns | 30,354,680 | 26,300,787 | 31,295,990 | 0.76 | 0.20 | +| published | schedule_c_income_amount | 472,289,642,162 | 488,894,776,129 | 820,989,668,708 | 0.94 | 0.00 | +| published | partnership_scorp_income_returns | 9,551,770 | 9,479,193 | 11,489,274 | 0.98 | 0.08 | +| published | partnership_scorp_income_amount | 1,157,602,275,777 | 1,128,517,678,096 | 1,020,542,634,029 | 0.90 | 0.57 | +| published | rental_royalty_income_returns | 9,398,010 | 9,321,529 | 14,090,443 | 1.00 | 0.00 | +| published | rental_royalty_income_amount | 95,949,567,370 | 253,318,942,421 | 248,787,751,423 | 0.47 | 0.33 | +| release | schedule_c_income_returns | 30,354,680 | 26,295,799 | 31,289,969 | 0.76 | 0.20 | +| release | schedule_c_income_amount | 472,289,642,162 | 488,979,977,389 | 820,954,710,358 | 0.94 | 0.00 | +| release | partnership_scorp_income_returns | 9,551,770 | 9,476,976 | 11,485,126 | 0.98 | 0.08 | +| release | partnership_scorp_income_amount | 1,157,602,275,777 | 1,128,981,889,247 | 1,021,022,818,601 | 0.90 | 0.57 | +| release | rental_royalty_income_returns | 9,398,010 | 9,318,330 | 14,087,963 | 1.00 | 0.00 | +| release | rental_royalty_income_amount | 95,949,567,370 | 253,129,223,409 | 248,630,372,193 | 0.47 | 0.33 | +| concept_exact | schedule_c_income_returns | 30,354,680 | 21,097,308 | 29,368,664 | 0.00 | 0.94 | +| concept_exact | schedule_c_income_amount | 472,289,642,162 | 436,541,144,439 | 506,922,149,127 | 0.22 | 0.92 | +| concept_exact | partnership_scorp_income_returns | 9,551,770 | 7,262,592 | 9,522,293 | 0.10 | 1.00 | +| concept_exact | partnership_scorp_income_amount | 1,157,602,275,777 | 1,248,728,293,658 | 1,119,829,818,708 | 0.51 | 0.86 | +| concept_exact | rental_royalty_income_returns | 9,398,010 | 6,146,225 | 9,320,362 | 0.00 | 1.00 | +| concept_exact | rental_royalty_income_amount | 95,949,567,370 | 111,266,691,574 | 98,375,781,509 | 0.33 | 0.82 | +| concept_exact_no_business_count | schedule_c_income_returns | 30,354,680 | 10,446,932 | 14,401,264 | 0.00 | 0.00 | +| concept_exact_no_business_count | schedule_c_income_amount | 472,289,642,162 | 422,542,007,445 | 493,450,521,143 | 0.27 | 0.94 | +| concept_exact_no_business_count | partnership_scorp_income_returns | 9,551,770 | 7,247,623 | 9,523,958 | 0.12 | 1.00 | +| concept_exact_no_business_count | partnership_scorp_income_amount | 1,157,602,275,777 | 1,252,526,622,019 | 1,120,216,282,646 | 0.47 | 0.86 | +| concept_exact_no_business_count | rental_royalty_income_returns | 9,398,010 | 6,140,639 | 9,323,595 | 0.00 | 1.00 | +| concept_exact_no_business_count | rental_royalty_income_amount | 95,949,567,370 | 113,142,552,295 | 99,319,784,559 | 0.31 | 0.84 | +| drop | schedule_c_income_returns | 30,354,680 | 14,705,483 | 19,545,854 | 0.00 | 0.04 | +| drop | schedule_c_income_amount | 472,289,642,162 | 1,013,049,234,570 | 1,320,738,618,653 | 0.00 | 0.00 | +| drop | partnership_scorp_income_returns | 9,551,770 | 5,125,505 | 7,448,438 | 0.00 | 0.18 | +| drop | partnership_scorp_income_amount | 1,157,602,275,777 | 986,909,097,088 | 811,085,770,783 | 0.06 | 0.10 | +| drop | rental_royalty_income_returns | 9,398,010 | 8,952,775 | 14,248,242 | 0.51 | 0.08 | +| drop | rental_royalty_income_amount | 95,949,567,370 | 96,244,798,434 | 95,051,311,739 | 0.84 | 0.49 | +| rebase_positive_part | schedule_c_income_returns | 30,354,680 | 16,765,414 | 22,493,042 | 0.00 | 0.00 | +| rebase_positive_part | schedule_c_income_amount | 472,289,642,162 | 524,427,962,673 | 613,289,949,111 | 0.49 | 0.00 | +| rebase_positive_part | partnership_scorp_income_returns | 9,551,770 | 7,268,319 | 9,523,299 | 0.12 | 1.00 | +| rebase_positive_part | partnership_scorp_income_amount | 1,157,602,275,777 | 1,247,921,767,077 | 1,119,446,019,406 | 0.51 | 0.86 | +| rebase_positive_part | rental_royalty_income_returns | 9,398,010 | 6,349,815 | 10,669,611 | 0.00 | 0.43 | +| rebase_positive_part | rental_royalty_income_amount | 95,949,567,370 | 119,793,997,240 | 118,216,927,758 | 0.14 | 0.16 | diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 7df8db9b95..61f5accb30 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -11,6 +11,7 @@ import json import math +import re from collections.abc import Iterable, Mapping from dataclasses import dataclass, replace from functools import cache @@ -60,7 +61,9 @@ "US_JCT_TAX_EXPENDITURE_REFORMS", "US_JCT_TAX_EXPENDITURE_TARGET_SPECS", "US_JCT_TAX_EXPENDITURE_TARGET_REFERENCES", + "SOI_NET_INCOME_LESS_LOSS_VARIABLES", "SOI_VARIABLE_MAP", + "UnregisteredSoiNetIncomeLessLossFamilyError", "US_SOI_STATE_AGI_BAND_MINIMUM_LOWER_BOUND", "US_SOI_FISCAL_TARGET_SPECS", "US_SOI_FISCAL_TARGET_REFERENCES", @@ -128,15 +131,18 @@ def _us_hierarchy_seed( "medical_dental_expense_amount": "medical_expense_deduction", "net_capital_gains_amount": "capital_gains_gross", "ordinary_dividends_amount": "ordinary_dividends", - "partnership_scorp_income_amount": "partnership_and_s_corp_income", + # Historic Table 2 and the congressional-district file publish these three + # as signed "net income (less loss)" columns (microcosm#1228); see + # SOI_NET_INCOME_LESS_LOSS_VARIABLES. + "partnership_scorp_income_amount": "partnership_and_s_corp_net_income_less_loss", "premium_tax_credit_amount": "assigned_aca_ptc", "qualified_business_income_deduction_amount": ( "qualified_business_income_deduction" ), "qualified_dividends_amount": "qualified_dividends", "real_estate_taxes_amount": "real_estate_taxes", - "rental_royalty_income_amount": "rent_and_royalty_net_income", - "schedule_c_income_amount": "business_net_profits", + "rental_royalty_income_amount": "rent_and_royalty_net_income_less_loss", + "schedule_c_income_amount": "business_net_income_less_loss", "tax_exempt_interest_amount": "tax_exempt_interest_income", "tax_filer_individual_count": "tax_filer_individual_count", "taxable_income_amount": "taxable_income", @@ -212,16 +218,17 @@ def _us_hierarchy_seed( "medical_dental_expense_returns": "medical_expense_deduction", "net_capital_gains_returns": "capital_gains_gross", "ordinary_dividends_returns": "ordinary_dividends", - "partnership_scorp_income_returns": "partnership_and_s_corp_income", + # Every return with a net income or a net loss on the line (microcosm#1228). + "partnership_scorp_income_returns": "partnership_and_s_corp_net_income_less_loss", "premium_tax_credit_returns": "assigned_aca_ptc", "qualified_business_income_deduction_returns": ( "qualified_business_income_deduction" ), "qualified_dividends_returns": "qualified_dividends", "real_estate_taxes_claims": "real_estate_taxes", - "rental_royalty_income_returns": "rent_and_royalty_net_income", + "rental_royalty_income_returns": "rent_and_royalty_net_income_less_loss", "return_count": "count", - "schedule_c_income_returns": "business_net_profits", + "schedule_c_income_returns": "business_net_income_less_loss", "tax_exempt_interest_returns": "tax_exempt_interest_income", "taxable_income_returns": "taxable_income", "taxable_interest_returns": "taxable_interest_income", @@ -253,8 +260,15 @@ def _us_hierarchy_seed( SOI_VARIABLE_MAP: dict[str, str] = { "adjusted_gross_income": "adjusted_gross_income", - "business_net_profits": "self_employment_income", - "business_net_losses": "self_employment_income", + # Schedule C income is all of a person's businesses, SSTB or not + # (Schedule 1 line 3). The frame moves an SSTB business's whole amount to + # sstb_self_employment_income (us_runtime/qbi_inputs.py), so only + # total_self_employment_income carries it all (microcosm#1228). The + # profit/loss legs are reserved for Table 1.4's "Business or profession" + # net income and net loss columns, which no pinned fact carries yet. + "business_net_income_less_loss": "total_self_employment_income", + "business_net_profits": "total_self_employment_income", + "business_net_losses": "total_self_employment_income", "capital_gains_distributions": "capital_gains", "capital_gains_gross": "capital_gains", "capital_gains_losses": "capital_losses", @@ -290,12 +304,25 @@ def _us_hierarchy_seed( "ordinary_dividends": "ordinary_dividend_income", "partnership_and_s_corp_income": "tax_unit_partnership_s_corp_income", "partnership_and_s_corp_losses": "tax_unit_partnership_s_corp_income", + # Schedule E line 32. PE-US means partnership_s_corp_income to be that + # total (passive_partnership_s_corp_income is its passive subset), but the + # processed PUF donor builds it from the nonpassive fields only + # ((E26190 - E26180) + (E25980 - E25960), not E26270), so this frame's + # column is narrower than the line it targets (microcosm#1228). + "partnership_and_s_corp_net_income_less_loss": ( + "tax_unit_partnership_s_corp_income" + ), "assigned_aca_ptc": "assigned_aca_ptc", "qualified_business_income_deduction": "qualified_business_income_deduction", "qualified_dividends": "qualified_dividend_income", "real_estate_taxes": "real_estate_taxes", + # Table 1.4's "Total rental and royalty" legs include Form 4835 farm + # rental (rental_income + farm_rent_income, summed in the materializer). + # Historic Table 2's A25870 is Schedule E line 26, rents and royalties + # only: farm rental goes on line 40 (microcosm#1228). "rent_and_royalty_net_income": "rent_and_royalty_net_income", "rent_and_royalty_net_losses": "rent_and_royalty_net_income", + "rent_and_royalty_net_income_less_loss": "rental_income", "refundable_ctc": "refundable_ctc", "salt_deduction": "salt_deduction", "tax_exempt_interest_income": "tax_exempt_interest_income", @@ -310,6 +337,49 @@ def _us_hierarchy_seed( "unemployment_compensation": "unemployment_compensation", } +#: SOI "net income (less loss)" concepts (microcosm#1228). Historic Table 2 +#: and the congressional-district file publish business income (Schedule 1 +#: line 3), partnership and S corporation income (Schedule E line 32) and +#: rental and royalty income (Schedule E line 26) as one signed column each: +#: the amount nets losses against income, and the count is every return with +#: an entry, net income or net loss. Table 1.4 splits each line in two, and +#: the identities hold: TY2022 HT2 N00900 30,354,680 is col 31 + col 33, +#: 30,356,401 (the profit returns alone are 72%). These concepts bind the +#: signed tax-unit total, and their indicator counts a nonzero total +#: (tools/build_us_fiscal_refresh_release.py ``_soi_component_row``). The +#: positive-part legs (``business_net_profits``, +#: ``partnership_and_s_corp_income``, ``rent_and_royalty_net_income``) and +#: their ``*_losses`` magnitudes stay reserved for Table 1.4's split columns. +SOI_NET_INCOME_LESS_LOSS_VARIABLES: frozenset[str] = frozenset( + { + "business_net_income_less_loss", + "partnership_and_s_corp_net_income_less_loss", + "rent_and_royalty_net_income_less_loss", + } +) +#: The return line each net-income-(less-loss) concept reports, stamped on its +#: specs as ``soi_source_concept``. +_SOI_NET_INCOME_LESS_LOSS_SOURCE_CONCEPTS: dict[str, str] = { + "business_net_income_less_loss": ( + "form_1040_schedule_1_line_3_net_income_less_loss" + ), + "partnership_and_s_corp_net_income_less_loss": ( + "schedule_e_line_32_net_income_less_loss" + ), + "rent_and_royalty_net_income_less_loss": ( + "schedule_e_line_26_net_income_less_loss" + ), +} +#: The SOI record-set families reviewed to publish the net-income-(less-loss) +#: columns under the ``schedule_c_income_*``, ``partnership_scorp_income_*`` +#: and ``rental_royalty_income_*`` ids. A fact with one of those ids from any +#: other family refuses the compile, because the same id could name a one-sided +#: column there (microcosm#1035 is the capital-gains case of one id naming two +#: IRS measures). +_SOI_NET_INCOME_LESS_LOSS_FAMILIES = frozenset( + {"historic_table_2", "congressional_district"} +) + _SOI_BASE_VARIABLE_OVERRIDES: dict[str, tuple[str, ...]] = { "ctc": ("ctc", "ctc_limiting_tax_liability"), "ordinary_dividends": ( @@ -3100,6 +3170,8 @@ def _soi_reference_from_fact( variable, is_count = override if variable is None: return None + if variable in SOI_NET_INCOME_LESS_LOSS_VARIABLES: + _require_soi_net_income_less_loss_family(fact, measure_id=measure_id) lower, upper = _agi_bounds(fact) cross_period_agi_slice = _is_untransformed_cross_period_agi_slice( @@ -3158,6 +3230,9 @@ def _soi_reference_from_fact( "soi_return_universe": soi_return_universe, **_soi_layout_filter_metadata(fact), } + source_concept = _SOI_NET_INCOME_LESS_LOSS_SOURCE_CONCEPTS.get(variable) + if source_concept is not None: + metadata["soi_source_concept"] = source_concept base_variables = _soi_base_variables(variable) if len(base_variables) == 1: metadata["base_variable"] = base_variables[0] @@ -3200,6 +3275,57 @@ def _soi_measure_mode(variable: str, *, is_count: bool) -> str: return "sum" +#: The district file's family token, with or without its plan vintage +#: (``congressional_district_2022``); nothing else that starts the same way. +_SOI_CONGRESSIONAL_DISTRICT_FAMILY = re.compile(r"congressional_district(_\d{4})?") + + +class UnregisteredSoiNetIncomeLessLossFamilyError(ValueError): + """A net-income-(less-loss) measure id arrived from an unreviewed family.""" + + +def _soi_source_family(record_set_id: str) -> str: + """An SOI record set's table family, without period or data vintage. + + ``irs_soi.ty2023.table_1_4`` is ``table_1_4``, + ``irs_soi.ty2022.historic_table_2.state_broad`` is ``historic_table_2`` and + ``irs_soi.ty2023.congressional_district_2022.all_returns`` is + ``congressional_district``. + """ + parts = record_set_id.split(".") + if parts[0] != "irs_soi": + return "" + rest = parts[2:] if len(parts) > 1 and _is_period_token(parts[1]) else parts[1:] + if not rest: + return "" + family = rest[0] + if _SOI_CONGRESSIONAL_DISTRICT_FAMILY.fullmatch(family): + return "congressional_district" + return family + + +def _require_soi_net_income_less_loss_family(fact: object, *, measure_id: str) -> None: + """Refuse a net-income-(less-loss) id from a family nobody reviewed. + + Historic Table 2 and the congressional-district file publish these ids as + one signed column per return line (microcosm#1228). Table 1.4 splits the + same lines into one-sided "net income" and "net loss" columns, so the same + id from it, or from any new family, could name a different measure. + """ + record_set_id = _str_at(fact, "layout", "record_set_id") + family = _soi_source_family(record_set_id) + if family in _SOI_NET_INCOME_LESS_LOSS_FAMILIES: + return + raise UnregisteredSoiNetIncomeLessLossFamilyError( + f"SOI measure {measure_id!r} from record set {record_set_id!r} (family " + f"{family or ''!r}) binds a signed net-income-(less-loss) " + "concept, which only these families are reviewed to publish: " + f"{sorted(_SOI_NET_INCOME_LESS_LOSS_FAMILIES)}. Review what the " + "column counts and register the family, or map the id to a " + "one-sided concept (microcosm#1228)." + ) + + def _soi_display_variable(variable: str) -> str: if variable == "ordinary_dividends": return "ordinary_dividend_income" diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_batched_target_materialization.py b/packages/microcosm-build/tests/engine_free/us/test_us_batched_target_materialization.py index 6cda417458..6c774431bd 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_batched_target_materialization.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_batched_target_materialization.py @@ -585,3 +585,105 @@ def engine(known, **extra): builder._refuse_batch_population_aggregates( root, batch=1, batches=3, n_households=6 ) + + +# microcosm#1228: pinned inputs in household 4 (tax units 40 = persons 7 and +# 8, 41 = person 9, 42 = person 10). Spouses 7 and 8 net to a Schedule C +# loss; person 10's business is an SSTB, so all of it sits outside +# self_employment_income; tax unit 40's only rental is Form 4835 farm rent. +_NET_INCOME_LESS_LOSS_PINS = { + ("total_self_employment_income", 7): 300.0, + ("total_self_employment_income", 8): -500.0, + ("total_self_employment_income", 9): 0.0, + ("total_self_employment_income", 10): 150.0, + ("self_employment_income", 10): 0.0, + ("rental_income", 7): 0.0, + ("rental_income", 8): 0.0, + ("rental_income", 9): -100.0, + ("rental_income", 10): 0.0, + ("farm_rent_income", 7): 400.0, + ("farm_rent_income", 8): 0.0, + ("farm_rent_income", 9): 0.0, + ("farm_rent_income", 10): 0.0, + ("tax_unit_partnership_s_corp_income", 40): -50.0, + ("tax_unit_partnership_s_corp_income", 41): 0.0, + ("tax_unit_partnership_s_corp_income", 42): 75.0, +} + + +def test_net_income_less_loss_rows_materialize_signed_totals_and_nonzero_counts( + monkeypatch, +) -> None: + """End to end through the SOI slice (microcosm#1228): each signed concept + sums its tax-unit total and counts the units with a nonzero total, after + the persons of a unit net; business reads every Schedule C, SSTB + included; line-26 rental leaves out farm rent, which only the Table 1.4 + leg carries. Hand-computed for household 4.""" + + import test_support.microcosm_build.us_batched_target_materialization as harness + from microcosm.build.us_runtime.fiscal_targets import SOI_VARIABLE_MAP + + real_record_value = harness._record_value + + def record_value(variable, record_id): + pinned = _NET_INCOME_LESS_LOSS_PINS.get((variable, int(record_id))) + return real_record_value(variable, record_id) if pinned is None else pinned + + monkeypatch.setattr(harness, "_record_value", record_value) + for variable, entity in ( + ("total_self_employment_income", "person"), + ("self_employment_income", "person"), + ("tax_unit_partnership_s_corp_income", "tax_unit"), + ): + monkeypatch.setitem(harness._VARIABLE_ENTITIES, variable, entity) + builder = _load_builder_module() + _install_fake_engine(builder, monkeypatch, reform_specs=()) + concepts = ( + "business_net_income_less_loss", + "business_net_profits", + "business_net_losses", + "partnership_and_s_corp_net_income_less_loss", + "rent_and_royalty_net_income_less_loss", + "rent_and_royalty_net_income", + ) + # The fake engine installs a reduced map; add the real bindings of the + # concepts under test. + monkeypatch.setattr( + builder, + "SOI_VARIABLE_MAP", + { + **builder.SOI_VARIABLE_MAP, + **{concept: SOI_VARIABLE_MAP[concept] for concept in concepts}, + }, + ) + specs = tuple( + spec + for concept in concepts + for spec in ( + _soi(f"{concept}_amount", concept), + _soi(f"{concept}_returns", concept, measure_mode="indicator_sum"), + ) + ) + + target_frame, _, compilation = builder._materialize_target_frame( + _nested_frame(), specs, maximum_microsim_batch_size=None + ) + assert compilation["dropped_target_names"] == [] + household = target_frame.table("household").set_index("household_id").loc[4] + + # Business: units 40 (300 - 500 = -200), 41 (0), 42 (SSTB-only 150). + assert household["business_net_income_less_loss_amount"] == -50.0 + assert household["business_net_income_less_loss_returns"] == 2.0 + assert household["business_net_profits_amount"] == 150.0 + assert household["business_net_profits_returns"] == 1.0 + assert household["business_net_losses_amount"] == 200.0 + assert household["business_net_losses_returns"] == 1.0 + # Partnership/S-corp: units 40 (-50), 41 (0), 42 (75). + assert household["partnership_and_s_corp_net_income_less_loss_amount"] == 25.0 + assert household["partnership_and_s_corp_net_income_less_loss_returns"] == 2.0 + # Line 26 rental: unit 40's farm rent does not count; unit 41 is a loss. + assert household["rent_and_royalty_net_income_less_loss_amount"] == -100.0 + assert household["rent_and_royalty_net_income_less_loss_returns"] == 1.0 + # The Table 1.4 leg includes farm rent and takes the positive part. + assert household["rent_and_royalty_net_income_amount"] == 400.0 + assert household["rent_and_royalty_net_income_returns"] == 1.0 diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py index 9adfe409b7..3df255921b 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py @@ -1,8 +1,10 @@ import dataclasses +import functools import hashlib import importlib.util import inspect import json +import math import os import sys from dataclasses import replace @@ -14,6 +16,8 @@ import numpy as np import pandas as pd import pytest +from hypothesis import given, settings +from hypothesis import strategies as st from microcosm.build.us_runtime import h5_io as builder_h5_io from microcosm.calibrate import ( @@ -1938,18 +1942,64 @@ def test_soi_component_amounts_use_source_specific_signs() -> None: builder._signed_component(np.array([-5.0, 0.0, 7.0]), "business_net_losses"), np.array([5.0, -0.0, -0.0]), ) + # The Table 1.4 "net income" leg is a positive part like every other + # income leg; the signed Historic Table 2 column has its own concept + # (microcosm#1228). assert np.array_equal( builder._signed_component( np.array([-5.0, 0.0, 7.0]), "rent_and_royalty_net_income" ), - np.array([-5.0, 0.0, 7.0]), + np.array([0.0, 0.0, 7.0]), ) + for concept in ( + "business_net_income_less_loss", + "partnership_and_s_corp_net_income_less_loss", + "rent_and_royalty_net_income_less_loss", + ): + assert np.array_equal( + builder._signed_component(np.array([-5.0, 0.0, 7.0]), concept), + np.array([-5.0, 0.0, 7.0]), + ) assert np.array_equal( builder._signed_component(np.array([-5.0, 7.0]), "adjusted_gross_income"), np.array([-5.0, 7.0]), ) +def test_net_income_less_loss_concepts_are_the_registered_signed_set() -> None: + """The materializer keeps the sign for exactly the registered concepts, + and each of them reads one model column from the SOI variable map.""" + builder = _load_builder_module() + + assert builder.SOI_NET_INCOME_LESS_LOSS_VARIABLES == { + "business_net_income_less_loss", + "partnership_and_s_corp_net_income_less_loss", + "rent_and_royalty_net_income_less_loss", + } + assert { + concept: builder.SOI_VARIABLE_MAP[concept] + for concept in builder.SOI_NET_INCOME_LESS_LOSS_VARIABLES + } == { + # Schedule 1 line 3 is every Schedule C, SSTB or not. + "business_net_income_less_loss": "total_self_employment_income", + "partnership_and_s_corp_net_income_less_loss": ( + "tax_unit_partnership_s_corp_income" + ), + # Schedule E line 26 leaves out Form 4835 farm rental (line 40). + "rent_and_royalty_net_income_less_loss": "rental_income", + } + values = np.array([-3.0, 0.0, 2.0]) + signed = { + concept + for concept in builder.SOI_VARIABLE_MAP + if (builder._signed_component(values, concept) < 0).any() + } + assert signed == { + "adjusted_gross_income", + *builder.SOI_NET_INCOME_LESS_LOSS_VARIABLES, + } + + def test_export_target_audit_is_opt_in(monkeypatch) -> None: builder = _load_builder_module() @@ -2877,7 +2927,9 @@ def test_env_default_treats_blank_as_unset_and_trims(monkeypatch) -> None: assert builder._env_default("POPULACE_TEST_ENV_DEFAULT", "fallback") == "org/repo" -def test_soi_indicator_rows_flag_positive_component_items() -> None: +def test_soi_indicator_rows_flag_component_items() -> None: + """A return counts when its component is positive; a net-income-(less-loss) + return counts when its signed total is nonzero (microcosm#1228).""" builder = _load_builder_module() assert np.array_equal( @@ -2904,6 +2956,119 @@ def test_soi_indicator_rows_flag_positive_component_items() -> None: ), np.array([1.0, 0.0, 0.0]), ) + # A "net income (less loss)" count is every return with an entry on the + # line, net income or net loss (microcosm#1228). + for concept in ( + "business_net_income_less_loss", + "partnership_and_s_corp_net_income_less_loss", + "rent_and_royalty_net_income_less_loss", + ): + assert np.array_equal( + builder._soi_component_row( + np.array([-5.0, 0.0, 7.0]), concept, indicator=True + ), + np.array([1.0, 0.0, 1.0]), + ) + assert np.array_equal( + builder._soi_component_row( + np.array([-5.0, 0.0, 7.0]), concept, indicator=False + ), + np.array([-5.0, 0.0, 7.0]), + ) + + +# Each signed column's (net income, net loss) legs: Table 1.4's split of the +# line Historic Table 2 publishes as one "net income (less loss)" column. +_NET_INCOME_LESS_LOSS_LEGS = { + "business_net_income_less_loss": ("business_net_profits", "business_net_losses"), + "partnership_and_s_corp_net_income_less_loss": ( + "partnership_and_s_corp_income", + "partnership_and_s_corp_losses", + ), + "rent_and_royalty_net_income_less_loss": ( + "rent_and_royalty_net_income", + "rent_and_royalty_net_losses", + ), +} + + +@functools.cache +def _cached_builder_module(): + return _load_builder_module() + + +@settings(max_examples=300, deadline=None) +@given( + concept=st.sampled_from(sorted(_NET_INCOME_LESS_LOSS_LEGS)), + rows=st.lists( + st.tuples( + st.one_of( + st.just(0.0), + st.floats( + min_value=-1e9, + max_value=1e9, + allow_nan=False, + allow_infinity=False, + ), + ), + st.integers(min_value=0, max_value=100_000), + ), + max_size=60, + ), +) +def test_net_income_less_loss_rows_are_the_sum_of_their_table_1_4_legs( + concept: str, rows: list[tuple[float, int]] +) -> None: + """For every signed tax-unit vector and nonnegative weights, read as one + signed column (microcosm#1228): + + - count conservation: the weighted less-loss count equals the net-income + leg's count plus the net-loss leg's count; + - amount conservation: the weighted less-loss amount equals the net-income + leg's amount minus the net-loss leg's amount; + - bounds: the less-loss count is at least the net-income count, and the + less-loss amount at most the net-income amount; + - sign symmetry: negating every value leaves the count unchanged and + negates the amount. + + For business the conservation laws are the published identities + N00900 = Table 1.4 col 31 + col 33 and A00900 = col 32 - col 34. Table + 1.4's rental legs report Schedule E lines 26 + 39 + 40 and its TY2021+ + partnership and S-corp legs are per entity type, so for those the property + checks the model's own decomposition of one column. + """ + builder = _cached_builder_module() + values = np.array([value for value, _ in rows], dtype=np.float64) + weights = np.array([weight for _, weight in rows], dtype=np.float64) + income_leg, loss_leg = _NET_INCOME_LESS_LOSS_LEGS[concept] + + def total(source_name: str, vector: np.ndarray, *, indicator: bool) -> float: + row = builder._soi_component_row(vector, source_name, indicator=indicator) + return math.fsum(row * weights) + + count = total(concept, values, indicator=True) + income_count = total(income_leg, values, indicator=True) + loss_count = total(loss_leg, values, indicator=True) + # Integer weights times 0/1 flags sum exactly in float64. + assert count == income_count + loss_count + assert count >= income_count + + amount = total(concept, values, indicator=False) + income_amount = total(income_leg, values, indicator=False) + loss_amount = total(loss_leg, values, indicator=False) + assert loss_amount >= 0.0 + assert math.isclose( + amount, income_amount - loss_amount, rel_tol=1e-12, abs_tol=1e-3 + ) + assert amount <= income_amount + 1e-3 + + assert total(concept, -values, indicator=True) == count + assert math.isclose( + total(concept, -values, indicator=False), + -amount, + rel_tol=1e-12, + abs_tol=1e-3, + ) def test_soi_eitc_child_count_filter_uses_ledger_filter_first() -> None: diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 3cd976bac3..f8c40a0823 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -895,9 +895,14 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) -> None: """The release surface on the pinned feed: 33,250 compiled targets after Medicaid substitution, 6,102 national_state targets at registry - e1f9906bf0e0, 32 CHIP rows none of them for an M-CHIP state, no - other-income row and no tips return count. microcosm#1206 moved the - version from f96be9f41781 by binding the Part B premium target to + 38c1db8880da, 32 CHIP rows none of them for an M-CHIP state, no + other-income row and no tips return count. microcosm#1228 moved the + version from e1f9906bf0e0 (full f5bd921d6b21 -> 9f6c65fe8734) by binding + the 312 Historic Table 2 business, partnership and rental rows to signed + net-income-(less-loss) concepts (3,294 on full: these, 2,928 + district-file rows and 54 single-district-state Historic Table 2 rows); + no value changed. microcosm#1206 moved it from f96be9f41781 by + binding the Part B premium target to gross_medicare_part_b_premium_if_enrolled; the counts are unchanged. The 408 state AGI-band rows of microcosm#940 (51 states x 4 bands x 2 measures) took it from 32,842 / @@ -908,7 +913,7 @@ def test_pinned_feed_national_state_surface_restores_the_fences( registry, surface, _ = pinned_feed_national_state_surface assert len(registry.specs) == 33_250 assert len(surface.specs) == 6_102 - assert surface.version == "e1f9906bf0e0" + assert surface.version == "38c1db8880da" chip = [ spec for spec in surface.specs @@ -935,6 +940,301 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) +# microcosm#1228: the measure ids Historic Table 2 and the district file use +# for their signed "net income (less loss)" columns, the concept each binds and +# the return line it reports. +_NET_INCOME_LESS_LOSS_CONCEPTS = { + "schedule_c_income": ( + "business_net_income_less_loss", + "form_1040_schedule_1_line_3_net_income_less_loss", + ), + "partnership_scorp_income": ( + "partnership_and_s_corp_net_income_less_loss", + "schedule_e_line_32_net_income_less_loss", + ), + "rental_royalty_income": ( + "rent_and_royalty_net_income_less_loss", + "schedule_e_line_26_net_income_less_loss", + ), +} +_NET_INCOME_LESS_LOSS_MEASURES = { + f"{stem}_{kind}": (concept, source_concept, kind) + for stem, (concept, source_concept) in _NET_INCOME_LESS_LOSS_CONCEPTS.items() + for kind in ("amount", "returns") +} +_ONE_SIDED_INCOME_LEGS = frozenset( + { + "business_net_profits", + "business_net_losses", + "partnership_and_s_corp_income", + "partnership_and_s_corp_losses", + "rent_and_royalty_net_income", + "rent_and_royalty_net_losses", + } +) + + +def _net_income_less_loss_fact( + measure_id: str, + value: float, + *, + record_set_id: str = "irs_soi.ty2022.historic_table_2.us", + geography_level: str = "country", + geography_id: str = "0100000US", + period: int = 2022, +) -> dict[str, object]: + return _soi_capital_gains_fact( + period, + source_record_id=f"{record_set_id}.all.{measure_id}", + measure_id=measure_id, + layout_record_set_id=record_set_id, + geography_level=geography_level, + geography_id=geography_id, + value=value, + ) + + +def _assert_binds_net_income_less_loss(spec, measure_id: str) -> None: + concept, source_concept, kind = _NET_INCOME_LESS_LOSS_MEASURES[measure_id] + metadata = spec.metadata + assert metadata["source_variable"] == concept + assert metadata["variable"] == concept + assert metadata["soi_source_concept"] == source_concept + assert metadata["base_variable"] == fiscal_targets.SOI_VARIABLE_MAP[concept] + assert metadata["measure_mode"] == ("indicator_sum" if kind == "returns" else "sum") + assert spec.hierarchy is not None + assert spec.hierarchy.category.id == f"irs_soi.{concept}" + + +def test_net_income_less_loss_rows_bind_signed_concepts() -> None: + """Historic Table 2 US and state rows and district-file rows of all six + ids bind the signed concept of their line, at their published value + (microcosm#1228). A district row of a net loss stays negative.""" + facts = [*packaged_reference_facts()] + expected: dict[str, tuple[str, float]] = {} + for index, measure_id in enumerate(sorted(_NET_INCOME_LESS_LOSS_MEASURES)): + rows = ( + ("irs_soi.ty2022.historic_table_2.us", "country", "0100000US"), + ("irs_soi.ty2022.historic_table_2.state_broad.ca", "state", "0400000US06"), + ) + for record_set_id, level, geography_id in rows: + value = 1_000.0 + index + fact = _net_income_less_loss_fact( + measure_id, + value, + record_set_id=record_set_id, + geography_level=level, + geography_id=geography_id, + ) + facts.append(fact) + expected[fact["lineage"]["source_record_id"]] = (measure_id, value) + district_value = -250.0 if measure_id.endswith("_amount") else 75.0 + district = _soi_congressional_district_fact(measure_id, district_value) + facts.append(district) + expected[district["lineage"]["source_record_id"]] = ( + measure_id, + district_value, + ) + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + by_record = { + spec.metadata["ledger_source_record_id"]: spec + for spec in registry.specs + if spec.metadata.get("source_measure_id") in _NET_INCOME_LESS_LOSS_MEASURES + } + assert set(by_record) == set(expected) + for source_record_id, (measure_id, value) in expected.items(): + spec = by_record[source_record_id] + _assert_binds_net_income_less_loss(spec, measure_id) + assert spec.value == value + assert spec.signed is (value < 0) + assert not [ + spec.name + for spec in registry.specs + if spec.metadata.get("source_variable") in _ONE_SIDED_INCOME_LEGS + ] + + +@pytest.mark.parametrize( + "record_set_id", + [ + "irs_soi.ty2023.table_1_4", + "irs_soi.ty2023.table_1_1", + "irs_soi.ty2023.itemized_all_returns", + "irs_soi.ty2022.zip_code.all_returns", + "irs_soi.historic_table_3", + ], +) +def test_net_income_less_loss_ids_refuse_unreviewed_families( + record_set_id: str, +) -> None: + """Table 1.4 splits these lines into one-sided columns, so a + net-income-(less-loss) id from it, or from any family nobody reviewed, + refuses the compile instead of binding a signed concept.""" + facts = [ + *packaged_reference_facts(), + _net_income_less_loss_fact( + "schedule_c_income_returns", 21_969_832.0, record_set_id=record_set_id + ), + ] + + with pytest.raises( + fiscal_targets.UnregisteredSoiNetIncomeLessLossFamilyError, + match=re.escape("schedule_c_income_returns"), + ): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + + +def test_soi_source_family_strips_period_and_district_vintage() -> None: + family = fiscal_targets._soi_source_family + assert family("irs_soi.ty2023.table_1_4") == "table_1_4" + assert family("irs_soi.ty2022.historic_table_2.state_broad.ca") == ( + "historic_table_2" + ) + assert family("irs_soi.ty2022.historic_table_2.us") == "historic_table_2" + assert family("irs_soi.historic_table_2.us") == "historic_table_2" + assert family("irs_soi.ty2023.congressional_district_2022.all_returns") == ( + "congressional_district" + ) + assert family("irs_soi.ty2026.congressional_district_2024.all_returns") == ( + "congressional_district" + ) + assert family("irs_soi.ty2023.congressional_district_table_1_4") == ( + "congressional_district_table_1_4" + ) + assert family("irs_soi") == "" + assert family("census_acs.acs1_2024.s0101") == "" + + +def test_net_income_less_loss_binding_depends_only_on_the_family() -> None: + """Property (microcosm#1228): for any id, tax year and record-set shape, + a fact binds the signed concept of its line exactly when its record set + is Historic Table 2 or the district file, and refuses otherwise; the + count/amount split follows the id.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + families = st.sampled_from( + [ + "historic_table_2", + "congressional_district_2022", + "congressional_district_2024", + "congressional_district_table_1_4", + "table_1_4", + "table_1_1", + "table_2_1", + "itemized_all_returns", + "zip_code", + "county", + ] + ) + + @settings(max_examples=200, deadline=None) + @given( + measure_id=st.sampled_from(sorted(_NET_INCOME_LESS_LOSS_MEASURES)), + period=st.integers(min_value=2010, max_value=2035), + family=families, + tail=st.sampled_from(["", ".us", ".state_broad.ny", ".all_returns"]), + dated=st.booleans(), + value=st.floats(min_value=-1e12, max_value=1e12, allow_nan=False), + ) + def check(measure_id, period, family, tail, dated, value) -> None: + record_set_id = ( + f"irs_soi.ty{period}.{family}{tail}" if dated else f"irs_soi.{family}{tail}" + ) + fact = _net_income_less_loss_fact( + measure_id, value, record_set_id=record_set_id, period=period + ) + registered = family == "historic_table_2" or bool( + re.fullmatch(r"congressional_district(_\d{4})?", family) + ) + if not registered: + with pytest.raises( + fiscal_targets.UnregisteredSoiNetIncomeLessLossFamilyError + ): + fiscal_targets._soi_reference_from_fact(fact, target_period=period) + return + reference = fiscal_targets._soi_reference_from_fact(fact, target_period=period) + assert reference is not None + concept, source_concept, kind = _NET_INCOME_LESS_LOSS_MEASURES[measure_id] + assert reference.metadata["source_variable"] == concept + assert reference.metadata["soi_source_concept"] == source_concept + assert reference.metadata["measure_mode"] == ( + "indicator_sum" if kind == "returns" else "sum" + ) + assert concept in fiscal_targets.SOI_NET_INCOME_LESS_LOSS_VARIABLES + + check() + + +def test_pinned_feed_net_income_less_loss_rows_bind_signed_concepts( + pinned_feed_national_state_surface, +) -> None: + """Pinned-feed invariant (microcosm#1228): every business, partnership + and rental row binds the signed concept of its line, only those rows bind + a signed concept, every one comes from Historic Table 2 or the district + file, and no row binds a one-sided Table 1.4 leg. national_state carries + the 6 ids x (51 states + the US) = 312 Historic Table 2 rows; full adds + 2,928 district-file rows and 54 Historic Table 2 rows of the nine + single-district states, which full classes as district rows. As a data + pin (the US row includes territories and other areas, so it is not a law + for signed amounts), each measure's 51 state rows sum to at most its + national row.""" + registry, surface, _ = pinned_feed_national_state_surface + + rows = [ + spec + for spec in registry.specs + if spec.metadata.get("source_measure_id") in _NET_INCOME_LESS_LOSS_MEASURES + ] + for spec in rows: + _assert_binds_net_income_less_loss(spec, spec.metadata["source_measure_id"]) + assert fiscal_targets._soi_source_family( + spec.metadata["ledger_layout_record_set_id"] + ) in {"historic_table_2", "congressional_district"} + assert { + spec.name + for spec in registry.specs + if spec.metadata.get("source_variable") + in fiscal_targets.SOI_NET_INCOME_LESS_LOSS_VARIABLES + } == {spec.name for spec in rows} + assert not [ + spec.name + for spec in registry.specs + if spec.metadata.get("source_variable") in _ONE_SIDED_INCOME_LEGS + ] + + surface_rows = [ + spec + for spec in surface.specs + if spec.metadata.get("source_measure_id") in _NET_INCOME_LESS_LOSS_MEASURES + ] + assert len(surface_rows) == 312 + assert { + fiscal_targets._soi_source_family(spec.metadata["ledger_layout_record_set_id"]) + for spec in surface_rows + } == {"historic_table_2"} + for measure_id in _NET_INCOME_LESS_LOSS_MEASURES: + states = [ + spec.value + for spec in surface_rows + if spec.metadata["source_measure_id"] == measure_id + and spec.metadata.get("ledger_geography_level") == "state" + ] + (national,) = [ + spec.value + for spec in surface_rows + if spec.metadata["source_measure_id"] == measure_id + and spec.metadata.get("ledger_geography_level") == "country" + ] + assert len(states) == 51 + assert math.fsum(states) <= national + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." diff --git a/packages/microcosm-calibrate/src/microcosm/calibrate/variable_labels.py b/packages/microcosm-calibrate/src/microcosm/calibrate/variable_labels.py index 32a1425bdf..63d5f205d0 100644 --- a/packages/microcosm-calibrate/src/microcosm/calibrate/variable_labels.py +++ b/packages/microcosm-calibrate/src/microcosm/calibrate/variable_labels.py @@ -95,6 +95,7 @@ def _immutable_registry( "irs_soi": { "adjusted_gross_income": "Adjusted gross income", "assigned_aca_ptc": "Assigned ACA PTC", + "business_net_income_less_loss": "Business net income (less loss)", "business_net_profits": "Business net profits", "capital_gains_gross": "Gross capital gains", "charitable_deduction": "Charitable deduction", @@ -119,6 +120,9 @@ def _immutable_registry( "non_sch_d_capital_gains": "Non-Schedule D capital gains", "ordinary_dividend_income": "Ordinary dividend income", "partnership_and_s_corp_income": "Partnership and S corporation income", + "partnership_and_s_corp_net_income_less_loss": ( + "Partnership and S corporation net income (less loss)" + ), "qualified_dividends": "Qualified dividends", "qualified_business_income_deduction": ( "Qualified business income deduction" @@ -126,6 +130,9 @@ def _immutable_registry( "real_estate_taxes": "Real estate taxes", "refundable_ctc": "Refundable CTC", "rent_and_royalty_net_income": "Rent and royalty net income", + "rent_and_royalty_net_income_less_loss": ( + "Rent and royalty net income (less loss)" + ), "salt_deduction": "SALT deduction", "tax_exempt_interest_income": "Tax-exempt interest income", "taxable_income": "Taxable income", diff --git a/packages/microcosm-calibrate/tests/engine_free/shared/test_variable_labels.py b/packages/microcosm-calibrate/tests/engine_free/shared/test_variable_labels.py index 41654f1835..bc5e768fb4 100644 --- a/packages/microcosm-calibrate/tests/engine_free/shared/test_variable_labels.py +++ b/packages/microcosm-calibrate/tests/engine_free/shared/test_variable_labels.py @@ -46,6 +46,7 @@ def _pairs(value: str) -> set[tuple[str, str]]: hhs_acf_tanf|cash_assistance_expenditures irs_soi|adjusted_gross_income irs_soi|assigned_aca_ptc +irs_soi|business_net_income_less_loss irs_soi|business_net_profits irs_soi|capital_gains_gross irs_soi|charitable_deduction @@ -67,11 +68,13 @@ def _pairs(value: str) -> set[tuple[str, str]]: irs_soi|non_sch_d_capital_gains irs_soi|ordinary_dividend_income irs_soi|partnership_and_s_corp_income +irs_soi|partnership_and_s_corp_net_income_less_loss irs_soi|qualified_dividends irs_soi|qualified_business_income_deduction irs_soi|real_estate_taxes irs_soi|refundable_ctc irs_soi|rent_and_royalty_net_income +irs_soi|rent_and_royalty_net_income_less_loss irs_soi|salt_deduction irs_soi|tax_exempt_interest_income irs_soi|taxable_income diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index 6b64ce7059..ddec9b1827 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -244,6 +244,7 @@ from microcosm.build.us_runtime.fiscal_targets import ( AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS, AGE_BOUND_STAMP_SOURCE_KEY, + SOI_NET_INCOME_LESS_LOSS_VARIABLES, SSA_SSI_AGE_BAND_RECIPIENTS_TARGET_ROLE, ) from microcosm.build.us_runtime.h5_io import ( @@ -6012,8 +6013,19 @@ def _assert_supported_ledger_filter_metadata( def _signed_component(values: np.ndarray, source_name: str) -> np.ndarray: + """The tax-unit amount an SOI concept sums. + + AGI and the net-income-(less-loss) concepts keep the sign: their IRS + columns net losses against income (microcosm#1228). A ``*_losses`` + concept on a signed column is the magnitude of its negative part (Table + 1.4 "net loss"); ``capital_gains_losses`` reads ``capital_losses``, which + is a magnitude already. Every other concept is the positive part. + """ values = np.asarray(values, dtype=np.float64) - if source_name in {"adjusted_gross_income", "rent_and_royalty_net_income"}: + if ( + source_name == "adjusted_gross_income" + or source_name in SOI_NET_INCOME_LESS_LOSS_VARIABLES + ): return values if source_name == "capital_gains_losses": return np.maximum(values, 0.0) @@ -6031,8 +6043,17 @@ def _signed_component(values: np.ndarray, source_name: str) -> np.ndarray: def _soi_component_row( values: np.ndarray, source_name: str, *, indicator: bool ) -> np.ndarray: + """One SOI target row per tax unit: the summed amount, or a return flag. + + A return counts when its component is positive, except for a + net-income-(less-loss) concept, whose IRS count is every return with an + entry on the line, net income or net loss (HT2 N00900 = Table 1.4 + col 31 + col 33, microcosm#1228). + """ component = _signed_component(values, source_name) if indicator: + if source_name in SOI_NET_INCOME_LESS_LOSS_VARIABLES: + return (component != 0).astype(np.float64) return (component > 0).astype(np.float64) return component