Rake the EITC child-split bins to their total rows and AGI groups

[MaxGhenis]

Depends on #1227. This branch sits on #1227's head (f44d26e), so until #1227 merges the diff also shows its two commits. This PR's own change is the last commit.

Methodology call for Max. No published source decides between the options below, so this is queued as a decision rather than merged on the usual gates. The recommendation, option (c), is implemented here and ready to merge. Options (b) and (d) are each a few lines if preferred.

What is inconsistent after #1227

_uprate_cross_period_eitc_decompositions scales the TY2023 SOI Table 2.5 EITC-by-AGI-and-children rows to the TY2024 filing-season EITC by AGI:

• each bin takes its AGI group's active / source ratio, so every AGI group keeps its TY2023 child mix;
• each split's all-AGI total row takes the all-AGI ratio (×1.0418 amount, ×0.9753 returns), so the totals keep the TY2023 all-AGI child mix.

The AGI groups grew at different rates (amount ×0.85 at $5,000–$10,000 to ×1.73 at $50,000 and over; returns ×0.81 to ×1.38). So per split, the total row and the sum of its 28 bins disagree, and both levels sit on national_state and full:

Split	Amount: total / bins	Returns: total / bins
Childless	1.157	1.062
One child	1.018	1.001
Two children	0.975	0.961
Three or more	0.993	0.951

The options, on the pinned feed

These are release-equivalent compiles of consumer_facts_us_51aa40f.jsonl, from experiments/eitc-child-split-rake-20261012/results/.

	(a) status quo	(b) totals := Σ bins	(c) rake (this PR)	(d) drop totals
What moves	nothing	8 total rows, ×0.864–1.052	224 bins, ×0.857–1.226; totals keep their values	8 total rows removed
Total row = Σ its bins	no (0.951–1.157)	yes	yes	n/a
Childless share of EITC, amount / returns	3.71% / 27.73% (totals), 3.21% / 26.11% (bins)	3.21% / 26.11%	3.71% / 27.73%	3.21% / 26.11%
national_state	6,102 at 3fad14f85468	6,102 at a211cbec63ec	6,102 at f96aac657fa6	6,094 at 41a1fa2b013f
full	33,250 at c3aaea9c1e8c	33,250 at a83fc4bad02f	33,250 at cb0eaa0418f2	33,242 at 469b8ed2c5db
Table 2.5 share of loss weight, national_state / full	1.555% / 0.605%	1.555% / 0.605%	1.548% / 0.603%	1.272% / 0.494%
Table 2.5 rows' weights vs (a)	—	×0.930–1.026	×0.926–1.101	×1.0003–1.0029
Largest weight move of any other row	—	8×10⁻⁶	1.5×10⁻⁴	2.9×10⁻³

• Totals. All four conserve the active EITC ($69.04B, 23.84M returns).
• Scope. None changes any target outside Table 2.5.
• (b) vs (d). They carry the same information, the bins' child mix. (b) adds weight on the bins' sums; (d) drops the totals' weight.

Why (c)

1. No TY2024 child mix is published (checked live 2026-10-12).
   • SOI Table 2.5 stops at TY2023: 24in25ic.xls returns 404.
   • SOI preliminary data stop at TY2022; the geographic data with EITC by children stop at TY2023.
   • The TY2024 filing-season file and IRS EITC Central give EITC by AGI or by state only.
2. Backtest on published Table 2.5, TY2013–2023. Each option uprates TY s to TY s+1, as the compiler does, and is scored against the published TY s+1 table. The active margins are either SOI's mid-November file (as production uses) or the true TY s+1 margins. There are seven year pairs without a structural change to the credit, scored in 11 runs (four pairs against both kinds of margins).
   • Keeping the all-AGI mix gave mean split-total errors of 2.9% (amount) and 2.1% (returns) on mid-November margins. The bins gave 5.4% and 5.0%.
   • On true margins it was 2.1% and 1.6%, against 3.0% and 2.7%.
   • The bins under-predicted childless EITC in all 11 runs (by 1.1–19.7%).
   • On split × AGI-group cells, (c) was as good or better on average.
   • The exceptions are TY2015→16, and TY2022→23 amount, when childless EITC really fell.
   • On individual bin rows it is roughly a wash: (c)'s sqrt-weighted capped error is 0.1299 against the bins' 0.1291 for amount, and 0.1204 against 0.1226 for returns, on mid-November margins. The case for (c) is national totals and consistency; no calibration solve here shows release quality improving.
3. Consistency with the rest of the surface. national_state already carries SOI Historic Table 2 TY2022 state EITC-by-children rows. Their national child mix is 0.55–0.63 pp TVD from (c) and 1.62–2.24 pp from the bins' mix in (b) and (d).
4. The mix (c) keeps is confirmed. SOI's full-population TY2023 state data (August 2026; not yet in the pinned feed) give childless 3.68% of the amount and 27.80% of returns. Table 2.5 TY2023 gives 3.71% and 27.73%. The bins project a further fall to 3.21% and 26.11% that no published source supports.

Full write-up: experiments/eitc-child-split-rake-20261012/README.md.

The change

• _rake_eitc_child_splits runs after the per-row uprating. For each decomposition (kind, source period, target period), it builds the child-split × AGI-group matrix of source bins (_eitc_rake_plan) and scales it (_rake_two_margins) to the splits' uprated total rows and the active AGI groups.

• _rake_two_margins finds the IPF solution: the unique positive scaling r_i·s_j·x_ij that meets both margins and keeps every cross-product ratio of the source.

  • It first checks whether that scaling exists (_require_scalable, the Menon 1968 / Brualdi 1968 condition, over subsets of the shorter side, at most 16) and refuses only when the margins are certainly infeasible: an excess beyond rounding, or an exact fill that other splits also reach. Margins within rounding of the boundary go to the solver.
  • It then takes damped Newton steps on the convex dual. The seed is normalized by its largest cell, the largest column of each connected support block is pinned, and the solve is Jacobi-scaled, with Levenberg–Marquardt damping when rounding makes it singular.
  • It finishes with a few exact IPF sweeps. Plain IPF needed more than 10,000 sweeps on ill-conditioned inputs (review round 1); this converges in a few steps.
  • Rows hold to rounding and columns to 1e-12 relative.

• When it rakes. A decomposition is raked when:

  • every split has one all-AGI total row and leaf bins that tile AGI;
  • and the active AGI bins give the total rows and every bin an AGI group.

  Otherwise it keeps its per-row factors: uprated on the all-returns total, every row already has one factor, and with no total row there is nothing to reconcile to. A group with no EITC in either year takes a zero factor.

• Refusals (ValueError):

  • negative EITC;
  • margins that disagree in total;
  • an AGI group with active EITC but no source EITC in any split;
  • a split with no EITC in any active group;
  • margins no positive scaling can meet;
  • more than 16 splits and more than 16 groups;
  • a published row over a group that is zero although a bin inside it is not.

• Metadata.

  • Every raked row carries uprating_reconciliation: two_margin_rake.
  • Each bin also records uprating_factor (its effective factor), uprating_agi_group_factor (the factor the per-row uprating recorded) and uprating_child_split_adjustment. The adjustment is relative to the recorded factor, so the group factor times the adjustment equals the bin's factor.
  • Total rows keep their Count each EITC child split once in the Table 2.5 uprating source total #1227 value and factor. A subtotal row, if a feed ever has one, becomes the sum of its raked bins.

• Refactor. The per-row AGI metadata moved into _eitc_agi_uprating_metadata, with no change for Count each EITC child split once in the Table 2.5 uprating source total #1227's rows.

Invariants

These hold for every decomposition the rake applies to:

1. Each split's bins sum to its total row.
2. Each active AGI group's bins (all splits) sum to the group's active EITC.
3. Conservation: the total rows sum to the active EITC, and so do the bins.
4. The total rows keep Count each EITC child split once in the Table 2.5 uprating source total #1227's all-AGI factor, active / source total rows.
5. Every bin of one split in one AGI group takes one factor, so the source shape inside the cell is kept.
6. The source's cross-product ratios between splits and groups are preserved. This is what makes it IPF rather than some other reconciliation.
7. Uniform growth across groups makes the rake the identity.
8. _rake_two_margins succeeds exactly when a positive scaling exists.

Tests

• test_us_eitc_decomposition_uprating.py (36 tests).
  • Example tests:
    • Count each EITC child split once in the Table 2.5 uprating source total #1227's total-row doubling;
    • a 2×2 rake against the hand solution of a² + 2,900a − 800,000 = 0;
    • a subtotal row;
    • a group with no EITC in either year;
    • a group with active but no source EITC, which is refused;
    • strict slack below rounding, which is raked (review round 2);
    • a zero subtotal over positive bins, which is refused (review round 2);
    • an ill-conditioned decomposition;
    • the adjustment against a rounded subtotal;
    • the no-total-row and all-returns cases, which are left unraked;
    • Count each EITC child split once in the Table 2.5 uprating source total #1227's overlap refusal;
    • eight _rake_two_margins refusals;
    • nine feasible ill-conditioned inputs, including both rounds' counterexamples (a singular Newton system, a seed whose sum overflows).
  • Hypothesis properties (150 examples each):
    • invariants 1–6 and invariant 7, through compile_us_fiscal_target_registry;
    • invariant 8, against an independent scipy.optimize.linprog oracle (the largest t such that some matrix on the seed's support, every positive cell ≥ t, meets both margins). It also checks margins and cross-product ratios;
    • positive seeds spanning ten orders of magnitude.
  • A differential test of _rake_two_margins against the closed-form 2×2 solution (bisection on the cross-product ratio).
• test_us_fiscal_targets.py.
  • New test_pinned_feed_eitc_child_split_bins_sum_to_their_total_rows: per split Σ bins = total row; per active group Σ bins = the group's EITC; adjustments within 0.85–1.23.
  • Re-pinned national_state 3fad14f85468 → f96aac657fa6.
• Local runs. The pinned-feed tests skip in CI because the feed is absent, so they were run locally:
  • test_us_eitc_decomposition_uprating.py: 36 passed;
  • test_us_fiscal_targets.py and test_us_fiscal_refresh_builder.py with -k "eitc or pinned_feed or decomposition": 25 passed;
  • test_us_spine_blindness.py: 498 passed. Round 1's CI failed this static check on lambda row: row[1:].

Separate findings, not changed here

• Mid-November totals vs complete year. The mid-November filing-season EITC totals mostly run 3–6% below complete-year Table 2.5 (TY2017–2023: returns 0.939–0.971; amount 0.957–0.973, except TY2018 at 1.012). They are the TY2024 EITC target levels and two release critical-fit gates. All four options conserve them, so this is a separate question, proposed as a follow-up.
• CD file vintage. 22incd.csv is SOI TY2022 data that Chronicle stamps TY2023, already tracked as CD record sets stamp 22incd.csv as ty2023 — a tax-year vintage error; add Table 1.4 taxable-interest facts (populace#489) chronicle#117.
• TY2023 state data. SOI's TY2023 state data (23in55cmcsv.csv, with EITC by children) are not yet in the pinned feed; Add IRS SOI Historic Table 2 TY2023 national, state broad and state EITC facts chronicle#295 is adding them. Their child mix sits 0.19–0.24 pp TVD from (c), so they would widen the gap to (b) and (d).

Review

• Round 1 (GPT-6.1 Sol, subfleet run --task review --tier hard, head a2a529d): REQUEST_CHANGES.
  • Two majors:
    • a group with no EITC in either year blocked the rake;
    • the IPF iteration cap rejected feasible, ill-conditioned margins.
  • Three minors:
    • the adjustment against rounded subtotals;
    • option (b)'s provenance;
    • qualifying the README claims.
  • It favoured (c) once those were fixed. All five are addressed above.
• Round 2 (head 9d94deb): REQUEST_CHANGES.
  • All five round-1 findings were fixed or partly fixed.
  • One major: the feasibility tolerance refused strict slack below rounding as an exact fill.
  • Three minors: extreme magnitudes (a singular Newton system, an overflowing seed sum); no exact check above 16 splits; a zero subtotal over positive bins.
  • All addressed above.
• Round 3: pending.

axiom: n/a: target-compilation methodology for calibration targets (no policy encoding)

🤖 Generated with Claude Code

[MaxGhenis]

A source anomaly that bears on this rake.

IRS's own TY2023 Table 2.5 (23in25ic.xls, sha256 c6a58f63…, byte-identical to the live file on 2026-10-14) misplaces one child-split cell pair. The three-or-more $25k–$30k cell holds 635,160 returns and $3.81B, and the $30k–$35k cell is 0.

• The evidence. In every other row, the all-returns column J equals Z+AP+BF+BV. In rows 31–33 the gap is −1,187, −310,212 and +311,398.
  • All-returns minus the other splits implies three-plus counts of about 324,948 (25k–30k) and 311,398 (30k–35k).
  • Each split's bins still add to its total row. TY2022 has no such anomaly.
• The uprating effect. The 25k–30k group's source is inflated, so every split's 25k–30k bin takes 0.832 (returns) instead of about 0.966. The 30k–40k group takes 1.030 instead of about 0.948.
• In the release. In populace-us-2024-spm-20260915, calibration cut three-plus EITC returns at 30k–35k from 174,894 to 16,231.
• Why it matters here. Raking the child × AGI matrix to the total rows and AGI groups would carry the misplaced cell into the raked bins. The repair, or a guard for it, is being split into its own task. It may need Chronicle to extract the all-returns AGI rows (B/J/K, rows 10–37) so microcosm can check child splits against all-returns per row.

Context: microcosm#1274, which adds CI tests on the real TY2023 rows after #1227.

🤖 Generated with Claude Code