Skip to content

Import logic from TaxData repo that creates and tests generation of growfactor.csv file - #543

Merged
martinholmer merged 3 commits into
masterfrom
growfactors
Sep 6, 2026
Merged

martinholmer merged 3 commits into
masterfrom
growfactors

Conversation

@martinholmer

@martinholmer martinholmer commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

What this branch does

Brings the growth-factors generation logic into TMD from the TaxData repo
(puf_stage1/), so that TMD produces its own growfactors.csv instead of
vendoring a file generated elsewhere. Two commits: one preparing the
make data wiring, one importing the pipeline.

make data changes

  • New tmd/create_raw_growth_factors.py reads
    tmd/storage/input/taxdata<VINTAGE>_growfactors.csv and writes
    tmd/storage/output/growfactors.csv.
  • tmd/create_taxcalc_growth_factors.py now reads that output file rather
    than reaching into storage/input itself; the TAXYEAR -> input file
    map moved into the new script.
  • Makefile: the tmd_growfactors.csv rule runs the new script first;
    logs are split into make_data_raw_growfactors.log and
    make_data_tmd_growfactors.log; clean removes growfactors.csv; the
    data target diffs both new/old growfactors files.

Net effect on make data output: none. The committed
taxdata26_growfactors.csv changes only in line endings (CRLF -> LF);
values are byte-identical otherwise.

New pipeline: tmd/growfactors/

data/ (Census population, CBO baseline, IRS return projection,
       SOI estimates, benefit programs)
  | build_stage1_factors.py     -> data/Stage_I_factors.csv   [level indexes, SYR=2011]
  | build_growfactors.py        -> storage/input/taxdata26_growfactors.csv
  | create_raw_growth_factors.py   -> storage/output/growfactors.csv       [make data]
  | create_taxcalc_growth_factors.py -> storage/output/tmd_growfactors.csv [make data]
  • Stage 1 extrapolates SOI aggregates past SOI_YR using Census
    population, CBO baseline, and IRS return-projection growth rates, then
    normalizes to a level index with 2011 = 1.
  • Stage 2 converts to per-capita, turns level indexes into
    one-plus-annual-proportional-change factors, appends benefit growth
    factors, and drops variables Tax-Calculator does not use. Takes
    --vintage so a data refresh writes a new vintage file rather than
    rewriting a frozen one.
  • tmd/growfactors/Makefile wires the stages: stage1, factors,
    test, all. Most users never run it.
  • update/update_cbo.py, update/update_soi.py,
    update/cbo_chained_cpiu.py refresh the source data from live CBO/BLS/
    SSA/IRS sites. Never invoked by make data, make all, or any test;
    gated behind the new pip install -e ".[growfactors-update]" extra
    (requests_html, jinja2) and run by hand via make update-cbo /
    make update-soi.
  • Docs: README.md (pipeline overview), docs/annual_update.md (refresh
    procedure), plus the CBO/SOI updating instructions carried over from
    TaxData. .gitignore now un-ignores tmd/growfactors/{data,docs}/*.csv.

Tests

  • tests/test_growfactors.py — ported value/range checks on the raw
    growth factors file used for TAXYEAR; runs in the normal make test.
  • tests/growfactors_pipeline/test_growfactors_pipeline.py —
    reproducibility tests that re-run each stage into a tmpdir and compare
    against the committed artifact, catching source data edited without
    regenerating downstream files.

Intentional differences from the TaxData original

  • Stage_II_targets.csv output dropped — it served only TaxData's stage-2
    weighting; TMD reweights against tmd/storage/input/soi.csv. The
    in-memory aggregates DataFrame (TaxData's Stage_II_targets) is kept
    because stage 1 factors are computed from it.
  • Benefit growth factors are padded with 1.0 past the last year in
    benefit_growth_rates.csv; TaxData silently writes empty values for
    2032-2036.
  • Two chained assignments that are no-ops under pandas copy-on-write were
    rewritten (see the NOTE comments).
  • update_soi.py no longer updates the CPS copy of SOI_estimates.csv,
    which has no TMD counterpart.

@martinholmer
martinholmer marked this pull request as draft September 5, 2026 14:58
@martinholmer martinholmer changed the title Include TaxData logic for creating raw output/growfactor.csv file Include TaxData logic for creating/using raw output/growfactor.csv file Sep 5, 2026
@martinholmer martinholmer changed the title Include TaxData logic for creating/using raw output/growfactor.csv file Import logic from TaxData repo that creates and tests generation of growfactor.csv file Sep 6, 2026
@martinholmer
martinholmer marked this pull request as ready for review September 6, 2026 19:13
@martinholmer
martinholmer merged commit 367f50a into master Sep 6, 2026
1 check passed
@martinholmer
martinholmer deleted the growfactors branch September 6, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant