Repository navigation
examples/us_budgetary_impact.py consumes excessive memory (peak ~7.5 GB RSS / 82 GB VMS) #328
Description
Reported in the discussion on #264 (JOSS paper PR).
When run on the joss-paper branch (4.2.1), examples/us_budgetary_impact.py exhibits very high memory pressure during the baseline + reform simulations on the enhanced CPS 2024 dataset (2026, ~101k people / 41k households):
- Vahid: OOM-killed at peak ~30 GB RSS on a Mac (rc=137).
- Anthony (instrumented run): peak 7.48 GiB RSS, 82.35 GiB VMS;
baseline_sim.run()546.6s wall,reform_sim.run()596.5s wall; VSCode crashed mid-run from memory pressure.
This makes the canonical US population example impractical for normal local development and for JOSS reviewers attempting to reproduce results.
Suggested next steps:
- Profile
baseline_sim.run()andreform_sim.run()to find the dominant allocations (likely inpolicyengine-uscore, but worth confirming). - Determine whether intermediate arrays can be released between runs or whether the two simulations can share underlying data.
- Document peak memory requirement in the example header until the footprint is reduced.
Related: #264
Activity
MaxGhenis commentedon Jul 7, 2026
Measurements from a 2026–2035 run on the certified national dataset (policyengine 4.18.10, populace_us_2024 sparse 57k build, M-series macOS, Python 3.13):
ensure()'s LRU pinning everySimulationobject for the life of the process (Simulation cache is write-only: uuid4 keys mean ensure() never reloads results, and data_folder accumulates orphaned h5 files #455), partly macOS RSS accounting overstating vs LinuxTwo implications for this issue: (1) the memory pressure isn't specific to
economic_impact_analysis— a plain per-yearensure()loop shows the same growth, so #455's cache-what-you-need fix would help both; (2) docs/microsim.md's "roughly 4 GB of memory and takes 15-30 seconds on a laptop" doesn't match the certified default today — worth re-truing in whichever direction (perf fix or doc fix).