Skip to content

Preserve supplied formula inputs in DataFrame exports - #2498

Merged
juaristi22 merged 1 commit into
mainfrom
fix-2297-formula-input-export
Oct 20, 2026
Merged

juaristi22 merged 1 commit into
mainfrom
fix-2297-formula-input-export

Conversation

@juaristi22

Copy link
Copy Markdown
Collaborator

Fixes #2297

Default input exports dropped dataset-supplied values for variables with formulas. In the reported example, £9,500 of State Pension became £0 after export and reload. Export surviving supplied arrays, including formula and adds inputs, while excluding calculated caches and model-generated values. Preserve period/entity handling, identifiers and the existing include_computed_variables=True path.

Reproducer: policyengine_uk/tests/test_input_dataframe_export.py::test_dataset_formula_input_round_trip_preserves_final_pension

Validation: 51 distinct export, DataFrame and input-reset cases pass independently on each Core version: installed 3.32.27 and an official-wheel 3.33.2 package overlay. The pension regression reproduces £9,500 → £0 before the fix and £9,500 → £9,500 afterward. Native formatting/lint passes; independent code and engineering/source reviews completed with no findings or material gaps. Runtime: Python 3.13.14 and microdf 1.2.1; full-lock environment validation was not run.

axiom: n/a — input serialization/provenance engineering; policy formulas and parameters are unchanged. No encoded-parity claim.

@juaristi22
juaristi22 marked this pull request as ready for review October 18, 2026 11:28

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right, fine to approve.

@juaristi22, the default export now keeps supplied values for variables with formulas. The state_pension_reported round trip in #2297 keeps £9,500, and include_computed_variables=True still goes through Core unchanged. Core 3.33.2's own to_input_dataframe excludes formula variables by design, so overriding it in UK is the right place for the fix.

  1. should: policyengine_uk/utils/input_dataframe_export.py:8-31, 39, 62 adds new reads of Core private state: holder._user_input_storage, _memory_storage, _disk_storage, _stores_user_input, simulation._get_visible_branch_names and, when present, simulation._get_recorded_input_array. #2175 (and #2495 for the parameter side) is moving the UK package off exactly these fields. Please either add this module to #2175's checklist, or raise the Core floor to 3.33.2 and drop _legacy_recorded_array. Otherwise the fallback will copy Core internals that can change underneath it.
  2. nit: _legacy_recorded_array mirrors Core 3.33.2's tier precedence by hand (memory then disk, skipping derived values). A test that runs both paths on the same fixture and asserts the same arrays would catch drift. If point 1 drops the fallback, this goes too.

What I checked

  • Read the full diff and issue #2297, and compared the code with Core 3.33.2's _get_recorded_input_array and to_input_dataframe.
  • pytest test_input_dataframe_export.py test_dataframe_dataset.py: 103 passed (locked Core 3.33.2).
  • Reverted simulation.py to main. test_input_dataframe_export.py then fails 14 of 25.
  • ruff check and ruff format --check on the changed Python files: clean.
  • Hosted CI on this head: all green.
  • Not checked: the Core 3.32.27 fallback path, since my environment has only the locked 3.33.2.

🤖 Generated with Claude Code

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, per my review above. The shoulds and nits can follow.

@juaristi22
juaristi22 added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 3c0e48d Oct 20, 2026
15 checks passed
@juaristi22
juaristi22 deleted the fix-2297-formula-input-export branch October 20, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_input_dataframe() drops dataset inputs for variables with formulas, such as state_pension_reported

2 participants