SimulationBuilder.build_from_dict should clear dataset provenance when rebuilding from a situation #627

Found while adopting PolicyEngine/policyengine-uk#1831 (UK hub, 2026-10-20).

PolicyEngine Core version: 3.33.2. Reproduced with synthetic MOCK data and the UK tax-benefit system, without private datasets or downloads.

SimulationBuilder.build_from_dict(system, situation, simulation) supports overwriting an existing simulation. When that simulation was built over a dataset, the method replaces its population structure and inputs but leaves is_over_dataset=True and retains dataset. It also leaves situation_input=None. Its clones and newly created branches inherit this stale provenance.

Reproducer

import pandas as pd

from policyengine_core.simulations import Simulation
from policyengine_core.simulations.simulation_builder import SimulationBuilder
from policyengine_uk.system import system

year = 2026
frame = pd.DataFrame(
    {
        f"person_id__{year}": [1],
        f"person_benunit_id__{year}": [1],
        f"person_household_id__{year}": [1],
        f"benunit_id__{year}": [1],
        f"household_id__{year}": [1],
        f"age__{year}": [70],
    }
)
situation = {
    "people": {"adult": {"age": {year: 70}}},
    "benunits": {"family": {"members": ["adult"]}},
    "households": {"home": {"members": ["adult"]}},
}
simulation = Simulation(tax_benefit_system=system, dataset=frame)
original_dataset = simulation.dataset
builder = SimulationBuilder()
builder.default_period = year
rebuilt = builder.build_from_dict(system, situation, simulation)

assert rebuilt is simulation
print(rebuilt.persons.ids)                   # ['adult']
print(rebuilt.is_over_dataset)               # True (expected False)
print(rebuilt.dataset is original_dataset)    # True (expected cleared)
print(rebuilt.situation_input)               # None (no new situation marker)
print(rebuilt.clone().is_over_dataset)        # True (expected False)
print(rebuilt.get_branch("rebuilt").is_over_dataset)  # True (expected False)

Expected behavior

Rebuilding from a situation should reset is_over_dataset, clear the previous dataset, and record the situation as the active source. The behavior should cover both build_from_entities and build_from_variables, including clones and branches derived after the rebuild. A canonical source marker or population-rebuild hook would also let subclasses keep their own provenance in sync.

Source finding and downstream effect

In Core 3.33.2, Simulation.__init__ sets is_over_dataset (simulation.py:801) and situation_input (simulation.py:897). SimulationBuilder.build_from_dict (simulation_builder.py:66–96) only dispatches the build and refreshes input_variables; build_from_entities (98–192) updates population counts, IDs, memberships and holder inputs without recording a source change. The builder's population bookkeeping is not retained as an independent source marker on the simulation. Population IDs or input variable names are not reliable provenance: either source can use the same IDs and variables, and variable-only situations also use NumPy ID arrays.

The PE-UK provenance probe classifies 20 of 24 routes correctly. Its four failures are a data-built core simulation overwritten by this builder, its clone, its new branch, and a data-built UK simulation overwritten directly by the core builder. PE-UK's supported Simulation.build_from_situation rebuild path resets its own source marker correctly. No supported production caller found by the UK hub uses these four direct core-builder overwrite routes.

This matters for model formulas that distinguish survey runs from explicit household situations. PE-UK currently documents the direct overwrite limitation instead of guessing from IDs, weights or population size.

🤖 Generated with Claude Code