[anth-volk]

Fixes #943

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Third PR in the stack, on top of #898 and #900. It adds the command an operator runs once per environment during the one-time migration: it reads the Public API's export of v1 report definitions, creates each report's simulations in simulation.simulations and then the report in simulation.reports, with its year and options and no run, and writes a file mapping each v1 report ID to its v2 report ID. Nothing runs it yet, and it is not an HTTP route. A later item removes it.

cd libs/policyengine-simulation-persistence
uv sync --frozen --extra provisioning
uv run --frozen --extra provisioning \
  python -m policyengine_simulation_persistence.migration_provisioning \
  --definitions report-definitions.json (--output report-mapping.json | --dry-run)

• Input format (defined here; the Public API's export will produce it). Report definitions, schema version 1: strict Pydantic models in migration_provisioning/definitions.py, documented with an example in that package's README. Each definition has the v1 report ID, kind, country, year, options, and its baseline and reform simulations, each with its v1 simulation ID and v2 references. Before anything is written, the file check rejects:

  • missing or unknown fields, and numbers given as text;
  • an economy report without a reform, and simulations of one report on different regions, datasets or households;
  • a v1 report listed twice, and a v1 simulation that two reports describe differently;
  • values the database cannot store: IDs beyond bigint, text holding a NUL character, and NaN or Infinity in the options.

  It builds the stores' NewSimulation and NewReport values, so the tables' own checks also run on every definition.

• Provisioning. For each definition, simulations first (keyed by legacy_simulation_id), then the report (keyed by legacy_report_id), through SimulationStore.create_migrated and ReportStore.create_migrated:

  • An existing row is reused when it matches the definition, so the same export can run again and maps the same reports. A simulation several reports share is created once.
  • Values are compared as the exact decimals JSONB stores, so a float such as 1e23 in the options matches the row it became.
  • Before it creates any of a report's rows, a run reads every row the report needs that already exists and checks it against the definition, so a report that conflicts with an existing row creates nothing. That matters because the runtime role cannot delete rows.
  • A conflicting row, or a definition the database rejects (a policy or household the Public API does not have, one of another country, or another value it rejects as data), fails that report alone, with a one-line reason; the other reports continue. Each row commits on its own, so a report the database rejects after its baseline was created leaves that simulation, and a later run reuses it.

• Dry run. --dry-run only reads. It counts the simulations and reports a run would create and those that already exist, and reports the same conflicts with existing rows. It cannot see the Public API's tables, so it does not find a missing policy or household; the run reports those.

• Connection. The command reads SIMULATION_DATABASE_URL, the runtime role's URL the Simulation Entrypoint uses, and refuses to start unless it logs in as simulation_runtime to a database at the schema revision the code expects. Error messages never repeat the URL.

• Output. A run writes the report mapping (schema version 1, migration_provisioning/mapping.py) to a new file only; it never overwrites one, and it checks that the directory can be written to before it connects. Both modes print a JSON summary. Exit status: 0 provisioned, 1 some reports failed, 2 refused to start (nothing written), 3 the database stopped it partway or the output file could not be written (running again reuses the rows).

Design choices

• The export carries v2 references. Each definition already holds the v2 policy ID that legacy_policy_mappings maps its v1 policy to (null for current law), the v2 household ID from legacy_household_mappings, and the region code from the Public API's region reference files (Publish v2 catalog regions from checked-in reference files and the certified data release policyengine-api#4020). The Public API owns those tables and files, will create any missing v2 policies during the migration, and knows v1's older region spellings (ca for state/ca). So this command reads no Public API table and no reference file, and the runtime role needs no new grant: it keeps no privilege on the Public API's tables. The command checks a region code's form, not that the catalog has it; a code the catalog lacks would fail when the report is first calculated.
• A v1 economy report with one simulation is calculated by v1 as that policy over itself, so the export names the same simulation as baseline and reform, and the report compares it with itself.
• It lives in the simulation persistence library as the migration_provisioning subpackage, with Pydantic in a new optional provisioning extra (the migration job and the library's own package don't import it). That keeps it next to the stores it uses, and lets the existing Simulation database CI job run its PostgreSQL tests without workflow changes. Removing it later means deleting the subpackage, its tests, the extra and one README section.

Environment variables

• SIMULATION_DATABASE_URL, read by the command in the operator's shell.
  • Provider: Google Secret Manager, the secret that sim's GitHub environment variable SIMULATION_DATABASE_URL_SECRET_NAME names (beta for staging, prod for production), in the project named by SIMULATION_ENTRYPOINT_GCP_PROJECT_ID (policyengine-simulation-entry in both, checked 2026-10-27), as the entrypoint deploy resolves it.
  • How it reaches the command: the operator reads it with gcloud secrets versions access and exports it in their own shell (the package README has the commands). No workflow, image or job passes it.
  • Not verified: SIMULATION_DATABASE_URL_SECRET_NAME is not set in either environment yet (checked 2026-10-27), so the secret could not be confirmed. Only the entrypoint's runtime service account is meant to read it, so the operator needs read access to it in each environment first.
• The tests read the existing SIMULATION_TEST_DATABASE_URL and SIMULATION_REQUIRE_DATABASE_TESTS, set as literals in the Simulation database job.

No workflow, Dockerfile or infrastructure file changes.

Waits for

• Give the Simulation Entrypoint its own simulation schema, migration chain and database connection #898 and Add simulation and report definitions to the simulation schema #900, which add the persistence foundation and the two tables; the stack merges bottom up.
• The region reference files, Publish v2 catalog regions from checked-in reference files and the certified data release policyengine-api#4020: this PR doesn't read them, but the export's region codes are theirs.
• Before anyone runs it (not before merging): the operator steps that create sim's database roles and the runtime URL's secret in each environment, read access to that secret for the operator, and the Public API's export command, which will produce this input format and apply the output file.

Testing

Every commit was built test first: the new tests failed for the expected reason, then passed with the code. Each commit then passed the checks below, run locally:

• the persistence library's tests with a PostgreSQL 17 container and SIMULATION_REQUIRE_DATABASE_TESTS=1 (190 passed at the tip), and without a server (106 passed, 84 database tests skipped, as in the Test job);
• the entrypoint's suite, which includes the database-access guard over libs/*/src (723 passed);
• Ruff 0.15.22 format and lint, Pyright through python -m scripts.rules pyright --sync for the library, the quality guards, the rules tool's tests, and uv lock --check for both changed lock files.

Review: three review rounds and one extra round for the code added after them.

• Round 1 found that a value the database rejects as data stopped the whole run instead of failing its report.
• Round 2 found that a float such as 1e23 in the options failed the report it had just created, on every run, because JSONB stores it as an exact decimal; and that a mapping file that could not be written ended the command with a traceback.
• Rounds 3 and the extra round found no bugs.

Each bug was fixed with a test that reproduced it first. The quality findings are fixed too: the file check rejects unstorable values, a conflicting report creates nothing, the output directory is checked first, the dry run's messages and the documentation are corrected, and test coverage, typing and test builders are improved.

Rebased onto #900 at its new head (2026-10-28, on main at 2a9212f) by segment 1B, without conflicts; every commit passed the fast checks (each test file collects, Ruff, the quality guards and the rules guard) and the head the full checks (190 persistence tests with the database required, and 797 entrypoint tests now that the branch includes main's). The checks above were run locally; this push's GitHub Actions runs report their own results.

🤖 Generated with Claude Code