[anth-volk]

Fixes #844

What this changes

Step 1 of the API v2 migration, item 1A-2. This is the first of three stacked PRs: #847 (this one), then #848 (job store), then #849 (report tables).

Ownership rules. The rule sim.stage12.no-migration-chain, which asserted that this repository has no Alembic chain, is removed. A new "Simulation persistence" category in rules/registry.toml replaces it:

• sim.persistence.one-migration-chain: the simulation schema changes only through the Alembic chain in migrations/, configured by migrations/alembic.ini, with its version table at simulation.alembic_version. No other Alembic configuration, environment or revision directory may exist, and runtime source code contains no schema DDL. Proof fixtures included.
• sim.persistence.migration-cycle: CI runs upgrade, alembic check, downgrade and upgrade on PostgreSQL 17 with a non-superuser migrator, and checks that the runtime role cannot change the schema.
• sim.persistence.executor-independent: the Modal executor and gateway never declare, lock or import the persistence library. Proof fixtures included.
• sim.persistence.no-modal-database-credentials: no step that creates a Modal secret or deploys a Modal application receives SIMULATION_DATABASE_URL or SIMULATION_MIGRATION_DATABASE_URL. The Stage 12 comparison's own secret stays the only database credential in Modal. Proof fixtures included.
• sim.persistence.migrate-before-entrypoint: the deploy ordering and secret scoping described below.

Persistence library. libs/policyengine-simulation-persistence holds the schema names, the runtime engine and a revision reader that needs no Alembic. The engine uses psycopg 3 with a small pool and no server-side prepared statements, so it works through either Supabase pooler mode. Alembic is an optional migrations extra; the Simulation Entrypoint image does not install it.

Initial revision (0001). migrations/env.py creates the simulation schema if it is missing (owned by the migrator) and keeps the version table there. Revision 0001 grants simulation_runtime USAGE on the schema, SELECT on the version table, and default SELECT, INSERT, UPDATE privileges on tables (and USAGE, SELECT on sequences) that the migrator creates later.

CI. A new Simulation database job in pr.yml runs the persistence tests marked postgres against a postgres:17 service container. SIMULATION_REQUIRE_DATABASE_TESTS=1 makes a missing server fail the job instead of skipping the tests. The job is in CI complete's needs and has the stack condition. The library is also added to the test matrix and to make test.

Deploy. simulation-deploy.reusable.yml gains a migrate_simulation_database job (needs prepare; GitHub environment beta or prod). It runs .github/scripts/simulation-database-migrate.sh, which applies alembic upgrade head and then requires the head with alembic current --check-heads. deploy_entrypoint now needs this job and runs only if it succeeded. The deploy script mounts SIMULATION_DATABASE_URL from Secret Manager, pinned to the newest enabled version like the existing secrets.

Readiness. When a database is configured, GET /ready adds simulation_database: {reachable, revision, expected_revision}. It does not change status: no old-gateway or Stage 12 route uses this database, and the rollback workflow's smoke check must keep passing when an older image runs against a newer schema. The candidate smoke check in the integration job sets REQUIRE_CURRENT_SIMULATION_DATABASE_REVISION=true, so a deploy fails unless the candidate revision reaches its database as the runtime role and finds the revision its image expects. Without a configured database, the response is unchanged.

Unchanged: the old-gateway pass-through routes, the Stage 12 comparison and the OpenAPI route table. The only OpenAPI change is the /ready response model (two added schemas).

Before merging: operator steps (1A-1) in both staging and production

Merging to main deploys staging and then production, and the deploy fails without these. None of them has been performed.

1. Create the simulation_migrator and simulation_runtime login roles in the staging and production v2 databases. simulation_migrator needs CREATE on the database, or the simulation schema must be created in advance and owned by it. simulation_runtime needs only CONNECT; the migrations grant the rest.
2. Add the migrator's URL (postgresql://...) as the SIMULATION_MIGRATION_DATABASE_URL secret in the beta and prod GitHub environments.
3. In each environment's Simulation Entrypoint Google Cloud project, create a Secret Manager secret holding the runtime role's URL (postgresql://...). Grant roles/secretmanager.secretAccessor only to sim-entry-beta-runtime or sim-entry-prod-runtime, and let the deploy service account list its versions (the deploy script runs gcloud secrets versions list). Set the secret's name as the SIMULATION_DATABASE_URL_SECRET_NAME variable in beta and prod.

Environment variables and secrets

Name	Read by	Provider	How it reaches the reader	Status
SIMULATION_MIGRATION_DATABASE_URL	migrations/env.py, in the migration job	GitHub environment secret (beta, prod)	simulation-deploy.yml and stage12-deploy.yml call the reusable workflow with secrets: inherit → job migrate_simulation_database with environment: ${{ inputs.release_environment }} → env: of the step "Apply simulation schema migrations" only. A test asserts no other step or job carries it.	Wiring checked in the repository. The secret does not exist yet (step 2): unverified
SIMULATION_DATABASE_URL_SECRET_NAME	deploy-cloud-run-simulation-entry.sh	GitHub environment variable (beta, prod)	env: of the step "Deploy Cloud Run entrypoint candidate" → the script resolves <name>:<newest enabled version> → gcloud run deploy --set-secrets SIMULATION_DATABASE_URL=...	Wiring checked. The variable and secret do not exist yet (step 3): unverified
SIMULATION_DATABASE_URL	Simulation Entrypoint runtime (config.py)	Secret Manager, Simulation Entrypoint project	Cloud Run secret environment variable set by the deploy script; readable only with the accessor binding	The accessor binding is step 3: unverified. The candidate smoke check confirms it on the first deploy.

No Modal secret changes.

Checks run

• libs/policyengine-simulation-persistence: uv run pytest tests/ gives 19 passed and 4 skipped without a database; against a local PostgreSQL 17 with SIMULATION_REQUIRE_DATABASE_TESTS=1, 23 passed.
• projects/policyengine-simulation-entry: uv run pytest tests/, 568 passed.
• projects/policyengine-simulation-executor: uv run pytest tests/, 1292 passed and 22 skipped (run on the top of the stack). The first CI run failed test_modal_scripts.py, which requires simulation-*.sh scripts to start with #!/bin/bash; the second commit fixes the new script.
• ruff check . and ruff format --check projects/*/src libs/*/src (ruff 0.15.22), python -m scripts.rules guard --base-ref origin/main, python scripts/run_quality_guards.py (with shellcheck) and python -m unittest discover -s scripts/rules/tests -t ..
• Built the entrypoint image locally. It contains the persistence library and not Alembic. Running it as simulation_runtime against a migrated local database, /ready reported revision 0001.
• Ran .github/scripts/simulation-database-migrate.sh twice against a local PostgreSQL 17 database owned by a non-superuser migrator. The second run changed nothing, and alembic check found no differences.
• Not run locally: the Docker integration tests (test-integration) and the deploy workflow; they need CI or the deployed environments.

Notes for review

• Changes under rules/ need approval from a core developer other than the author (sim.process.rule-machinery-code-owned).
• Only one migration PR in this chain may be open at a time. Add the simulation job store and background result collection through Cloud Tasks #848 and Add the simulation report tables #849 add revisions 0002 and 0003 on top of this one, in that order.
• Likely conflicts: Register weighted-microdata rules, weight the builder tests, and check change weights and household counts #807 (rules registry and its rendered views), and Enable ruff argument, idiom and pytest-style rules (ARG, B, C4, FBT, PERF, PLW, PT, PTH, RUF, SIM and others) #802 and Refactor within ruff's complexity limits, then select every ruff rule (ALL) #804 (ruff rule families, which edit app.py, config.py and tests that this PR also edits). Whichever merges second rebases.

🤖 Generated with Claude Code