[anth-volk]

Fixes #851

Summary

Registers sim.boundaries.sqlalchemy-and-alembic-only, an enforced mechanical rule, with an ast guard, a proof and a skill doc. Stacked on #807.

Statement: Code connects to databases only through SQLAlchemy engines and connections, never by importing a database driver directly, and runtime code never changes the schema, which changes only through Alembic migrations.

The wording doesn't say which repository's migrations own the schema, so it holds both now and after #847:

• Today: the Public API's Alembic migrations own the only tables this repository uses, the Stage 12 comparison tables (sim.stage12.no-migration-chain, unchanged here).
• After Add the simulation schema's persistence library, migration chain and pre-deploy migration job #847: this repository's own migrations/ chain owns the simulation schema.

How the guard finds violations

projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py runs in the entry project's leg of the Test job. It parses every Python file under projects/*/src (including the executor's src/modal), libs/*/src and .github/scripts: 154 files on this branch. The checks live in tests/database_access_support.py.

It fails on:

• Database driver imports: psycopg, psycopg2, pymysql, MySQLdb, mysql.connector, sqlite3, asyncpg, pg8000, aiosqlite and aiomysql, including submodules and from mysql import connector. A driver named in a SQLAlchemy URL, such as postgresql+psycopg, is not an import.
• alembic imports outside a migrations directory. None exist today.
• Schema changes:
  • create_all and drop_all.
  • create() and drop() on an object the guard can identify as a table: a Table(...) call; a name or attribute bound to one, or annotated as Table, in any scanned file; a __table__ attribute; or an item of .tables. Tables usually come from a shared module, so a table bound in one file is recognised in all of them. A create() or drop() call that passes checkfirst= also counts, since only SQLAlchemy's methods take it. The pandas frame.drop(...) in stage12_artifacts.py is not flagged.
  • SQLAlchemy DDL statements (CreateTable, DropTable, CreateSchema, DDL and the rest), imported from sqlalchemy or used as an attribute. This goes beyond the issue's list: without it, connection.execute(CreateTable(table)) would pass.
  • DDL text in string literals and f-string fragments, matched by the existing DDL_PATTERN. Docstrings and comments don't count.
• migrations directories: files under one may import alembic and change the schema, but driver imports are still rejected there.

A second test fails if any of the named directories stops contributing files to the scan.

What it can't see: imports through importlib or __import__, a table reached through some other local variable, SQL text assembled from separate literals, and shell scripts and SQL files. Today no shell script calls a database client, and the repository has no SQL files.

Widened DDL and Alembic coverage

Check	Before	After
DDL text (DDL_PATTERN)	Stage 12 persistence library, entry and executor src (sim.stage12.no-schema-ddl, whole files)	Also the gateway, policyengine-fastapi, policyengine-simulation-contract, policyengine-simulation-observability and .github/scripts (string literals only)
create_all / drop_all	Substrings in the Stage 12 persistence library (sim.stage12.persistence-typed-dml-only) and create_all( in the three directories above	Calls and attribute references in every source directory
Table.create / Table.drop, SQLAlchemy DDL statements	Not checked	Every source directory
Alembic	Configuration and revision directories only (sim.stage12.no-migration-chain)	Also import alembic in every source directory outside migrations
Database drivers	Not checked	Every source directory

DDL_PATTERN moved from test_stage12_schema_authority.py into database_access_support.py. The Stage 12 test imports it from there and still matches whole files, so sim.stage12.no-schema-ddl is unchanged. There is still one copy of the regex in this repository.

Interaction with #847

#847 adds its own schema-change check: SCHEMA_CHANGE and schema_change_violations() in libs/policyengine-simulation-persistence/tests/ownership_checks.py. Its regex also matches INDEX. That is a second regex covering the same ground, in a different uv project. The two should be combined when #847 is rebased onto this stack.

• The guard passes on the top of the Add the simulation schema's persistence library, migration chain and pre-deploy migration job #847 stack. I ran it against a git archive copy of apiv2/1a-report-tables (Add the simulation report tables #849, 0391726b). It scanned 163 files, including 6 in libs/policyengine-simulation-persistence/src, with 0 findings.
• Why Add the simulation schema's persistence library, migration chain and pre-deploy migration job #847's code passes: its alembic imports are in the root-level migrations/ and in the persistence library's tests/, neither of which this guard scans. Under a migrations directory they would be allowed anyway.
• Expected textual conflicts on that rebase:
  • test_stage12_schema_authority.py: Add the simulation schema's persistence library, migration chain and pre-deploy migration job #847 removes the migration-chain test; this PR changes the file's imports.
  • rules/registry.toml, AGENTS.md, docs/generated/rules.md and changelog_entry.yaml. Rerun python -m scripts.rules render after resolving them.

Out of scope

Non-relational stores: Modal Dicts, object storage such as the Google Cloud Storage artifact and cache buckets, and in-process or on-disk caches.

libs/policyengine-fastapi/src/policyengine_fastapi/database.py builds an SQLite engine with SQLAlchemy create_engine. Nothing in the repository imports it. It passes the rule and is unchanged.

Proof

rules/fixtures/sim.boundaries.sqlalchemy-and-alembic-only/:

• Bad fixture: the real past mistake from 08ebe1c. That commit's Stage 12 adapter in policyengine-simulation-contract did import psycopg and from psycopg.rows import dict_row, and connected with psycopg.connect and cursors, until 7e7fa00 replaced it with a SQLAlchemy engine. The fixture adds a runtime metadata.create_all(engine). The fixture's docstring says the create_all half is illustrative: no commit has called it.
• Good fixture: today's pattern. It uses create_engine with the postgresql+psycopg driver name and a typed select. It also has a docstring that mentions CREATE TABLE, and a pandas frame.drop(columns=...).
• Proof test: test_rule_proof_sqlalchemy_and_alembic_only asserts the exact findings: the two psycopg imports and the create_all call for the bad fixture, and none for the good one.
• Unit tests: test_database_access_support.py covers the scanner's edge cases (31 cases). Its file name doesn't match the registry's guard-test patterns, so the cases aren't rule nodes.

Builds on

• sim.stage12.no-schema-ddl: its DDL_PATTERN is now shared.
• sim.stage12.no-migration-chain: cited for who migrates the Stage 12 tables today; not edited.
• sim.stage12.persistence-typed-dml-only and sim.stage12.deployment-uses-api-schema: the Stage 12 library's typed-DML rules.

Docs

• New skill doc: docs/engineering/skills/database-access.md, the rule's rationale. No existing skill doc covered database access, and the Stage 12 persistence README goes away with that library. It cites the rule and the related rules, says which chain owns which schema today and after Add the simulation schema's persistence library, migration chain and pre-deploy migration job #847, and says non-relational stores are out of scope.
• Links: AGENTS.md's skills list and the skills README point to it.
• Rendered views: docs/generated/rules.md and the AGENTS.md Rules block are regenerated.
• Changelog: one entry in changelog_entry.yaml.

Testing

• uvx ruff@0.15.22 check --extend-select RUF100 .: all checks passed.
• uvx ruff@0.15.22 format --check projects/*/src libs/*/src (CI's scope): 153 files already formatted. The new tests and fixtures are formatted too.
• python -m scripts.rules validate: passed. GITHUB_BASE_REF=microdata-weight-rules python -m scripts.rules guard --base-ref origin/microdata-weight-rules: passed.
• python scripts/run_quality_guards.py: rules-registry and ci-scripts passed.
• python -m unittest discover -s scripts/rules/tests -t .: 115 tests OK.
• Entry project, after uv sync --frozen --extra test --python 3.13: uv run pytest tests/ gave 652 passed. These include the 3 guard tests, the 31 scanner cases and the 4 Stage 12 schema-authority tests.
• On the apiv2/1a-report-tables archive: the guard and scanner tests gave 34 passed. Conftest loading was off, because that branch's conftest needs its own dependencies.
• No inline suppressions were added.

🤖 Generated with Claude Code