Require SQLAlchemy-only database connections and Alembic-only schema changes

AGENTS.md

@@ -22,6 +22,9 @@ When changing code that aggregates simulation microdata (the output builders,
 the national reduce, Stage 12 aggregation, or anything that reads
 `output_dataset.data`), read `docs/engineering/skills/microdata-weights.md`.
 
+Before adding code that connects to a database, declares tables, or changes a
+database schema, read `docs/engineering/skills/database-access.md`.
+
 When adding or changing a GitHub Actions workflow step or a script under
 `.github/scripts/`, read `docs/engineering/skills/ci-workflows.md`.
 
@@ -97,6 +100,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 ### Service and package boundaries
 
 - `sim.boundaries.gateway-executor-decoupled`: The gateway and executor never import each other; they communicate only through the models and state helpers in policyengine-simulation-contract. (doc-only, not checked: follow it yourself)
+- `sim.boundaries.sqlalchemy-and-alembic-only`: Code connects to databases only through SQLAlchemy engines and connections, never by importing a database driver directly, and runtime code never changes the schema, which changes only through Alembic migrations. (blocking: pytest `projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py`)
 - `sim.boundaries.unit-tests-need-no-services`: Unit tests never require real network credentials, Modal, Hugging Face, GCP or deployed services; those seams are mocked. (doc-only, not checked: follow it yourself)
 
 ### Stage 12 persistence

changelog_entry.yaml

@@ -1,6 +1,7 @@
 - bump: patch
   changes:
     added:
+      - Added the rule sim.boundaries.sqlalchemy-and-alembic-only, with proof from the psycopg adapter of 08ebe1ca; code connects to databases only through SQLAlchemy, never by importing a database driver, and runtime code never changes the schema, which changes only through Alembic migrations; a guard parses every file under projects/*/src, libs/*/src and .github/scripts and rejects driver imports, alembic imports outside a migrations directory, create_all and drop_all, create and drop on tables, SQLAlchemy DDL statements and DDL text in string literals, which extends the Stage 12 DDL check's pattern to the gateway and every library
       - Made the budget's reform-minus-baseline totals log a warning when the baseline and reform household frames carry different weights, since the change is then weighted by the reform's weights alone
       - Added weighted-microdata rules to the registry, with proof from the #767 household count; the executor's code never aggregates a weight column as a value, reads raw values from weighted frames only in simulation_output_common.py (the UK local-authority breakdown's plain household frame is its one recorded exception there, in a list that may only shrink), and its output-builder tests build entity frames through one shared factory with non-uniform weights, while a doc-only rule records that weights, values, labels and counts each allow different operations
       - Made CLAUDE.md and the Copilot instructions thin pointers to AGENTS.md, required workflow logic longer than one line to live in tested and shellchecked scripts, and registered the tracking-issue rule

docs/engineering/skills/README.md

@@ -11,6 +11,8 @@ Current skills:
 
 - `ci-workflows.md`: one line of code per workflow `run:` block, tested
   workflow scripts, and shellcheck.
+- `database-access.md`: SQLAlchemy-only database connections, schema changes
+  only through Alembic migrations, and the guard that checks both.
 - `github-prs.md`: same-repository draft PR workflow, issue linkage, PR head
   verification, and title conventions.
 - `testing.md`: test placement, dependency boundaries, and expected validation

docs/engineering/skills/database-access.md

@@ -0,0 +1,80 @@
+# Database Access Skill
+
+Read this before adding code that connects to a relational database, declares
+tables, or changes a database schema.
+
+## Rules
+
+`sim.boundaries.sqlalchemy-and-alembic-only` has two parts:
+
+- **Connections go through SQLAlchemy.** Code connects to databases only
+  through SQLAlchemy engines and connections, never by importing a database
+  driver directly. Select the driver in the URL's driver name instead, as the
+  Stage 12 store does with `postgresql+psycopg`.
+- **Runtime code never changes the schema.** Schema changes happen only
+  through Alembic migrations, never through `create_all`, `drop_all`,
+  `Table.create`, `Table.drop`, SQLAlchemy DDL statements or DDL text in
+  runtime code.
+
+It builds on the Stage 12 rules:
+
+- `sim.stage12.no-schema-ddl`: the simulation runtimes and the Stage 12
+  persistence library contain no schema DDL. Its `DDL_PATTERN` regex is the
+  one this rule matches against every source directory's string literals.
+- `sim.stage12.no-migration-chain`: this repository has no migration chain
+  of its own; policyengine-api migrates the Stage 12 tables.
+- `sim.stage12.persistence-typed-dml-only` and
+  `sim.stage12.deployment-uses-api-schema`: the Stage 12 persistence
+  library uses typed SQLAlchemy Core DML only, with no raw SQL text.
+
+## Who owns the schema
+
+- **Today**, policyengine-api's Alembic migrations own the only tables this
+  repository uses, the Stage 12 comparison tables
+  (`sim.stage12.no-migration-chain`). This repository's code reads and
+  writes their rows, and deployment validation compares their live schema
+  with its SQLAlchemy mappings; nothing here changes them.
+- **Once #847 merges**, this repository's own Alembic chain at `migrations/`
+  owns the `simulation` schema, and the Stage 12 tables stay with
+  policyengine-api.
+
+The rule holds either way: whichever chain owns a table, only that chain's
+migrations change it.
+
+## What the guard checks
+
+`projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py`
+parses every Python file under `projects/*/src` (including the executor's
+`src/modal`), `libs/*/src` and `.github/scripts`, and runs in the entry
+project's `Test` job. The checks live in `database_access_support.py` next to
+it. It fails on:
+
+- an import of a database driver: `psycopg`, `psycopg2`, `pymysql`,
+  `MySQLdb`, `mysql.connector`, `sqlite3`, `asyncpg`, `pg8000`, `aiosqlite`
+  or `aiomysql`, or any of their submodules;
+- an import of `alembic` outside a `migrations` directory;
+- a `create_all` or `drop_all` call;
+- a `create()` or `drop()` call on a table: a `Table(...)` call, a name or
+  attribute bound to one or annotated as `Table` in any scanned file, a
+  `__table__` attribute or an item of `.tables`, or any `create()` or `drop()`
+  call passing `checkfirst=`. Other `drop()` calls, such as pandas'
+  `frame.drop(columns=...)`, are not schema changes;
+- SQLAlchemy's DDL statements (`CreateTable`, `DropTable`, `CreateSchema`,
+  `DDL` and the rest);
+- DDL text in a string literal: `CREATE`, `ALTER`, `DROP` or `TRUNCATE` of a
+  table, type, schema or database. Docstrings and comments don't count.
+
+Files under a `migrations` directory may import alembic and change the schema,
+but still connect through SQLAlchemy.
+
+The guard reads source, not types. It misses imports through `importlib` or
+`__import__`, a table reached through some other local variable, and SQL text
+assembled from separate literals. It doesn't read shell scripts or SQL files;
+today no shell script calls a database client, and the repository has no SQL
+files.
+
+## Out of scope
+
+The rule covers relational databases only. Non-relational stores are out of
+scope: Modal Dicts, object storage such as the Google Cloud Storage artifact
+and cache buckets, and in-process or on-disk caches.

docs/generated/rules.md

@@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 
 | Tier | enforced | ratcheted | advisory | doc-only | Total |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| mechanical | 42 | 2 | 0 | 15 | 59 |
+| mechanical | 43 | 2 | 0 | 15 | 60 |
 | metric | 0 | 0 | 0 | 0 | 0 |
 | judgment | 0 | 0 | 0 | 0 | 0 |
 
-**Blocking:** 44 of 59 rules.
+**Blocking:** 45 of 60 rules.
 
 ## Gaps
 
@@ -77,14 +77,20 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 | Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
 | --- | --- | --- | --- | --- | --- | --- | --- |
 | `sim.boundaries.gateway-executor-decoupled` | The gateway and executor never import each other; they communicate only through the models and state helpers in policyengine-simulation-contract. | mechanical | doc-only | no | nothing | — | — |
+| `sim.boundaries.sqlalchemy-and-alembic-only` | Code connects to databases only through SQLAlchemy engines and connections, never by importing a database driver directly, and runtime code never changes the schema, which changes only through Alembic migrations. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py` | `pr.yml:test` | yes |
 | `sim.boundaries.unit-tests-need-no-services` | Unit tests never require real network credentials, Modal, Hugging Face, GCP or deployed services; those seams are mocked. | mechanical | doc-only | no | nothing | — | — |
 
 ```mermaid
 flowchart LR
   r_sim_boundaries_gateway_executor_decoupled("sim.boundaries.gateway-executor-decoupled (doc-only)")
   gap_unenforced>"not enforced"]
+  r_sim_boundaries_sqlalchemy_and_alembic_only("sim.boundaries.sqlalchemy-and-alembic-only (enforced)")
+  e_pytest_projects_policyengine_simulation_entry_tests_test_database_access_boundaries_py["pytest projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py"]
+  j_CI_pr_yml_test["CI pr.yml:test"]
   r_sim_boundaries_unit_tests_need_no_services("sim.boundaries.unit-tests-need-no-services (doc-only)")
   r_sim_boundaries_gateway_executor_decoupled -.-> gap_unenforced
+  r_sim_boundaries_sqlalchemy_and_alembic_only --> e_pytest_projects_policyengine_simulation_entry_tests_test_database_access_boundaries_py
+  e_pytest_projects_policyengine_simulation_entry_tests_test_database_access_boundaries_py --> j_CI_pr_yml_test
   r_sim_boundaries_unit_tests_need_no_services -.-> gap_unenforced
 ```