Schema change: put the real-PostgreSQL checks in the repo, and say what has and has not been checked

README.md

@@ -110,23 +110,31 @@ The **Schema Change** page works on a saved live connection, not on the diagram
 
 1. **Find the tables (optional).** You do not have to choose any. The catalog (names, columns, keys, estimated rows, sizes) is read in one round trip. The words of your request are matched against table, schema and column names, weighted by how rare each word is in *your* database, and the foreign-key neighbours of the best matches are added. The scope is shown with the reason for each table and you can add or remove tables. It matches names, not meanings: a request that calls `clients` "customers" finds nothing, and the plan then goes ahead with no table in scope. That is how a request for **new tables** works: the model is told the names of the existing tables (not their contents), creates the new ones with `create_table`, and any existing table its foreign keys point at is read afterwards so the keys are checked against that table's real columns. Names already taken in the database are refused.
 2. **Give the planner facts.** Only the tables in scope are sent in full (the others by name): columns, keys, existing indexes with how often each was scanned (`pg_stat_user_indexes`), row estimates, data sizes, dependent views, and two aggregate statistics per column (NULL fraction, distinct-value estimate). Never rows, default expressions, index predicates or credentials.
-3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The lock levels and the "rewrites the table" predictions are checked against PostgreSQL 16 (`npm run test:schema-pg`).
-4. **Open it in the SQL Editor.** The runbook opens with the transaction switch off, because `CONCURRENTLY` and the batched backfills need each statement on its own. Nothing is run for you.
+3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check` and `drop_constraint`. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
+4. **Check the database, then run it.** *Check the database first* reads, without writing, who holds or waits for the tables (found through `pg_locks`, so even a role that may not read other roles' sessions sees them), transactions open for 30 seconds or more, strong locks, prepared transactions, replicas that are behind, replication slots holding WAL, vacuum running on the tables, and the timeouts the role runs with; each check says "can't tell" instead of "ok" when it cannot read something. Free disk space cannot be read from a connection, so that is for you to check. *Run it here, one step at a time* then runs each step on its own in a SQL session of its own (no transaction, because `CONCURRENTLY` and the batched backfills need that): only the next step can run, the first failure stops the run, a step can be run again or stopped, and the most recent finished step can be undone with the statement the compiler wrote for it. A step that needs a strong lock on an existing table waits at most 2 seconds per attempt and tries up to 8 times, backing off between tries; if the table never frees up it stops with the sessions holding it and changes nothing. After the last step the catalog is read again and compared, table by table, with the state the compiler said it would leave (columns, types, NOT NULL, keys, unique and check constraints, foreign keys, indexes, views); every difference is listed. Optional clean-up steps (dropping an old column, a compatibility view) are never run for you. The runbook can also be opened in the SQL Editor with the transaction switch off, or copied.
 5. **Health.** A read-only report from catalog queries: tables without a primary key, foreign keys without an index, duplicate, redundant, unused and invalid indexes, constraints that were never validated, wide tables, missing or stale statistics, tables with many dead rows, sequences past half their range. A model may be asked to order and explain the findings; it can only point at findings that exist.
 6. **Governance.** The model is called through the OpenAI API with your key (see above; GoForge does not store it), an allow-list of model ids, optional redaction of schema, table, column, index and constraint names into placeholders before anything leaves the server (the answer is translated back), and an audit log of every request with exactly what was sent, every plan, every runbook opened and every settings change, exportable as CSV or JSON. The audit log records that a runbook was *opened*; what you run in the SQL Editor is not tracked.
 
-Limits worth knowing: lock levels and the compiler's rules assume PostgreSQL 12 or newer; the size classes (under 16 MiB small, under 1 GiB medium, larger large) are a rule of thumb, not a timing; index usage counts are per server and only since the statistics were last reset; expand/contract is not compiled for a column that is part of a key, a foreign key or a check, or that a view uses; a table without a primary key or unique NOT NULL column can only be backfilled by page on PostgreSQL 14 or newer. A plan is checked against the facts read when it was made: build it again if the schema changed.
+Limits worth knowing: the compiler's rules assume PostgreSQL 12 or newer; the size classes (under 16 MiB small, under 1 GiB medium, larger large) are a rule of thumb, not a timing; index usage counts are per server and only since the statistics were last reset; expand/contract is not compiled for a column that is part of a key, a foreign key or a check, or that a view uses; a table without a primary key or unique NOT NULL column can only be backfilled by page on PostgreSQL 14 or newer; a PL/pgSQL function body that uses a column or table is invisible to PostgreSQL's dependency tracking, so it cannot be warned about (a count of triggers is shown instead); a compatibility view copies only SELECT, INSERT, UPDATE and DELETE grants, not column-level grants. A plan is checked against the facts read when it was made: build it again if the schema changed.
+
+**What has and has not been checked.**
+
+- *Checked, on real servers:* `npm run test:schema-pg:versions` runs 15 sections against PostgreSQL 12, 13, 14, 15, 16, 17 and 18, one container each, and all pass. They check the lock level each step takes against `pg_locks`; which type changes rewrite a table, by watching the table's file; every compiled scenario executed, with the end state read back from the catalog and compared with the compiler's prediction; the drop and type-change refusals against what PostgreSQL itself refuses; the pre-flight check and the lock retry with real sessions (a transaction held open, a queued `ACCESS EXCLUSIVE`, an unread replication slot, a role timeout, a role that cannot read other sessions); the table rename with real roles and grants, row-level security, undo and the contract step; and the probes of defaults and checks.
+- *Checked on small data only:* the test tables hold up to a few hundred thousand rows. Whether a step stays short on a table of many gigabytes, how long a validation or index build takes, and replica lag during a large backfill have not been measured. The size classes and risk classes are fixed rules, not measurements.
+- *Not run against a real server:* the index build of a partitioned table (the SQL is written and unit-tested; only the catalog reading of partitioned tables is checked on a server), replicas and logical replication (the pre-flight reads them, and the rules for them are unit-tested only), prepared transactions beyond the empty case, and extensions that change how types or indexes behave.
+- *Models:* the planner's answers are accepted by OpenAI's structured-output check (`npm run check:openai` in `server/`), and a few real requests were planned by `gpt-5.6-luna` and their runbooks read through by hand; there is no scored evaluation of how often a model proposes the right operations. The compiler checks every operation against the catalog, so a wrong proposal is refused or shown, but a plausible wrong one that PostgreSQL accepts is not caught: read the plan.
 
 How it is tested:
 
 ```bash
-docker run -d --name goforge-schema -e POSTGRES_PASSWORD=schema -e POSTGRES_DB=schema -p 54330:5432 postgres:16 -c max_locks_per_transaction=1024
-npm run test:schema-pg                                   # catalog, facts, locks, rewrites, compiler runs, health, redaction, on a real PostgreSQL 16
-npm run eval:schema                                      # table selection, plan checks, lint, execution and latency on generated 50 / 300 / 1,000 table databases
-npm run eval:schema -- --sizes 300 --per-size 30 --keep  # keep sc_eval_300 to browse it (save a connection to 127.0.0.1:54330)
-OPENAI_API_KEY=sk-... npm run eval:schema -- --live   # a real model (gpt-5.6-luna, or EVAL_MODEL) plans instead of the reference plans
+npm run test:schema-pg:versions                          # every section on PostgreSQL 12 to 18 (starts and removes one Docker container per version)
+docker run -d --name goforge-schema -e POSTGRES_PASSWORD=schema -p 54330:5432 postgres:16 -c max_locks_per_transaction=1024
+npm run test:schema-pg                                   # every section against that one server
+npm run test:schema-pg -- model                          # one section: drops, dependents, lockwait, preflight, model, rename, planner, health, locks, rewrites, compile, catalog, retrieval, facts, probe
 ```
 
+The generated-database evaluation below (`scripts/schema-change-eval.ts`, `npm run eval:schema`) is not in this repository yet.
+
 `npm run eval:schema` builds the databases (real foreign keys, tables with 260,000 rows), writes change requests in the words a person would use (with typos, partial names, a column name only, the same name in two schemas), runs each through retrieval, the checks and the compiler, executes the runbook on a copy of the database and verifies the result. It also feeds in requests that must be refused. Without `--live` the plans are the reference plans, so the plan-validity numbers measure the checks and the compiler, not a model; the model's own latency and token use are reported as not measured. `scripts/fake-model-server.ts` is a stand-in OpenAI endpoint for trying the page without a key.
 
 ### Query Modernizer: converting a whole file

package.json

@@ -13,6 +13,8 @@
     "mcp": "bun run src/mcp/cli.ts",
     "test:corpus": "server/node_modules/.bin/tsx scripts/modernizer-corpus.ts",
     "corpus:golden": "server/node_modules/.bin/tsx scripts/modernizer-golden.ts",
+    "test:schema-pg": "server/node_modules/.bin/tsx scripts/schema-change-pg.ts",
+    "test:schema-pg:versions": "bash scripts/schema-change-versions.sh",
     "queryshift": "server/node_modules/.bin/tsx queryshift/run.ts",
     "queryshift:check-model": "server/node_modules/.bin/tsx queryshift/check-model.ts"
   },

scripts/lib/pg-scratch.ts

@@ -0,0 +1,111 @@
+/**
+ * Small helpers for the scripts that run the schema-change code against a real PostgreSQL 16:
+ *
+ *   docker run -d --name goforge-schema -e POSTGRES_PASSWORD=schema -e POSTGRES_DB=schema -p 54330:5432 postgres:16 -c max_locks_per_transaction=1024
+ *
+ * Every run works in a database of its own that is dropped afterwards, so nothing here touches anything you own.
+ */
+import { createRequire } from "node:module";
+
+const require = createRequire(new URL("../../server/package.json", import.meta.url));
+const { Client } = require("pg") as typeof import("pg");
+
+export type PgClient = InstanceType<typeof Client>;
+
+const ADMIN_URL = process.env.SCHEMA_CHANGE_DATABASE_URL ?? "postgres://postgres:schema@127.0.0.1:54330/schema";
+
+const withDatabaseName = (url: string, name: string): string => {
+    const parsed = new URL(url);
+    parsed.pathname = `/${name}`;
+    return parsed.toString();
+};
+
+/** DROP DATABASE ... WITH (FORCE) only exists from PostgreSQL 13: elsewhere the sessions are ended first. */
+async function dropForce(admin: PgClient, name: string): Promise<void> {
+    await admin.query("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = $1 AND pid <> pg_backend_pid()", [name]);
+    await admin.query(`DROP DATABASE IF EXISTS ${name}`);
+}
+
+export async function connect(database?: string): Promise<PgClient> {
+    const client = new Client({ connectionString: database ? withDatabaseName(ADMIN_URL, database) : ADMIN_URL });
+    await client.connect();
+    return client;
+}
+
+let counter = 0;
+
+/** A fresh, empty database for the duration of `fn`; `template` clones an existing one (which must have no open connections). */
+export async function withDatabase<T>(fn: (client: PgClient, name: string) => Promise<T>, options: { template?: string; prefix?: string } = {}): Promise<T> {
+    const admin = await connect();
+    const name = `${options.prefix ?? "sc"}_${process.pid}_${Date.now().toString(36)}_${counter++}`;
+    try {
+        // A template must have no sessions; anything still connected to this scratch copy (an app that was pointed at it) is disconnected.
+        if (options.template) await admin.query("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = $1 AND pid <> pg_backend_pid()", [options.template]);
+        await admin.query(`CREATE DATABASE ${name}${options.template ? ` TEMPLATE ${options.template}` : ""}`);
+        const client = await connect(name);
+        try {
+            return await fn(client, name);
+        } finally {
+            await client.end().catch(() => undefined);
+        }
+    } finally {
+        await dropForce(admin, name).catch(() => undefined);
+        await admin.end().catch(() => undefined);
+    }
+}
+
+/** A database that stays after the call (the caller drops it): used for the large generated schemas the evaluation clones. */
+export async function createDatabase(name: string, build: (client: PgClient) => Promise<void>): Promise<void> {
+    const admin = await connect();
+    try {
+        await dropForce(admin, name);
+        await admin.query(`CREATE DATABASE ${name}`);
+    } finally {
+        await admin.end();
+    }
+    const client = await connect(name);
+    try {
+        await build(client);
+    } finally {
+        await client.end();
+    }
+}
+
+export async function dropDatabase(name: string): Promise<void> {
+    const admin = await connect();
+    try {
+        await dropForce(admin, name);
+    } finally {
+        await admin.end();
+    }
+}
+
+/** Tiny assertion collector: every check is counted and listed, and the process exits 1 if any failed. */
+export class Checks {
+    private passed = 0;
+    private readonly failures: string[] = [];
+
+    constructor(private readonly title: string) {}
+
+    ok(condition: unknown, label: string, detail?: unknown): void {
+        if (condition) {
+            this.passed++;
+            return;
+        }
+        this.failures.push(detail === undefined ? label : `${label}\n      ${typeof detail === "string" ? detail : JSON.stringify(detail)}`);
+    }
+
+    match(actual: string | undefined, pattern: RegExp, label: string): void {
+        this.ok(actual !== undefined && pattern.test(actual), label, `got ${JSON.stringify(actual)}`);
+    }
+
+    equal(actual: unknown, expected: unknown, label: string): void {
+        this.ok(JSON.stringify(actual) === JSON.stringify(expected), label, `got ${JSON.stringify(actual)}, expected ${JSON.stringify(expected)}`);
+    }
+
+    report(): boolean {
+        console.log(`${this.title}: ${this.passed} passed, ${this.failures.length} failed`);
+        for (const failure of this.failures) console.log(`  FAIL ${failure}`);
+        return this.failures.length === 0;
+    }
+}