Phase 2.1-2.2: parse boundary and execute-and-introspect declarative diff

.github/workflows/ci.yml

@@ -1,4 +1,4 @@
-name: ci
+name: CI
 
 on:
   push:
@@ -91,6 +91,7 @@ jobs:
   # images — real Aurora engine-version validation is a separate gate that
   # cannot run in public CI.
   test:
+    name: test (PostgreSQL ${{ matrix.pg }})
     needs: changes
     if: needs.changes.outputs.code == 'true'
     runs-on: ubuntu-latest
@@ -107,9 +108,10 @@ jobs:
           go-version-file: go.mod
       - run: make test
 
-  # Single required status for branch protection. Succeeds when nothing
-  # failed — including docs-only PRs where the heavy jobs were skipped.
-  ci-ok:
+  # Single required status for branch protection ("all-green" is the
+  # context to require). Succeeds when nothing failed — including
+  # docs-only PRs where the heavy jobs were skipped.
+  all-green:
     if: always()
     needs: [changes, lint, build, test]
     runs-on: ubuntu-latest

SAFETY.md

@@ -27,7 +27,7 @@ The invariant registry (invariant IDs referenced below) lives in
 | `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
 | slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
 | `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
-| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` exists (Phase 1: type gate); rest planned (Phase 2) | (CO-7 holds at the parse boundary) |
+| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary) and `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect) exist (Phase 2); `pkg/planner`, `pkg/lint` planned (Phase 2) | (CO-7 holds at the parse boundary) |
 | `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
 | `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`/`status` exist (Phase 1); rest stubs | — |
 | status / progress / advisory rendering, metrics | ❌ periphery | planned | — |

docs/low-level-design.md

@@ -249,15 +249,28 @@ live schema (introspected)  ─┘                          │
 4. **Diff** the two models and emit the minimal set of statements: `ADD/DROP/ALTER COLUMN`,
    `ADD/DROP CONSTRAINT`, `CREATE/DROP INDEX`, default/nullability changes, etc., in a
    **dependency-correct order** (e.g. add a column before an index that references it).
+   Columns are compared **by name**: a live table whose columns are ordered differently from
+   the desired file converges to "no changes". Attribute order carries no semantics in
+   PostgreSQL and cannot be changed in place, so — unlike some declarative MySQL tooling —
+   column order is deliberately out of scope for convergence.
 5. **Hand the derived statements to the same classifier**, so a declarative change that turns
    out to be, say, a binary-coercible type widening still takes the native fast path, and only
    a genuine rewrite triggers a copy.
 
 ### Safety rules (inherited philosophy: surprise-free, decisions-not-options)
 
-- **Destructive diffs are gated.** Dropping a column or constraint, or anything that loses
-  data, requires an explicit confirmation flag — never inferred silently from "it's missing in
-  the desired file".
+- **Destructive diffs are gated.** Dropping a column, constraint, or index — anything that
+  loses data or a guarantee (a unique index discards the same uniqueness guarantee as a unique
+  constraint) — requires an explicit confirmation flag — never inferred silently from "it's
+  missing in the desired file".
+- **Unsupported constructs are refused, never guessed.** The desired file admits one
+  unqualified `CREATE TABLE` plus `CREATE INDEX` statements on it; each rule is a typed error.
+  Foreign keys are refused at admission — a `REFERENCES` clause cannot be faithfully executed
+  in the transaction-scoped scratch schema (an unqualified reference resolves against the
+  scratch search_path, not the target schema), and FK support needs its own design. Changes
+  the plan cannot express — identity or generation changes on an existing column, adopting a
+  sequence-backed (serial) default whose sequence only existed in the rolled-back scratch
+  transaction — are refused as unsupported rather than emitted as an unexecutable plan.
 - **Renames are ambiguous and are not guessed.** A column present in live but absent in desired
   plus a new column in desired is, by default, a *drop + add*, not a rename. Rename intent must
   be stated explicitly (the engine will not heuristically pair columns), mirroring Spirit's
@@ -544,6 +557,15 @@ The scratch database is engine-owned and disposable: preflight may reset it (dro
 contents) at any time. Restricted environments that won't grant `CREATEDB` pre-provision
 instead.
 
+**Plan-time diffing uses a lighter mechanism.** `pkg/schemadiff` materializes the desired
+state inside a single always-rolled-back transaction in the *target* database, in a
+randomly named transaction-scoped schema (`pgsprite_scratch_<random>`). This keeps the
+same-server semantic-truth property (same version, extensions, and defaults as the live
+table) while requiring no `CREATEDB`, no pre-provisioning, and leaving zero footprint —
+appropriate because diffing is read-only planning. The durable `pg_sprite_scratch`
+database above is required only by the migration path proper (shadow-DDL derivation and
+checkpoint fingerprints), where objects must outlive a transaction.
+
 ### Postgres-only preconditions Spirit has no analog for
 
 These have **no MySQL counterpart** but are hard requirements for the logical-decoding path:

docs/testing.md

@@ -26,8 +26,9 @@ Concretely:
 - **Real database, race-enabled.** Unit tests run with `-race`; core (`pkg/`)
   logic is never validated against mocks — integration tests run against
   real PostgreSQL, across every supported major in CI.
-- **The matrix is a gate, not advisory.** The `ci-ok` sentinel requires the
-  full version matrix; docs-only changes are the only path that skips it.
+- **The matrix is a gate, not advisory.** The `all-green` sentinel job
+  requires the full version matrix; docs-only changes are the only path
+  that skips it.
 - **Coverage never regresses.** Deleting or skipping a test to get green is
   forbidden (same rule as the hooks: no `--no-verify`, no `nolint`). A
   numeric coverage ratchet on `pkg/` packages is wired into CI once Phase 1

internal/cli/cli.go

@@ -92,23 +92,28 @@ type MigrateCmd struct {
 // Run implements the migrate subcommand.
 func (c *MigrateCmd) Run() error { return c.run(context.Background(), os.Stdout) }
 
-// DiffCmd derives statements from a desired-state schema (declarative front-end).
+// DiffCmd derives statements from a desired-state schema (declarative
+// front-end): introspect the live table, materialize the desired state on a
+// rolled-back scratch schema, and print the ordered plan without executing
+// anything.
 type DiffCmd struct {
 	DBFlags `embed:""`
 
 	Desired string `help:"Path to the desired-state CREATE TABLE .sql file." name:"desired" type:"existingfile" required:""`
+	Schema  string `help:"Schema containing the live table." default:"public"`
+	JSON    bool   `help:"Emit the plan as JSON."`
 }
 
 // Run implements the diff subcommand.
-func (c *DiffCmd) Run() error { return notImplemented("diff") }
+func (c *DiffCmd) Run() error { return c.run(context.Background(), os.Stdout) }
 
 // FmtCmd canonicalizes a schema file. It is offline — no database flags.
 type FmtCmd struct {
-	Path string `arg:"" optional:"" help:"Schema file to format." type:"existingfile"`
+	Path string `arg:"" optional:"" help:"Schema file to format; stdin when omitted." type:"existingfile"`
 }
 
 // Run implements the fmt subcommand.
-func (c *FmtCmd) Run() error { return notImplemented("fmt") }
+func (c *FmtCmd) Run() error { return c.runFmt(os.Stdin, os.Stdout) }
 
 // LintCmd checks DDL for unsafe or unsupported patterns.
 type LintCmd struct{}

internal/cli/diff.go

@@ -0,0 +1,202 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+
+	"github.com/block/pg-sprite/pkg/dbconn"
+	"github.com/block/pg-sprite/pkg/schemadiff"
+	"github.com/block/pg-sprite/pkg/statement"
+)
+
+// diffReport is the diff command's JSON output contract.
+type diffReport struct {
+	// Schema is the live schema the diff targeted.
+	Schema string `json:"schema"`
+	// Table is the desired (and live) table name.
+	Table string `json:"table"`
+	// TableExists reports whether the live table was found; when false the
+	// changes are the full desired schema.
+	TableExists bool `json:"table_exists"`
+	// Changes is the ordered statement plan; empty means the live table
+	// already matches the desired state.
+	Changes []schemadiff.Change `json:"changes"`
+}
+
+// run is the diff flow: parse and admit the desired file, introspect the
+// live table and the desired state (execute-and-introspect on a rolled-back
+// scratch schema), and print the ordered plan. Nothing is ever executed
+// against the live table.
+func (c *DiffCmd) run(ctx context.Context, out io.Writer) error {
+	logger := c.diag()
+	raw, err := os.ReadFile(c.Desired)
+	if err != nil {
+		return fmt.Errorf("read desired schema: %w", err)
+	}
+	ds, err := statement.ParseDesired(string(raw))
+	if err != nil {
+		return err
+	}
+	logger.Debug("desired schema parsed", "table", ds.Table, "statements", len(ds.Statements))
+
+	pool, err := dbconn.NewPool(ctx, c.Config())
+	if err != nil {
+		return err
+	}
+	defer pool.Close()
+
+	report := diffReport{Schema: c.Schema, Table: ds.Table, TableExists: true}
+	live, err := schemadiff.Introspect(ctx, pool, c.Schema, ds.Table)
+	switch {
+	case errors.Is(err, schemadiff.ErrTableNotFound):
+		// No live table: the plan is the desired schema itself, qualified
+		// onto the target schema.
+		report.TableExists = false
+		if report.Changes, err = qualifiedDesired(ds, c.Schema); err != nil {
+			return err
+		}
+	case err != nil:
+		return err
+	default:
+		desired, err := schemadiff.IntrospectDesired(ctx, pool, ds)
+		if err != nil {
+			return err
+		}
+		if report.Changes, err = schemadiff.Diff(c.Schema, live, desired); err != nil {
+			return err
+		}
+	}
+	logger.Debug("diff derived",
+		"schema", c.Schema, "table", ds.Table, "changes", len(report.Changes), "table_exists", report.TableExists)
+
+	if c.JSON {
+		return writeJSON(out, report)
+	}
+	return writePlanText(out, report)
+}
+
+// qualifiedDesired renders the desired statements as the plan for a table
+// that does not exist yet, qualified onto the target schema.
+func qualifiedDesired(ds statement.DesiredSchema, schema string) ([]schemadiff.Change, error) {
+	changes := make([]schemadiff.Change, 0, len(ds.Statements))
+	for _, st := range ds.Statements {
+		qualified, err := statement.Qualify(st.SQL(), schema)
+		if err != nil {
+			return nil, fmt.Errorf("qualify desired statement: %w", err)
+		}
+		kind := schemadiff.ChangeCreateTable
+		if st.Kind() == statement.KindCreateIndex {
+			kind = schemadiff.ChangeCreateIndex
+		}
+		changes = append(changes, schemadiff.Change{SQL: qualified, Kind: kind})
+	}
+	return changes, nil
+}
+
+// writeJSON emits the report as JSON.
+func writeJSON(out io.Writer, report diffReport) error {
+	if report.Changes == nil {
+		report.Changes = []schemadiff.Change{}
+	}
+	enc := json.NewEncoder(out)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(report); err != nil {
+		return fmt.Errorf("write diff report: %w", err)
+	}
+	return nil
+}
+
+// writePlanText emits the plan as an executable SQL script: one statement
+// per line, destructive and lock-hazardous statements flagged with leading
+// comment lines, and SQL comments for the no-change and missing-table cases
+// so the output stays valid SQL. The header points at migrate as the
+// executing front door: running this script directly bypasses the gate that
+// refuses blocking statements.
+func writePlanText(out io.Writer, report diffReport) error {
+	if len(report.Changes) == 0 {
+		if _, err := fmt.Fprintln(out, "-- no changes: live table matches the desired schema"); err != nil {
+			return fmt.Errorf("write plan: %w", err)
+		}
+		return nil
+	}
+	if _, err := fmt.Fprintln(out, "-- plan derived by pg-sprite diff; execute statements via pg-sprite migrate,"); err != nil {
+		return fmt.Errorf("write plan: %w", err)
+	}
+	if _, err := fmt.Fprintln(out, "-- which refuses blocking forms — running this script directly bypasses that gate"); err != nil {
+		return fmt.Errorf("write plan: %w", err)
+	}
+	if !report.TableExists {
+		if _, err := fmt.Fprintf(out, "-- table %s.%s does not exist; the plan is the full desired schema\n",
+			report.Schema, report.Table); err != nil {
+			return fmt.Errorf("write plan: %w", err)
+		}
+	}
+	for _, ch := range report.Changes {
+		if ch.Destructive {
+			if _, err := fmt.Fprintln(out, "-- destructive"); err != nil {
+				return fmt.Errorf("write plan: %w", err)
+			}
+		}
+		if hazard := lockHazard(ch.Kind); hazard != "" {
+			if _, err := fmt.Fprintf(out, "-- %s\n", hazard); err != nil {
+				return fmt.Errorf("write plan: %w", err)
+			}
+		}
+		if _, err := fmt.Fprintf(out, "%s;\n", ch.SQL); err != nil {
+			return fmt.Errorf("write plan: %w", err)
+		}
+	}
+	return nil
+}
+
+// lockHazard describes the blocking behavior of a change kind, empty when
+// the statement is not expected to block writers. This is presentation for
+// the text plan; the machine contract is the kind itself.
+func lockHazard(kind schemadiff.ChangeKind) string {
+	switch kind {
+	case schemadiff.ChangeAlterType:
+		return "rewrites the table under ACCESS EXCLUSIVE, blocking reads and writes"
+	case schemadiff.ChangeSetNotNull:
+		return "full table scan under ACCESS EXCLUSIVE"
+	case schemadiff.ChangeAddConstraint:
+		return "validation scan or index build that blocks writes"
+	case schemadiff.ChangeCreateIndex:
+		return "blocks writes for the whole index build"
+	default:
+		return ""
+	}
+}
+
+// runFmt canonicalizes a desired-state schema file: every statement is
+// parsed through the PostgreSQL grammar, admitted by the same rules as diff,
+// and printed back in the deparser's canonical form. Offline — no database.
+// Commented input is refused (statement.ErrCommentLoss): the parser drops
+// comments, and a formatter must never silently discard content.
+func (c *FmtCmd) runFmt(in io.Reader, out io.Writer) error {
+	var src []byte
+	var err error
+	if c.Path == "" {
+		if src, err = io.ReadAll(in); err != nil {
+			return fmt.Errorf("read schema from stdin: %w", err)
+		}
+	} else if src, err = os.ReadFile(c.Path); err != nil {
+		return fmt.Errorf("read schema file: %w", err)
+	}
+	if err := statement.CheckNoComments(string(src)); err != nil {
+		return err
+	}
+	ds, err := statement.ParseDesired(string(src))
+	if err != nil {
+		return err
+	}
+	for _, st := range ds.Statements {
+		if _, err := fmt.Fprintf(out, "%s;\n", st.SQL()); err != nil {
+			return fmt.Errorf("write formatted schema: %w", err)
+		}
+	}
+	return nil
+}