Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phases 2.1–2.4); `pkg/lint` planned | (CO-7 holds at the parse boundary) |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/plan`, `pkg/lint` — classify/diff/route/report | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), `pkg/router` (backend assignment + availability policy), and `pkg/plan` (versioned dry-run plan report) exist (Phases 2.1–2.5); `pkg/lint` planned | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, and `fmt` exist; `lint` is a stub | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ checkpoint/resume, tuned for Aurora. That is the gap this engine targets.
| [change-capture-tradeoff.md](change-capture-tradeoff.md) | The canonical **triggers vs logical-decoding** trade-off for copy-and-swap — overhead, failover survival, WAL risk, and whether either lets us drop the checksum/checkpoint (answer: keep the checksum; triggers simplify but don't remove the checkpoint). Any doc proposing logical decoding as the default points here. |
| [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. |
| [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. |
| [plan-report.md](plan-report.md) | The **plan report contract** — the versioned JSON shape both front doors emit for dry-run plans: fields, closed vocabularies, the fingerprint identity, required consumer behavior for unknown versions/values, and one generated example per source (pinned by test). |
| [testing.md](testing.md) | The **test-suite guide** — how to run the suite (unit, per-major, all supported majors, compose database), current coverage, the remaining executor-phase test obligations, and the vanilla-PostgreSQL-matrix vs real-Aurora validation boundary. |
| [schemabot-integration.md](schemabot-integration.md) | The **single home for orchestrator integration** — how SchemaBot (the reference orchestrator) drives the engine: the pluggable-engine overview, the verb mappings, the concrete adapter contract, and the design constraints (OC-* invariants) the integration imposes on the core. |

Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ different levels of commitment:
| `pkg/schemadiff` | Execute-and-introspect desired state, introspect the live catalog, and produce an ordered declarative diff | exists |
| `pkg/planner` | Classify typed operations and emit safer native SQL | exists |
| `pkg/lint` | Policy-level rejection of unsafe or unsupported operations | planned |
| `pkg/plan` | Versioned machine-readable dry-run plan report — the one JSON contract both front doors emit and an orchestrator consumes | exists (Phase 2.5) |
| `pkg/router` | Route classified statements to native / copy-and-swap / refuse dispositions; copy-and-swap reports unavailable until that backend lands | exists (Phase 2.4) |
| `pkg/executor` | Bounded optimistic native attempt; the `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) lands in Phase 3 | bounded optimistic attempt exists |
| `pkg/table` | PK-range chunkers (single-column fast path, composite), dynamic time-based sizing | Phase 4 |
Expand Down
1 change: 1 addition & 0 deletions docs/low-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -599,6 +599,7 @@ pkg/statement/ -> Wasm go-pgquery boundary + typed operation descriptors
pkg/schemadiff/ -> execute-and-introspect desired state + live introspection + ordered diff
pkg/planner/ -> classify each operation and construct safer native SQL
pkg/router/ -> assign classified statements to available backends
pkg/plan/ -> versioned machine-readable dry-run plan report (both front doors)
pkg/executor/ -> bounded optimistic native attempt only
pkg/dbconn/ -> bounded database connections
pkg/preflight/ -> migration preflight checks
Expand Down
256 changes: 256 additions & 0 deletions docs/plan-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
# The plan report contract

The plan report is the machine-readable dry-run plan both front doors emit — `migrate
--dry-run --json` (imperative) and `diff --json` (declarative). It is the one JSON shape an
operator or orchestrator consumes to decide whether and how a change would execute. This
document is the contract: the fields, the closed vocabularies, the identity rules, and the
behavior required of a consumer. The Go source of truth is `pkg/plan`; tests in `pkg/plan`,
`pkg/planner`, `pkg/router`, and `pkg/schemadiff` pin everything documented here, including
the examples at the end of this page.

## Versioning: `format_version`

Every report carries `format_version`. A consumer that does not recognize the version must
**reject the report** — never guess at field semantics. The version covers more than the
field shape: the closed vocabularies below (sources, routes, reasons, backends, dispositions,
kinds) and the fingerprint serialization are all pinned to it. Adding a vocabulary value or
changing the fingerprint definition is a contract change and bumps `format_version`, even if
no field is added or renamed.

## Consumer behavior for unknown values

Every enum field in the report draws from a closed vocabulary listed here. A consumer that
meets a value it does not recognize must **treat the statement as unknown and refuse it** —
never ignore it and proceed. This is the same fail-closed posture the engine itself takes
with SQL it does not fully understand.

## Report fields

| Field | Type | Presence | Meaning |
|---|---|---|---|
| `format_version` | int | always | Contract version; reject unknown versions. |
| `source` | string | always | Front door that derived the plan (see Sources). |
| `schema` | string | when resolved | Target schema. The **resolved** name the engine planned against — an unqualified alter reports the schema the engine introspected (`public`), never an empty echo of the submitted text. Absent only when the statement has no single table target. |
| `table` | string | when targeted | Target table; absent for statements with no single table target (index maintenance). |
| `server_version` | string | when connected | The PostgreSQL `server_version` the plan was derived against. Classification is version-sensitive; a stored or forwarded report names the server whose rules produced it. |
| `table_exists` | bool | diff source only | Whether the live table was found. Absent means "not introspected" (alter source); `false` means the plan is the full desired schema. |
| `disposition` | string | always | Aggregate disposition across all statements (see Dispositions). |
| `fingerprint` | string | always | The plan's stable identity (see Fingerprint). |
| `statements` | array | always | The ordered plan; `[]` (never `null`) means nothing to do. |

## Statement fields

| Field | Type | Presence | Meaning |
|---|---|---|---|
| `sql` | string | always | The statement in the engine's **canonical rendering**: parsed and reprinted through the PostgreSQL deparser, whichever front door derived it. Never a verbatim echo of submitted text — the same change carries the same string through either door. Commented input is refused rather than silently stripped; optional noise words follow the grammar's canonical spelling. |
| `kind` | string | diff source only | Classifies a diff-derived statement (see Kinds) so a consumer can gate whole classes of change. Absent for the alter source: a submitted statement may carry several operations and has no single kind. |
| `destructive` | bool | always | Marks statements that discard live structure — a dropped column, constraint, or index. Derived from the classifier's decisions, so both sources report it identically by construction. Always emitted, never omitted: a safety flag a consumer gates on must be explicit even when false. |
| `route` | string | always | The planner's aggregate route for the statement (see Routes). |
| `backend` | string | except refusals | The assigned execution strategy (see Backends); absent for refusals. |
| `disposition` | string | always | What execution would do with this statement now (see Dispositions). |
| `decisions` | array | always | The planner's per-operation classifications (below). |
| `exec_sql` | array | native route | The ordered SQL the native backend would run — the safer sequence when the planner constructed one. Absent for non-native routes. |

## Decision fields

| Field | Type | Presence | Meaning |
|---|---|---|---|
| `operation` | string | always | Operator-facing label (`DROP COLUMN legacy_status`). Display only — never branch on it. |
| `destructive` | bool | always | Whether this operation discards live structure. Always emitted, never omitted. |
| `route` | string | always | Where the operation goes (see Routes). |
| `reason` | string | always | The typed cause of the routing decision (see Reasons). Automation branches on this, never on prose. |
| `safer_sql` | array | safer-idiom only | The ordered native sequence to run instead of the submitted form, when the planner could construct it. |

## Closed vocabularies

### Sources (`source`)

| Value | Meaning |
|---|---|
| `alter` | Derived from a submitted DDL statement (`migrate --dry-run`). |
| `diff` | Derived from a desired-state schema diff (`diff --desired`). |

### Routes (`route`)

| Value | Meaning |
|---|---|
| `native` | PostgreSQL runs it online natively — directly or via the safer idiom in `exec_sql`. |
| `copy-and-swap` | Needs a table rewrite; only the engine's shadow copy + cutover can do it online. |
| `refuse` | No known safe path; not executed. |

### Reasons (`reason`)

| Value | Meaning |
|---|---|
| `metadata-only` | A brief ACCESS EXCLUSIVE catalog change, no scan and no rewrite. |
| `online-idiom` | Already the safe native form (CONCURRENTLY, NOT VALID, VALIDATE, USING INDEX). |
| `fast-default` | ADD COLUMN with a constant default — the catalog stores the default, no rewrite. |
| `binary-coercible` | A type change PostgreSQL relabels without a rewrite (widen varchar, varchar to text, widen numeric precision). |
| `safer-idiom` | Native, but the submitted form blocks; `safer_sql` carries the online rewrite when one can be constructed. |
| `volatile-default` | ADD COLUMN whose default the planner cannot prove constant — PostgreSQL rewrites the table. |
| `generated-stored` | Adding a stored generated column computes every row — a full rewrite. |
| `type-rewrite` | A type conversion PostgreSQL cannot relabel — rewrite plus reindex. |
| `relocation` | SET TABLESPACE moves the heap — a rewrite-scale copy. |
| `partition-parent-lock` | Partition attach/detach in its lock-taking form. |
| `unsupported-operation` | The planner does not recognize the operation or knows no safe path for it. |

### Backends (`backend`)

| Value | Meaning |
|---|---|
| `native` | Direct PostgreSQL DDL (the safer sequence when one exists). |
| `copy-and-swap` | Shadow-table copy with checksum-gated cutover. |

### Dispositions (`disposition`)

| Value | Meaning |
|---|---|
| `execute` | The engine would run it now. |
| `rewrite-required` | Native but blocking as submitted, and no safer sequence could be constructed — resubmit in the online form. |
| `unavailable` | Routed to a backend that is not yet implemented. |
| `refuse` | The planner refused the statement; no backend is assigned. |

### Kinds (`kind`, diff source only)

| Value | Meaning |
|---|---|
| `create-table` | Creates the table (missing-table plans only). |
| `drop-index` | Drops an index. |
| `drop-constraint` | Drops a table constraint. |
| `drop-column` | Drops a column. |
| `add-column` | Adds a column. |
| `alter-type` | Changes a column's type. |
| `set-default` | Sets or replaces a column default. |
| `drop-default` | Drops a column default. |
| `set-not-null` | Adds the NOT NULL attribute. |
| `drop-not-null` | Removes the NOT NULL attribute. |
| `add-constraint` | Adds a table constraint. |
| `create-index` | Creates an index. |

## Fingerprint

`fingerprint` is the plan's stable identity: `sha256:` plus the hex digest over what would
execute. It exists for one consumer protocol: an approver pins it when the plan is
reviewed, and an executor recomputes it at apply time and refuses on mismatch — that is how
"the plan a reviewer approves is the plan that executes" survives storage and forwarding.
The engine computes and reports the fingerprint on every plan; enforcing the pin at apply
time is the consumer's side of the contract.

The serialization is exact and pinned by test: for each statement in plan order, hash the
canonical `sql`, `route`, `backend`, and `disposition`, then each `exec_sql` entry — every
field followed by a unit separator (`0x1F`) — and close each statement with a record
separator (`0x1E`). Explanatory fields (`decisions`, `kind`, `destructive`) are excluded: a
reworded reason does not change identity, but a rerouted, resequenced, or rewritten plan
does. An empty plan has a defined identity (the digest of no input).

This is a **plan identity, not a schema fingerprint**. The engine's schema-state comparisons
only ever compare server-decompiled output against server-decompiled output (see
`pkg/statement`); the plan fingerprint never participates in them.

## Examples

Both examples are generated by the real classify-and-route pipeline and pinned by a test in
`pkg/plan` — if the code drifts from this page, CI fails.

### `source: alter` — `migrate --dry-run --json`

`ALTER TABLE app.orders DROP COLUMN legacy_status` against a live table:

```json
{
"format_version": 1,
"source": "alter",
"schema": "app",
"table": "orders",
"server_version": "16.4",
"disposition": "execute",
"fingerprint": "sha256:acca39fb0630089005cb0ce6519406b1c1cfa8e122aeef044f5502a6b16accbc",
"statements": [
{
"sql": "ALTER TABLE app.orders DROP legacy_status",
"destructive": true,
"route": "native",
"backend": "native",
"disposition": "execute",
"decisions": [
{
"operation": "DROP COLUMN legacy_status",
"destructive": true,
"route": "native",
"reason": "metadata-only"
}
],
"exec_sql": [
"ALTER TABLE app.orders DROP legacy_status"
]
}
]
}
```

Note the canonical rendering: the deparser spells `DROP legacy_status` (the grammar treats
`COLUMN` as optional noise), and there is no `table_exists` — the alter source does not
introspect for existence.

### `source: diff` — `diff --json`

A desired state that drops an index and adds a column with a constant default:

```json
{
"format_version": 1,
"source": "diff",
"schema": "app",
"table": "orders",
"server_version": "16.4",
"table_exists": true,
"disposition": "execute",
"fingerprint": "sha256:cb7ec645948ff1d239ba4ce3f0e051e4e2bcebec799fdd6f8401399d4a246f53",
"statements": [
{
"sql": "DROP INDEX app.orders_legacy_idx",
"kind": "drop-index",
"destructive": true,
"route": "native",
"backend": "native",
"disposition": "execute",
"decisions": [
{
"operation": "DROP INDEX app.orders_legacy_idx",
"destructive": true,
"route": "native",
"reason": "safer-idiom",
"safer_sql": [
"DROP INDEX CONCURRENTLY app.orders_legacy_idx"
]
}
],
"exec_sql": [
"DROP INDEX CONCURRENTLY app.orders_legacy_idx"
]
},
{
"sql": "ALTER TABLE app.orders ADD COLUMN region text DEFAULT 'emea'",
"kind": "add-column",
"destructive": false,
"route": "native",
"backend": "native",
"disposition": "execute",
"decisions": [
{
"operation": "ADD COLUMN region",
"destructive": false,
"route": "native",
"reason": "fast-default"
}
],
"exec_sql": [
"ALTER TABLE app.orders ADD COLUMN region text DEFAULT 'emea'"
]
}
]
}
```

Note `kind` on each statement (diff source only), the blocking `DROP INDEX` replaced by its
CONCURRENTLY form in `exec_sql`, and `table_exists: true`.
1 change: 1 addition & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ this repository's CI.
| Parse boundary, typed operations, and advisory rewrites | [pkg/statement](../pkg/statement/statement_test.go), [operation tests](../pkg/statement/ops_test.go) |
| Native / copy-and-swap / refuse classification and safer SQL | [pkg/planner](../pkg/planner/planner_test.go) |
| Backend routing and copy-and-swap unavailable disposition | [pkg/router](../pkg/router/router_test.go) |
| Plan report contract: exact JSON shape, versioning, field omissions | [pkg/plan](../pkg/plan/plan_test.go) |
| Scratch execute-and-introspect, ordered diff, and convergence (`TestDiffConverges`) | [pkg/schemadiff](../pkg/schemadiff/schemadiff_integration_test.go), [diff tests](../pkg/schemadiff/diff_test.go) |
| CLI `diff`, `fmt`, and classified `migrate --dry-run`, including applying text output and re-diffing to empty (`TestDiffTextPlanIsExecutableSQL`) | [diff integration](../internal/cli/diff_integration_test.go), [fmt](../internal/cli/diff_test.go), [dry-run integration](../internal/cli/dryrun_integration_test.go) |
| Bounded optimistic native attempt and table preflight | [pkg/executor](../pkg/executor/optimistic_integration_test.go), [pkg/preflight](../pkg/preflight/preflight_integration_test.go) |
Expand Down
12 changes: 12 additions & 0 deletions internal/cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"time"

"github.com/alecthomas/kong"
"github.com/jackc/pgx/v5/pgxpool"

"github.com/block/pg-sprite/pkg/dbconn"
)
Expand Down Expand Up @@ -64,6 +65,17 @@ func (f DBFlags) Config() dbconn.Config {
return cfg
}

// serverVersion reads the connected server's server_version setting for
// the plan report: classification is version-sensitive, so a stored report
// names the server whose rules produced it.
func serverVersion(ctx context.Context, pool *pgxpool.Pool) (string, error) {
var v string
if err := pool.QueryRow(ctx, "SELECT current_setting('server_version')").Scan(&v); err != nil {
return "", fmt.Errorf("read server_version: %w", err)
}
return v, nil
}

// diag returns the diagnostics logger: debug-level text on stderr (or the
// test override) under --debug, a discarding logger otherwise. Diagnostics
// never share stdout with command output.
Expand Down
Loading
Loading