Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary) and `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect) exist (Phase 2); `pkg/planner`, `pkg/lint` planned (Phase 2) | (CO-7 holds at the parse boundary) |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phase 2); `pkg/lint` planned (Phase 2) | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`/`status` exist (Phase 1); rest stubs | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
Expand Down
4 changes: 4 additions & 0 deletions docs/postgres-online-ddl-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ the same dynamic via metadata locks. Read it first if any of that is unfamiliar.
| `ADD COLUMN ... DEFAULT <constant>` | ACCESS EXCLUSIVE (brief) | **No** (PG 11+) | Yes | ❌ No | "Fast default" stored in catalog; pre-PG11 this rewrote |
| `ADD COLUMN ... DEFAULT <volatile>` (e.g. `now()`, `random()`, `uuid_generate_v4()`) | ACCESS EXCLUSIVE | **Yes** (full rewrite) | No | ✅ **Yes** | The expensive case |
| `ADD COLUMN ... GENERATED ALWAYS AS (...) STORED` | ACCESS EXCLUSIVE | Yes | No | ✅ **Yes** | Values must be computed |
| `ADD COLUMN ... UNIQUE` / `PRIMARY KEY` / `REFERENCES` / `CHECK` (inline constraint) | ACCESS EXCLUSIVE + index build or validation | No | No | ➖ Native pattern | Same work as the `ADD CONSTRAINT` form, under the `ADD COLUMN` lock — split: add the column first, then build the constraint online (`CONCURRENTLY` + `USING INDEX`, or `NOT VALID` + `VALIDATE`) |
| `DROP COLUMN` | ACCESS EXCLUSIVE (brief) | No | Yes | ❌ No | Metadata only; disk space reclaimed lazily by VACUUM |
| `ALTER COLUMN TYPE` — binary-coercible (`varchar(50)→varchar(100)`, `varchar→text`, `numeric(10,2)→numeric(12,2)`) | ACCESS EXCLUSIVE (brief) | **No** | No (brief) | ❌ No | No scan when binary-coercible and no length restriction is added |
| `ALTER COLUMN TYPE` — general (`int→bigint`, `text→jsonb`, `timestamp→timestamptz` w/ conversion) | ACCESS EXCLUSIVE | **Yes** (rewrite + reindex + revalidate FKs) | No | ✅ **Yes** | The classic "needs a tool" case |
Expand Down Expand Up @@ -96,6 +97,7 @@ online path, so the heavy **shadow-copy + atomic cutover** path is required ·
| `ALTER COLUMN SET STATISTICS` / `SET STORAGE` / `SET (n_distinct=...)` | No | varies | ❌ No | SHARE UPDATE EXCLUSIVE / ACCESS EXCLUSIVE |
| `ALTER COLUMN TYPE` — binary-coercible | No | No (brief) | ❌ No | ACCESS EXCLUSIVE (brief) |
| `ALTER COLUMN SET NOT NULL` | No, but full scan | No | ➖ Native pattern | ACCESS EXCLUSIVE |
| `ADD COLUMN ...` (inline constraint) | No, but index build / validation | No | ➖ Native pattern | ACCESS EXCLUSIVE |
| `ADD COLUMN ... DEFAULT <volatile>` | Yes | No | ✅ **Yes** | ACCESS EXCLUSIVE |
| `ADD COLUMN ... GENERATED ALWAYS AS (...) STORED` | Yes | No | ✅ **Yes** | ACCESS EXCLUSIVE |
| `ALTER COLUMN TYPE` — general | Yes | No | ✅ **Yes** | ACCESS EXCLUSIVE |
Expand All @@ -116,6 +118,7 @@ online path, so the heavy **shadow-copy + atomic cutover** path is required ·
| `ALTER COLUMN SET STATISTICS` / `SET STORAGE` / `SET (n_distinct=...)` | varies | No | ❌ No | SHARE UPDATE EXCLUSIVE / ACCESS EXCLUSIVE |
| `ALTER COLUMN TYPE` — binary-coercible | No (brief) | No | ❌ No | ACCESS EXCLUSIVE (brief) |
| `ALTER COLUMN SET NOT NULL` | No | No, but full scan | ➖ Native pattern | ACCESS EXCLUSIVE |
| `ADD COLUMN ...` (inline constraint) | No | No, but index build / validation | ➖ Native pattern | ACCESS EXCLUSIVE |
| `ADD COLUMN ... DEFAULT <volatile>` | No | Yes | ✅ **Yes** | ACCESS EXCLUSIVE |
| `ADD COLUMN ... GENERATED ALWAYS AS (...) STORED` | No | Yes | ✅ **Yes** | ACCESS EXCLUSIVE |
| `ALTER COLUMN TYPE` — general | No | Yes | ✅ **Yes** | ACCESS EXCLUSIVE |
Expand Down Expand Up @@ -188,6 +191,7 @@ ALTER TABLE t ADD CONSTRAINT t_pkey PRIMARY KEY USING INDEX t_pkey; -- brief lo
| `SET TABLESPACE` | ACCESS EXCLUSIVE | **Yes** (moves heap) | No | ✅ **Yes** (repack-style) | Rewrite/move; use a repack-style copy instead |
| `SET (fillfactor=...)` and most reloptions | SHARE UPDATE EXCLUSIVE | No | Yes | ❌ No | Applies to new rows |
| `CLUSTER` / `VACUUM FULL` | ACCESS EXCLUSIVE | **Yes** (full rewrite) | No | ✅ **Yes** (`pg_repack`) | Use `pg_repack` |
| `CREATE TABLE ... PARTITION OF` | ACCESS EXCLUSIVE on **parent** (brief) | No | Blocked on parent while held | ❌ No | Brief and no scan, but it queues behind long-running queries and then blocks every reader of the parent |
| `ATTACH PARTITION` | SHARE UPDATE EXCLUSIVE on parent + scan of child | No | Yes | ➖ Native pattern | Add a validated `CHECK` matching the bound on the child first to skip the scan |
| `DETACH PARTITION` | ACCESS EXCLUSIVE | No | No | ➖ Use `CONCURRENTLY` | |
| `DETACH PARTITION CONCURRENTLY` | SHARE UPDATE EXCLUSIVE | No | Yes | ❌ No | PG 14+ |
Expand Down
3 changes: 2 additions & 1 deletion internal/cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ type MigrateCmd struct {

Alter string `help:"Imperative ALTER statement to run." name:"alter" required:""`
MaxTableSize byteSize `help:"Size threshold above which the optimistic attempt is skipped, measured as the table's full on-disk footprint: heap, indexes, and TOAST, all partitions (binary units: B, KiB, MiB, GiB, TiB)." default:"1GiB"`
JSON bool `help:"Emit the verdict as JSON."`
DryRun bool `help:"Classify and route the statement, print the plan, and execute nothing."`
JSON bool `help:"Emit the verdict (or dry-run plan) as JSON."`
}

// Run implements the migrate subcommand.
Expand Down
158 changes: 125 additions & 33 deletions internal/cli/diff.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,11 @@ import (
"fmt"
"io"
"os"
"strings"

"github.com/block/pg-sprite/pkg/dbconn"
"github.com/block/pg-sprite/pkg/planner"
"github.com/block/pg-sprite/pkg/router"
"github.com/block/pg-sprite/pkg/schemadiff"
"github.com/block/pg-sprite/pkg/statement"
)
Expand All @@ -22,9 +25,68 @@ type diffReport struct {
// TableExists reports whether the live table was found; when false the
// changes are the full desired schema.
TableExists bool `json:"table_exists"`
// Disposition is the routed plan's aggregate disposition: what would
// happen if the engine executed this plan now.
Disposition router.Disposition `json:"disposition"`
// Changes is the ordered statement plan; empty means the live table
// already matches the desired state.
Changes []schemadiff.Change `json:"changes"`
Changes []plannedChange `json:"changes"`
}

// plannedChange is one diff statement with its classification and routing:
// the derived SQL plus where the engine would send it and what would run.
type plannedChange struct {
schemadiff.Change
// Route is the planner's aggregate route for the statement.
Route planner.Route `json:"route"`
// Backend is the assigned execution strategy; empty for refusals.
Backend router.Backend `json:"backend,omitempty"`
// Disposition is what execution would do with the statement now.
Disposition router.Disposition `json:"disposition"`
// Decisions are the planner's per-operation classifications.
Decisions []planner.Decision `json:"decisions"`
// ExecSQL is the ordered SQL the native backend would run — the safer
// sequence when the planner constructed one. Empty for non-native
// routes.
ExecSQL []string `json:"exec_sql,omitempty"`
}

// classifyChanges routes every derived change through the shared
// classify-and-route pipeline. facts sharpen type-change classification;
// the zero value is valid and strictly more conservative.
func classifyChanges(changes []schemadiff.Change, facts planner.Facts) ([]plannedChange, router.Disposition, error) {
plans := make([]planner.Plan, 0, len(changes))
for _, ch := range changes {
plan, err := planner.Classify(ch.SQL, facts)
if err != nil {
return nil, "", fmt.Errorf("classify derived statement %q: %w", ch.SQL, err)
}
plans = append(plans, plan)
}
routed := router.Route(plans)
planned := make([]plannedChange, 0, len(changes))
for i, ch := range changes {
st := routed.Statements[i]
planned = append(planned, plannedChange{
Change: ch,
Route: st.Route,
Backend: st.Backend,
Disposition: st.Disposition,
Decisions: st.Decisions,
ExecSQL: st.ExecSQL,
})
}
return planned, routed.Disposition, nil
}

// liveFacts extracts the planner facts the live model provides: the
// canonical type of every live column.
func liveFacts(live schemadiff.Model) planner.Facts {
types := make(map[string]string, len(live.Columns))
for _, col := range live.Columns {
types[col.Name] = col.Type
}
return planner.Facts{ColumnTypes: types}
}

// run is the diff flow: parse and admit the desired file, introspect the
Expand All @@ -50,28 +112,36 @@ func (c *DiffCmd) run(ctx context.Context, out io.Writer) error {
defer pool.Close()

report := diffReport{Schema: c.Schema, Table: ds.Table, TableExists: true}
var changes []schemadiff.Change
var facts planner.Facts
live, err := schemadiff.Introspect(ctx, pool, c.Schema, ds.Table)
switch {
case errors.Is(err, schemadiff.ErrTableNotFound):
// No live table: the plan is the desired schema itself, qualified
// onto the target schema.
// onto the target schema, classified with zero facts (there are no
// live columns to sharpen type-change decisions).
report.TableExists = false
if report.Changes, err = qualifiedDesired(ds, c.Schema); err != nil {
if changes, err = qualifiedDesired(ds, c.Schema); err != nil {
return err
}
case err != nil:
return err
default:
facts = liveFacts(live)
desired, err := schemadiff.IntrospectDesired(ctx, pool, ds)
if err != nil {
return err
}
if report.Changes, err = schemadiff.Diff(c.Schema, live, desired); err != nil {
if changes, err = schemadiff.Diff(c.Schema, live, desired); err != nil {
return err
}
}
if report.Changes, report.Disposition, err = classifyChanges(changes, facts); err != nil {
return err
}
logger.Debug("diff derived",
"schema", c.Schema, "table", ds.Table, "changes", len(report.Changes), "table_exists", report.TableExists)
"schema", c.Schema, "table", ds.Table, "changes", len(report.Changes),
"table_exists", report.TableExists, "disposition", string(report.Disposition))

if c.JSON {
return writeJSON(out, report)
Expand Down Expand Up @@ -100,7 +170,7 @@ func qualifiedDesired(ds statement.DesiredSchema, schema string) ([]schemadiff.C
// writeJSON emits the report as JSON.
func writeJSON(out io.Writer, report diffReport) error {
if report.Changes == nil {
report.Changes = []schemadiff.Change{}
report.Changes = []plannedChange{}
}
enc := json.NewEncoder(out)
enc.SetIndent("", " ")
Expand All @@ -111,11 +181,13 @@ func writeJSON(out io.Writer, report diffReport) error {
}

// writePlanText emits the plan as an executable SQL script: one statement
// per line, destructive and lock-hazardous statements flagged with leading
// comment lines, and SQL comments for the no-change and missing-table cases
// so the output stays valid SQL. The header points at migrate as the
// executing front door: running this script directly bypasses the gate that
// refuses blocking statements.
// per line, each annotated with its route, destructive statements flagged,
// and SQL comments for the no-change and missing-table cases so the output
// stays valid SQL. Safer sequences appear as comment lines — never
// substituted into the script body, which stays the literal convergence
// plan (a CONCURRENTLY rewrite could not run inside a transaction block).
// The header points at migrate as the executing front door: running this
// script directly bypasses the gate that refuses blocking statements.
func writePlanText(out io.Writer, report diffReport) error {
if len(report.Changes) == 0 {
if _, err := fmt.Fprintln(out, "-- no changes: live table matches the desired schema"); err != nil {
Expand All @@ -136,39 +208,59 @@ func writePlanText(out io.Writer, report diffReport) error {
}
}
for _, ch := range report.Changes {
if ch.Destructive {
if _, err := fmt.Fprintln(out, "-- destructive"); err != nil {
return fmt.Errorf("write plan: %w", err)
}
if err := writeChangeText(out, ch); err != nil {
return err
}
if hazard := lockHazard(ch.Kind); hazard != "" {
if _, err := fmt.Fprintf(out, "-- %s\n", hazard); err != nil {
}
return nil
}

// writeChangeText emits one annotated statement of the text plan.
func writeChangeText(out io.Writer, ch plannedChange) error {
if _, err := fmt.Fprintf(out, "-- %s\n", annotate(ch)); err != nil {
return fmt.Errorf("write plan: %w", err)
}
if len(ch.ExecSQL) > 0 && ch.ExecSQL[0] != ch.SQL {
if _, err := fmt.Fprintln(out, "-- the engine would run instead:"); err != nil {
return fmt.Errorf("write plan: %w", err)
}
for _, safer := range ch.ExecSQL {
if _, err := fmt.Fprintf(out, "-- %s;\n", safer); err != nil {
return fmt.Errorf("write plan: %w", err)
}
}
if _, err := fmt.Fprintf(out, "%s;\n", ch.SQL); err != nil {
}
if ch.Destructive {
if _, err := fmt.Fprintln(out, "-- destructive"); err != nil {
return fmt.Errorf("write plan: %w", err)
}
}
if _, err := fmt.Fprintf(out, "%s;\n", ch.SQL); err != nil {
return fmt.Errorf("write plan: %w", err)
}
return nil
}

// lockHazard describes the blocking behavior of a change kind, empty when
// the statement is not expected to block writers. This is presentation for
// the text plan; the machine contract is the kind itself.
func lockHazard(kind schemadiff.ChangeKind) string {
switch kind {
case schemadiff.ChangeAlterType:
return "rewrites the table under ACCESS EXCLUSIVE, blocking reads and writes"
case schemadiff.ChangeSetNotNull:
return "full table scan under ACCESS EXCLUSIVE"
case schemadiff.ChangeAddConstraint:
return "validation scan or index build that blocks writes"
case schemadiff.ChangeCreateIndex:
return "blocks writes for the whole index build"
default:
return ""
// annotate renders one statement's route annotation: the route, the
// distinct decision reasons, and the availability note for backends this
// build does not implement.
func annotate(ch plannedChange) string {
var reasons []string
seen := map[planner.Reason]bool{}
for _, d := range ch.Decisions {
if !seen[d.Reason] {
seen[d.Reason] = true
reasons = append(reasons, string(d.Reason))
}
}
s := fmt.Sprintf("%s (%s)", ch.Route, strings.Join(reasons, ", "))
switch ch.Disposition {
case router.DispositionUnavailable:
s += ": needs the " + string(ch.Backend) + " backend, which is not implemented yet"
case router.DispositionRewriteRequired:
s += ": blocks as submitted and no online rewrite was constructed — the engine will not run it"
}
return s
}

// runFmt canonicalizes a desired-state schema file: every statement is
Expand Down
57 changes: 57 additions & 0 deletions internal/cli/diff_integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import (

"github.com/block/pg-sprite/internal/testutil"
"github.com/block/pg-sprite/pkg/dbconn"
"github.com/block/pg-sprite/pkg/planner"
"github.com/block/pg-sprite/pkg/router"
"github.com/block/pg-sprite/pkg/schemadiff"
)

Expand Down Expand Up @@ -78,6 +80,61 @@ func TestDiffPrintsOrderedPlanJSON(t *testing.T) {
schemadiff.ChangeCreateIndex,
}, kinds)
assert.Equal(t, []bool{true, false, false, false}, destructive)

// Every derived statement is classified and routed: the widen is proven
// binary-coercible by the live facts, SET NOT NULL and CREATE INDEX
// carry their safer native sequences, and the whole plan would execute.
assert.Equal(t, router.DispositionExecute, report.Disposition)
routes := make([]planner.Route, 0, len(report.Changes))
for _, ch := range report.Changes {
routes = append(routes, ch.Route)
assert.Equal(t, router.BackendNative, ch.Backend, ch.SQL)
assert.Equal(t, router.DispositionExecute, ch.Disposition, ch.SQL)
require.NotEmpty(t, ch.Decisions, ch.SQL)
}
assert.Equal(t, []planner.Route{
planner.RouteNative, planner.RouteNative, planner.RouteNative, planner.RouteNative,
}, routes)
assert.Equal(t, planner.ReasonBinaryCoercible, report.Changes[1].Decisions[0].Reason,
"live column types must feed the classifier")
assert.Equal(t, planner.ReasonSaferIdiom, report.Changes[2].Decisions[0].Reason)
assert.NotEqual(t, []string{report.Changes[2].SQL}, report.Changes[2].ExecSQL,
"SET NOT NULL carries its safer native sequence")
assert.Equal(t, planner.ReasonSaferIdiom, report.Changes[3].Decisions[0].Reason)
require.Len(t, report.Changes[3].ExecSQL, 1)
assert.NotEqual(t, report.Changes[3].SQL, report.Changes[3].ExecSQL[0],
"CREATE INDEX carries its concurrent rewrite")
}

// A desired state that needs a table rewrite routes to the copy-and-swap
// backend, and the routed plan says that backend is unavailable in this
// build — the plan is honest about what execution would do.
func TestDiffRoutesRewriteToCopyAndSwap(t *testing.T) {
url := testutil.StartPostgres(t)
pool, err := dbconn.NewPool(t.Context(), dbconn.Config{URL: url})
require.NoError(t, err)
defer pool.Close()
schema := testutil.NewSchema(t, pool)
_, err = pool.Exec(t.Context(), fmt.Sprintf(
"CREATE TABLE %s.events (id int PRIMARY KEY)", schema))
require.NoError(t, err)

cmd := newDiffCmd(t, url, schema, "CREATE TABLE events (id bigint PRIMARY KEY)")
cmd.JSON = true
var out strings.Builder
require.NoError(t, cmd.run(t.Context(), &out))

var report diffReport
require.NoError(t, json.Unmarshal([]byte(out.String()), &report))
assert.Equal(t, router.DispositionUnavailable, report.Disposition)
require.Len(t, report.Changes, 1)
ch := report.Changes[0]
assert.Equal(t, planner.RouteCopyAndSwap, ch.Route)
assert.Equal(t, router.BackendCopyAndSwap, ch.Backend)
assert.Equal(t, router.DispositionUnavailable, ch.Disposition)
assert.Empty(t, ch.ExecSQL)
require.Len(t, ch.Decisions, 1)
assert.Equal(t, planner.ReasonTypeRewrite, ch.Decisions[0].Reason)
}

// diff must never write: the live table is bit-identical before and after.
Expand Down
Loading
Loading