Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ change to the safest sequence that exists, and refuses with a structured
verdict when it can't prove one (see
[docs/postgres-online-ddl-reference.md](docs/postgres-online-ddl-reference.md)).

**Status: Phases 1 and 2.1–2.4.** The parse boundary, declarative diff,
classifier, and router seam are implemented. `pg-sprite migrate --alter '…'`
runs a bounded optimistic native attempt; routed execution beyond that attempt
lands in Phase 3. Changes without an available backend get a structured
refusal (exit code 2). `lint` is still a stub. The design docs and the phased
**Status: Phases 1 and 2.1–2.5.** The parse boundary, declarative diff,
classifier, router seam, versioned dry-run plan report, and offline linter
are implemented. `pg-sprite migrate --alter '…'` runs a bounded optimistic
native attempt; routed execution beyond that attempt lands in Phase 3.
Changes without an available backend get a structured refusal (exit code 2).
The design docs and the phased
build plan live in [docs/](docs/) — start with
[docs/README.md](docs/README.md); the vision — what pg-sprite is and is not —
is [docs/vision.md](docs/vision.md).
Expand Down
4 changes: 2 additions & 2 deletions SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/plan`, `pkg/lint` — classify/diff/route/report | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), `pkg/router` (backend assignment + availability policy), and `pkg/plan` (versioned dry-run plan report) exist (Phases 2.1–2.5); `pkg/lint` planned | (CO-7 holds at the parse boundary) |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/plan`, `pkg/lint` — classify/diff/route/report | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), `pkg/router` (backend assignment + availability policy), `pkg/plan` (versioned dry-run plan report), and `pkg/lint` (offline typed findings) exist (Phases 2.1–2.5) | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, and `fmt` exist; `lint` is a stub | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, `fmt`, and `lint` exist | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
| orchestrator adapter | ❌ periphery | planned (Phase 11) | OC-* hold *at* the boundary |
| `internal/testutil` | ❌ test-only | exists | — |
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ checkpoint/resume, tuned for Aurora. That is the gap this engine targets.
| [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. |
| [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. |
| [plan-report.md](plan-report.md) | The **plan report contract** — the versioned JSON shape both front doors emit for dry-run plans: fields, closed vocabularies, the fingerprint identity, required consumer behavior for unknown versions/values, and one generated example per source (pinned by test). |
| [lint-report.md](lint-report.md) | The **lint report contract** — the versioned JSON shape `pg-sprite lint` emits for offline CI gating: finding fields (verbatim SQL, line/column), the codes table, severities and exit behavior, the offline-conservatism rules, and how the contract versions relative to the plan report. |
| [testing.md](testing.md) | The **test-suite guide** — how to run the suite (unit, per-major, all supported majors, compose database), current coverage, the remaining executor-phase test obligations, and the vanilla-PostgreSQL-matrix vs real-Aurora validation boundary. |
| [schemabot-integration.md](schemabot-integration.md) | The **single home for orchestrator integration** — how SchemaBot (the reference orchestrator) drives the engine: the pluggable-engine overview, the verb mappings, the concrete adapter contract, and the design constraints (OC-* invariants) the integration imposes on the core. |

Expand Down
12 changes: 6 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,10 @@ judged by the same rules regardless of how it arrived:
| **Introspect** | `pkg/schemadiff` | live catalog (and desired DDL applied to a scratch schema) → schema models | The classifier and diff need *facts*, not text: column types, defaults, and constraint state come from PostgreSQL's own catalog, not a reimplementation of its semantics |
| **Diff** | `pkg/schemadiff` | desired model vs live model → ordered DDL operations | Declarative mode is a front-end that *produces statements*; its output enters the same pipeline as hand-written DDL, so both modes get identical safety treatment |
| **Classify** | `pkg/planner` | each operation + introspected facts → native-safe · needs-rewrite · refuse, with the safer native sequence where one exists | The safety decision lives in one pure, testable place — PostgreSQL's missing `ALGORITHM=`/`LOCK=` declaration ([design-principles.md](design-principles.md)) |
| **Lint** | `pkg/lint` | classified operations → pass or structured refusal | Policy-level rejection of unsafe or unsupported changes *before* any write — separate from the mechanical can-this-run-online judgment |
| **Lint** | `pkg/lint` | classified operations → typed findings (errors refuse, warnings advise) | Policy-level rejection of unsafe or unsupported changes *before* any write — separate from the mechanical can-this-run-online judgment |

Parse, introspect, diff, and classify exist today (Phases 1–2); lint is the one stage not
yet built (see the [package map](#package-map) for per-package status).
All five stages exist today (Phases 1–2.5); see the [package map](#package-map) for
per-package status.

The planner's verdicts are **requests, not permissions** — executors re-verify their own
preconditions. Which components are safety-critical (and the stricter rules inside that
Expand Down Expand Up @@ -122,16 +122,16 @@ different levels of commitment:

| Package | Role | Status |
| --- | --- | --- |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `diff` · `fmt` · `lint` · `status` | `migrate` · `diff` · `fmt` · `status` exist; `lint` is a stub |
| `internal/cli` | Command tree and flag handling | `migrate` · `diff` · `fmt` · `status` exist; `lint` is a stub |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `diff` · `fmt` · `lint` · `status` | all five exist |
| `internal/cli` | Command tree and flag handling | all five exist |
| `internal/testutil` | Test harness: containerized PostgreSQL, throwaway schemas | exists |
| `pkg/dbconn` | Pool with bounded session timeouts, retries, RDS/Aurora auto-TLS (embedded CA bundle), terminate-blockers; advisory-lock mutual exclusion lands here | exists |
| `pkg/statement` | `go-pgquery` (Wasm `libpg_query`) parse boundary, typed per-operation descriptors, and advisory rewrites (never hand-parse SQL); migration-time shadow DDL + fingerprints are derived by `pkg/schemadiff` via scratch-DB execute-and-introspect | exists |
| `pkg/preflight` | Precondition verification and refusals before any write | exists (Phase 1: table-size guard); grows through Phase 2 |
| `pkg/verdict` | Structured outcome contract (executed / refused + reason + safer idiom), rendering, exit codes | exists (Phase 1) |
| `pkg/schemadiff` | Execute-and-introspect desired state, introspect the live catalog, and produce an ordered declarative diff | exists |
| `pkg/planner` | Classify typed operations and emit safer native SQL | exists |
| `pkg/lint` | Policy-level rejection of unsafe or unsupported operations | planned |
| `pkg/lint` | Offline lint findings with typed codes: unsupported operations are errors; blocking idioms, rewrites, and destructive drops are warnings | exists (Phase 2.5) |
| `pkg/plan` | Versioned machine-readable dry-run plan report — the one JSON contract both front doors emit and an orchestrator consumes | exists (Phase 2.5) |
| `pkg/router` | Route classified statements to native / copy-and-swap / refuse dispositions; copy-and-swap reports unavailable until that backend lands | exists (Phase 2.4) |
| `pkg/executor` | Bounded optimistic native attempt; the `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) lands in Phase 3 | bounded optimistic attempt exists |
Expand Down
97 changes: 97 additions & 0 deletions docs/lint-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# The lint report contract

The lint report is the machine-readable result of `pg-sprite lint` — the offline checker
that runs the same parse-and-classify pipeline as the front doors, with zero live facts and
no database. It exists so CI can gate a DDL script before any environment sees it. This
document is the contract: the fields, the closed vocabularies, and the behavior required of
a consumer. The Go source of truth is `pkg/lint`; tests in `pkg/lint` and `internal/cli` pin
everything documented here.

## Versioning: `format_version`

Every report carries `format_version`. A consumer that does not recognize the version must
**reject the report** — never guess at field semantics. The version covers the field shape
and the closed vocabularies below (codes, severities, and the embedded planner reasons):
adding a value to any of them is a contract change and bumps `format_version`, even if no
field is added or renamed.

### Relationship to the plan report

The lint report and the [plan report](plan-report.md) are **versioned independently** — each
carries its own `format_version`, and they move separately. They share one vocabulary: a
finding's `reason` draws from the plan report's Reasons set, and the lint `format_version`
pins the reason set a lint consumer must understand, exactly as the plan `format_version`
does for plan consumers. A consumer of both must track both versions; understanding plan v1
says nothing about lint v1.

## Consumer behavior for unknown values

Every enum field draws from a closed vocabulary listed here. A consumer that meets a value
it does not recognize must **treat the finding as an error and fail the gate** — never
ignore it and proceed. This mirrors the linter's own posture: an unknown planner route
becomes an error-severity refusal, never a silent pass.

## Offline conservatism

The linter sees only the script — no catalog, no column types, no server. Every judgment is
therefore the planner's fail-closed judgment: a change lint passes without findings can
still sharpen at execution time, but a change lint flags will never quietly get worse. Two
codes make the conservatism visible instead of burying it:

- `possible-table-rewrite` marks a decision the planner could not verify (see the codes
table) — the engine would take the heavy path, but a live database might prove the change
free.
- `destructive` includes every index drop, because the linter cannot see whether an index
is unique — and a dropped unique index whose gap admitted duplicate writes cannot be
recreated at all.

## Report fields

| Field | Type | Presence | Meaning |
|---|---|---|---|
| `format_version` | int | always | Contract version; reject unknown versions. |
| `postgres_versions` | string | always | The inclusive PostgreSQL major-version range the offline rules are derived for (`14-18`, see [postgresql-version-support.md](postgresql-version-support.md)). The linter never sees a server, so a stored report names the assumptions behind it instead. |
| `findings` | array | always | The findings in statement order; `[]` (never `null`) means the script is clean. |
| `errors` | int | always | Count of error-severity findings. |
| `warnings` | int | always | Count of warning-severity findings. |

## Finding fields

| Field | Type | Presence | Meaning |
|---|---|---|---|
| `statement` | int | always | 1-based index of the statement in the script. |
| `line` | int | always | 1-based source line of the statement's first token, for CI annotations. |
| `column` | int | always | 1-based source column of the statement's first token. |
| `sql` | string | always | The statement's **verbatim source text** (without the trailing semicolon), so it can be found in the source by exact match. Unlike the plan report's canonical rendering, a lint finding points back at the file the author wrote. |
| `operation` | string | always | Operator-facing label of the flagged operation (`DROP COLUMN legacy_a`). Display only — never branch on it. |
| `code` | string | always | The typed finding kind (see Codes). Automation branches on this, never on prose. |
| `severity` | string | always | What the engine would do about it (see Severities). |
| `reason` | string | classifier findings | The planner's typed cause, drawn from the plan report's Reasons vocabulary. Absent for destructive findings, which are a property of the operation, not a routing decision. |
| `suggestion` | array | when constructible | The ordered safer SQL to run instead, present only for `blocking-idiom` findings where the planner constructed the rewrite. Its absence still means the submitted form blocks — the planner does not construct rewrites for multi-operation statements or for operations that need catalog knowledge (ATTACH PARTITION's proving CHECK). |

## Codes (`code`)

| Value | Severity | Example statement | Meaning |
|---|---|---|---|
| `unsupported-operation` | error | `ALTER TABLE t ADD CONSTRAINT x EXCLUDE USING gist (room WITH =)` | No known safe path — the engine refuses it. |
| `blocking-idiom` | warning | `CREATE INDEX i ON t (c)` | The submitted form blocks readers or writers and a safer native form exists; `suggestion` carries it when the linter can construct one. |
| `table-rewrite` | warning | `ALTER TABLE t ALTER COLUMN c TYPE jsonb USING c::jsonb` | The operation provably rewrites the table — only the engine's copy-and-swap path can run it online. |
| `possible-table-rewrite` | warning | `ALTER TABLE t ALTER COLUMN c TYPE bigint` | The linter cannot verify the operation against live column facts, so the engine would fail closed to the rewrite path — but the change may be a free relabel a live database would prove. |
| `destructive` | warning | `ALTER TABLE t DROP COLUMN legacy` | The operation discards live structure (a column, constraint, or index drop) and cannot be undone by re-running the schema. |

## Severities (`severity`)

| Value | Meaning | Exit behavior |
|---|---|---|
| `error` | The engine would refuse the statement — it cannot execute as written. | `pg-sprite lint` exits non-zero. |
| `warning` | The engine would execute the statement, but it has a safer form, needs a heavier path, or discards live structure. | Warnings alone exit zero. |

Only error-severity findings flip the exit code today; a policy layer (per-code gating,
inline suppression) is a planned extension and will be introduced as a contract change.

## Text output

Without `--json`, findings render one per line in the conventional linter shape —
`name:line:column: severity: code — operation` — where `name` is the linted file path or
`<stdin>`. The text form is for humans and editors; automation consumes the JSON report.
A clean script prints nothing and exits zero.
12 changes: 6 additions & 6 deletions docs/low-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ seam inside the copy-and-swap executor is the same idea applied one level down.
│ pkg/statement parse ALTER/CREATE (go-pgquery) │
│ pkg/schemadiff introspect live schema → diff vs desired → ordered ALTERs │
│ pkg/planner per op: native-safe | needs-rewrite | refuse │
│ pkg/lint reject unsafe/unsupported up front (planned)
│ pkg/lint reject unsafe/unsupported up front (offline findings)
│ │ │
│ ▼ Plan (ordered steps, classified per operation) │
└──────┬─────────────────────────────────────────────────────────────────────┘
Expand Down Expand Up @@ -177,8 +177,8 @@ pattern *per migration*:
- **expand/contract via pgroll** for prod-critical breaking changes where **instant
reversibility** and **two live schema versions** matter more than transparency.

The classifier, declarative diff, dry-run, and status reporting are shared by every backend;
linting remains planned. An `Executor` interface (`Plan`, `Execute`, `Status`, `Abort`) is also
The classifier, declarative diff, dry-run, lint, and status reporting are shared by every
backend. An `Executor` interface (`Plan`, `Execute`, `Status`, `Abort`) is also
planned; `pkg/executor` currently provides only the bounded optimistic native attempt. Until the
in-house copy-and-swap executor
lands in a later phase, every `needs-rewrite` change is refused as **not native-safe** rather than
Expand Down Expand Up @@ -318,13 +318,13 @@ For each parsed statement the classifier produces a record along the lines of:
- Richer `risk`, `reversible`, and `requires_app_coordination` metadata is a future extension.

Classification belongs to `pkg/planner`; `pkg/statement` supplies typed operations and
`pkg/lint` remains a stub.
`pkg/lint` maps the classifier's decisions to offline findings with typed codes.

### CLI behaviour (modes)

| Invocation | Behaviour |
| --- | --- |
| `lint` | Stub; no lint engine is implemented yet. |
| `lint` | Offline (no database): classify every statement with zero live facts and report typed findings — unsupported operations are errors (non-zero exit), blocking idioms (with the safer SQL), conservative rewrites, and destructive drops are warnings. |
| `diff` / `migrate --dry-run` | Print the classified, routed plan and safer SQL where applicable. **Never executes.** `diff` has no `--dry-run` flag because it never executes. |
| `migrate` (default) | Run the Phase 1 statement gate, preflight, and bounded optimistic native attempt. It does not yet execute classifier-produced safer SQL. |
| `migrate --force` (planned Phase 3) | Run each statement **exactly as submitted**, bypassing the safe rewrite. Gated — see below. |
Expand Down Expand Up @@ -600,13 +600,13 @@ pkg/schemadiff/ -> execute-and-introspect desired state + live introspecti
pkg/planner/ -> classify each operation and construct safer native SQL
pkg/router/ -> assign classified statements to available backends
pkg/plan/ -> versioned machine-readable dry-run plan report (both front doors)
pkg/lint/ -> offline typed lint findings (errors refuse, warnings advise)
pkg/executor/ -> bounded optimistic native attempt only
pkg/dbconn/ -> bounded database connections
pkg/preflight/ -> migration preflight checks
pkg/verdict/ -> typed outcomes

Planned:
pkg/lint/ -> unsafe-DDL linters (currently a CLI stub)
pkg/migration/ -> orchestrator + runner + cutover
pkg/decode/ -> logical-decoding client
pkg/copier/ -> parallel chunked copy
Expand Down
Loading
Loading