Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ on:
branches: [main]
pull_request:

# Read-only by default; the one job that needs more (changes) grants it
# job-scoped.
permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Expand Down Expand Up @@ -56,8 +61,10 @@ jobs:
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
with:
go-version-file: go.mod
# Pin the same golangci-lint major used locally (v2 config format);
Expand All @@ -71,8 +78,10 @@ jobs:
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
with:
go-version-file: go.mod
- run: make build
Expand Down Expand Up @@ -102,8 +111,10 @@ jobs:
env:
PG_VERSION: ${{ matrix.pg }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
with:
go-version-file: go.mod
- run: make test
Expand Down
63 changes: 63 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Release

on:
push:
tags:
- "v*"

# All grants are job-scoped so a future job added to this file inherits
# nothing by default.
permissions: {}

jobs:
release:
runs-on: ubuntu-latest
permissions:
# Publish the GitHub release and its assets.
contents: write
# Sign build provenance for the released artifacts.
id-token: write
attestations: write
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
fetch-depth: 0
persist-credentials: false

# A tag alone must not be able to publish. Requiring the tagged commit
# to be an ancestor of main means it went through the PR gate (the
# PG 14-18 matrix behind the required all-green check) before it could
# be released — a tag pointing at an unmerged or never-pushed commit
# fails here.
- name: Require the tagged commit to be on main
run: git merge-base --is-ancestor "$GITHUB_SHA" origin/main

- name: Set up Go
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
with:
go-version-file: go.mod
# Release artifacts must never be built from a shared cache a
# previous workflow could have poisoned.
cache: false

# Belt and braces on top of the ancestry gate: re-run the suite against
# the exact tree being released.
- name: Test the tagged tree
run: make test

- name: Run GoReleaser
uses: goreleaser/goreleaser-action@9c156ee8a17a598857849441385a2041ef570552 # v6.3.0
with:
version: v2.8.2
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# Provenance ties each artifact to this workflow run and commit, so a
# consumer can verify a binary was built here rather than trusting the
# checksums file published alongside it.
- name: Attest build provenance
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-checksums: dist/checksums.txt
44 changes: 44 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
version: 2

builds:
- id: pg-sprite
main: ./cmd/pg-sprite
binary: pg-sprite
# The SQL parser is Wasm (wasilibs/go-pgquery), so the binary is pure Go
# and cross-compiles without cgo.
env:
- CGO_ENABLED=0
ldflags:
- -X main.version={{.Tag}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64

archives:
- id: pg-sprite
builds:
- pg-sprite
format: tar.gz
name_template: "pg-sprite_{{ .Version }}_{{ .Os }}_{{ .Arch }}"

release:
# A published version is a fixed artifact: never replace the assets of an
# existing release, so re-running the workflow for a tag cannot silently
# swap binaries under a version people may have pinned. If a draft release
# was created via the GitHub UI before the workflow ran, delete the draft
# and re-run the workflow.
mode: keep-existing

checksum:
name_template: "checksums.txt"

changelog:
sort: asc
filters:
exclude:
- "^docs:"
- "^test:"
- "^ci:"
46 changes: 33 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,48 @@

> Working name — see the naming task in the research build tracker.

An online schema-change engine for **Aurora PostgreSQL** (and RDS/community
PostgreSQL 14+): a decoupled **planner → router → executor** design where the
An online schema-change engine for **PostgreSQL** (community, RDS, and
Aurora; 14+): a decoupled **planner → router → executor** design where the
planner classifies each change, the router picks a strategy, and
interchangeable executors carry it out — the cheap native PostgreSQL idiom
when one exists (`CONCURRENTLY`, `NOT VALID` + `VALIDATE`, fast default,
`USING INDEX`), and a log-based, checksum-gated, resumable copy-and-swap when
a genuine table rewrite is unavoidable.

**Status: Phase 1 (optimistic front door).** `pg-sprite migrate --alter '…'`
runs easy `ALTER TABLE` changes directly under tight lock/statement budgets
and refuses everything else with a structured verdict (exit code 2): index
maintenance gets a pointer to the `CONCURRENTLY` idiom, and changes that need
a table rewrite — caught by the size guard or a cancelled bounded attempt —
get an explicit **not native-safe** verdict. `diff`, `fmt`, and `lint` are
still stubs. The design docs and the phased build plan live in
[docs/](docs/) — start with [docs/README.md](docs/README.md).
`USING INDEX`), while a log-based, checksum-gated, resumable copy-and-swap for
genuine table rewrites lands in a later phase.

The planner is PostgreSQL's missing `ALGORITHM=` / `LOCK=` declaration: MySQL
lets authors assert a cost bracket and a concurrency impact and fails closed
when either can't be honored — PostgreSQL silently runs whichever cost
applies. pg-sprite proves both dimensions before execution, routes each
change to the safest sequence that exists, and refuses with a structured
verdict when it can't prove one (see
[docs/postgres-online-ddl-reference.md](docs/postgres-online-ddl-reference.md)).

**Status: Phases 1 and 2.1–2.4.** The parse boundary, declarative diff,
classifier, and router seam are implemented. `pg-sprite migrate --alter '…'`
runs a bounded optimistic native attempt; routed execution beyond that attempt
lands in Phase 3. Changes without an available backend get a structured
refusal (exit code 2). `lint` is still a stub. The design docs and the phased
build plan live in [docs/](docs/) — start with
[docs/README.md](docs/README.md); the vision — what pg-sprite is and is not —
is [docs/vision.md](docs/vision.md).

The codebase is partitioned into a small safety-critical core and a
periphery — **[SAFETY.md](SAFETY.md)** says which packages are which and the
rules that apply inside the core. Read it before changing anything under
`pkg/`.

## Install

Release archives for linux/darwin on amd64/arm64, with `checksums.txt`, are
published on the [releases page](https://github.com/block/pg-sprite/releases)
once tags exist. The binary is pure Go (the SQL parser is Wasm), so on any
other platform — or without waiting for a release — `go install` works with
no C toolchain:

```sh
go install github.com/block/pg-sprite/cmd/pg-sprite@latest
```

## Development

```sh
Expand Down
23 changes: 12 additions & 11 deletions SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The invariant registry (invariant IDs referenced below) lives in

| Package | Core? | Status | Invariants enforced |
| --- | --- | --- | --- |
| `pkg/dbconn` — pool defaults, advisory lock, terminate-blockers, retries, RDS TLS | ✅ core | exists (Phase 0) | LK-1, LK-2 primitives |
| `pkg/dbconn` — pool defaults, terminate-blockers, retries, RDS TLS; advisory table lock planned | ✅ core | exists; advisory table lock planned | LK-2 primitives; LK-1 planned |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | exists (Phase 1: table-size guard); grows through Phase 2 | ST-6, RF-1..RF-5 |
| `pkg/executor` — bounded optimistic attempt; native executor later | ✅ core | exists (Phase 1: attempt-under-budget); Executor contract at Phase 2–3 | LK-2 (attempt bound) |
| `pkg/checksum` — chunk verifier, continuous checker, repair | ✅ core | planned (Phase 5) | CO-1, CO-2, CO-3 |
Expand All @@ -27,16 +27,17 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phase 2); `pkg/lint` planned (Phase 2) | (CO-7 holds at the parse boundary) |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phases 2.1–2.4); `pkg/lint` planned | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`/`status` exist (Phase 1); rest stubs | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, and `fmt` exist; `lint` is a stub | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
| orchestrator adapter | ❌ periphery | planned (Phase 11) | OC-* hold *at* the boundary |
| `internal/testutil` | ❌ test-only | exists | — |

¹ **The planner is deliberately outside the core.** Its verdicts are *requests*, not
permissions: a wrong "native-safe" verdict is capped by the executor's own `lock_timeout` bound;
a wrong "copy" verdict produces a wasteful but *correct* schema change (the checksum still gates).
permissions: a wrong "native-safe" verdict is capped by the executor's own `lock_timeout` bound.
Today a "copy" route reports unavailable; once copy-and-swap exists, a wrong "copy" verdict will
produce a wasteful but *correct* schema change because the checksum will still gate it.
The core executors re-verify their own preconditions and never trust that the planner checked.

## Rules inside the core
Expand All @@ -46,18 +47,18 @@ The short version — the full rules live in [docs/tcb-model.md](docs/tcb-model.
- **Never trust callers.** Every dangerous operation re-verifies its preconditions, whoever the
requester is (CLI, planner, orchestrator). The periphery may request; the core enforces.
- **Domain types make illegal states unrepresentable.** Validating passages return proof types
with package-private constructors (`statement.Classified`, `PreflightedTable`,
`VerifiedShadow`, `CleanWatermark`, `TableLock`); dangerous APIs accept only proof types —
e.g. the cutover swap accepts only a `VerifiedShadow`.
with package-private constructors (today `preflight.PreflightedTable`; later phases add
`VerifiedShadow`, `CleanWatermark`, and `TableLock`); dangerous APIs accept only proof types —
e.g. the planned cutover swap will accept only a `VerifiedShadow`.
- **Put a limit on everything.** Every loop bounded, every queue bounded, every retry counted,
every wait deadlined. An unbounded anything in a core package is a review-blocking defect.
- **Assert the positive and the negative space; pair assertions across boundaries.** Invariant
violations use a distinct error class (`ErrInvariantViolation`) naming the invariant ID, and
always abort fail-closed — never a warning, never retried.
violations will use a distinct error class (`ErrInvariantViolation`) naming the invariant ID
once the executor phases land, and always abort fail-closed — never a warning, never retried.
- **Locality of behavior.** The enforcement point of an invariant carries a `// INV: <id>`
comment so a reviewer or agent can grep the ID and see the whole enforcement in one screen.
- **Dependencies inside the core become part of the core.** Current core dependency list:
`pgx/v5`, `pglogrepl`, stdlib. Adding one requires a recorded decision (see the rubric in
`pgx/v5`, stdlib. The future decode path will add `pglogrepl`. Adding one requires a recorded decision (see the rubric in
[docs/tcb-model.md](docs/tcb-model.md) — copy small things, take pinned dependencies only
for load-bearing expertise).
pg-sprite **never imports `block/spirit` as a module**: we port ideas with citations, not
Expand Down
4 changes: 2 additions & 2 deletions cmd/pg-sprite/main.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Command pg-sprite is an online schema-change engine for Aurora PostgreSQL.
// Command pg-sprite is an online schema-change engine for PostgreSQL.
package main

import (
Expand All @@ -17,7 +17,7 @@ var version = "dev"
func main() {
k := kong.Parse(cli.New(),
kong.Name("pg-sprite"),
kong.Description("An online schema-change engine for Aurora PostgreSQL."),
kong.Description("An online schema-change engine for PostgreSQL."),
kong.UsageOnError(),
kong.Vars{"version": version},
)
Expand Down
Loading
Loading