diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fea3440120..1ad86f3ab8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -963,7 +963,7 @@ jobs: exit 1 - name: pytest (Postgres authority, memory, knowledge, isolation and the SQL surface) run: >- - pytest -q + pytest -vv tests/integration/test_pg_authority.py tests/integration/test_pg_durable_publish.py tests/integration/test_pg_pipeline_metrics.py @@ -1011,6 +1011,9 @@ jobs: tests/integration/test_cell_prepare.py tests/integration/test_cell_attach.py tests/integration/test_cell_place.py + tests/integration/test_cell_restore.py + tests/integration/test_cell_restore_directory.py + tests/integration/test_cell_restore_reconcile.py tests/integration/test_cell_update_directory.py tests/integration/test_cell_update_addresses.py tests/integration/test_cell_update_run_pg.py @@ -1108,6 +1111,34 @@ jobs: export LOOM_CELL_PLACE_DRILL_DSN="postgres://loom:loomdev@127.0.0.1:$port/loom" docker update --memory 1g --memory-swap 1g loom-scale-place-drill-pg loom-scale-place-drill-pg-standby pytest -q tests/integration/test_cell_place_durability.py + - name: pytest restore durability on a private physical standby + timeout-minutes: 8 + env: + LOOM_PG_IMAGE: postgres:${{ matrix.postgres }} + LOOM_PG_CONTAINER: loom-scale-restore-drill-pg + LOOM_PG_HOST_PORT: "55432" + LOOM_CELL_RESTORE_DRILL: "1" + LOOM_CELL_RESTORE_DRILL_DSN: postgres://loom:loomdev@localhost:55432/loom + run: | + set -euo pipefail + for name in loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby; do + if docker container inspect "$name" >/dev/null 2>&1; then + echo "drill container already exists: $name" >&2 + exit 1 + fi + done + if docker network inspect loom-scale-restore-drill-pg-net >/dev/null 2>&1; then + echo "drill network already exists" >&2 + exit 1 + fi + cleanup() { + docker rm -f -v loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby || true + docker network rm loom-scale-restore-drill-pg-net || true + } + trap cleanup EXIT + bash dev/ci_start_postgres.sh + docker update --memory 1g --memory-swap 1g loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby + pytest -q tests/integration/test_cell_restore_durability.py # The tenant-isolation gates again, against a second server that enforces auth: # the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens. # They run whether or not the memory suites passed (`!cancelled()`). diff --git a/docs/architecture/cells.md b/docs/architecture/cells.md index a78e60bce4..be8ca39096 100644 --- a/docs/architecture/cells.md +++ b/docs/architecture/cells.md @@ -1490,6 +1490,7 @@ entry points and the named exceptions: | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cell_prepare._write_rows` | Fresh claim, observed local rows, exact lock backend | | `cell_attach_pg._mode_transaction` | Attach's claim, supervised mode try, recording/first/fingerprint comparison, exact lock backend | +| `cell_restore` schema, tenancy, state and trigger repair | Fresh claim before repair, same lock backend, retained generation at restart and final acknowledgement | | `cell_control.backout` | Fresh claim, fenced close and zero-placement read, observed local rows, exact lock backend | | `cell_update_pg.write.change` | Fresh directory claim; owner's bounded transaction on the exact ATTACH backend, paired registration and exact observed local identity/pair before tenancy repair and acceptance write | | `directory.attach_begin`, `attach_place`, `attach_enforced`, `attach_serving`, `remove` | Claimed generation, full registration and exact pair, including retries/no-ops | @@ -1511,6 +1512,91 @@ required fleet turnover and offline restore of generations. N-1 data-plane build read and insert on the additive schema; old control clients and router handlers cannot participate in this protocol. +## Restoring a cell + +Stop every server, worker and CLI writer before restoring a cell's stores. Keep old +endpoints and restored copies stopped too. Run `loom router migrate` with the router's +configuration, and apply pending ClickHouse schema with `loom migrate` while the cell +is stopped. Then, with the restored cell's own configuration and the router operator +credential, run: + +```bash +loom cell restore --cell --wait 300 +``` + +The command verifies the recorded Postgres and ClickHouse deployment identities and +epochs against each other and the directory, checks the bucket name and prefix, and +probes the configured stores directly. A different address is allowed when the store +identity matches. It applies pending cell Postgres migrations on its marked operator +session, then copies every permanent directory placement back into `loom.tenancy`, +repairs first/mode/own rows and verifies the complete trigger set. A closed cell or a +placement quota does not exclude its existing orgs. The directory's placement count +and limits stay unchanged. The command holds the same operator lock as prepare, attach +and place; another operation on that cell must finish first. + +Before Postgres migration, probes or metadata repair, restore reads the full operator +registration and claims a fresh control generation on that exact lock backend. It +compares the registration and exact fingerprint pair with the bounded restore +descriptor and recorded stores. A count-only traversal restart retains that claim; +a fresh full operator read at restart and final acknowledgement must match its +registration, fingerprints and generation. A delayed claim from an earlier controller +can invalidate the attempt even while restore holds its local lock. Restore refuses +that change and requires a new attempt; it never adopts another generation mid-run. +A lost claim acknowledgement or local session also ends the attempt. + +The single `--wait` deadline includes connection setup, every repair phase and cleanup. +The raw connection belongs to its cleanup owner before deployment checks or codec +initialization can wait. Cleanup sends public connection close before cancelling work; +a second signal does not cancel that close early or extend the original deadline. +A late connection cannot become usable after the attempt ends. A transport abort +leaves an unknown outcome rather than proving that the backend released its locks. +Restore, placement and cell control bind their cleanup owner synchronously inside the +shared Postgres connector, before its deployment or codec waits. A closed owner refuses +initialization and closes a late session within its original deadline. The connector +still enforces process admission, participant attribution and retirement seals; callers +without an explicit owner retain connector cleanup until initialization finishes. +After verifying the ClickHouse authority, restore checks the first-cell write policy +through that owned Postgres session before probes or repair. This preflight uses +the command's existing deadline and close path. + +Each directory page uses the shared callback session owner with its own seven-statement +budget. The owner begins and commits one audited transaction, closes that proxy before +pool release and reserves cleanup inside the same deadline. A page locks its named cell +row; it does not take the directory-wide control lock. + +Pages have fixed row and byte bounds and use the directory's `(cell, org)` index. A +placement added during the scan restarts it; a configuration change refuses the run. +There is no maximum number of orgs. Select a longer `--wait` for a larger cell. Every +completed batch, including a no-op retry, acknowledges new WAL before success. A +failure or interrupted commit can leave completed batches in place: repeat the same +command while the cell stays stopped. + +Mode rises from `recording` to `enforced` when either store record requires it. The +immutable first row must match the directory. After attach records fingerprints, the +directory's current and next values replace the restored own row exactly. Before that, +the own row stays as restored, including its absence. The command creates all org and +tenancy guards even with that approval absent; the cell remains gated. Its result names +whether prepare is still required. Complete reconciliation first, then start gated +processes and rerun prepare before serving if the own row is absent. A retained own +fingerprint that differs from the configured values is reported separately as +`configuration_recovery_required`; the own row stays unchanged. Restore a matching +configuration or follow the normal configuration update procedure before starting +processes. `prepare_required` describes an absent own row and is not a readiness claim. + +The first cell can still hold orgs that attach has not placed yet, even after it records +`enforced`: attach records directory enforcement before completing the placement +batches. While it is `attaching` and the sole directory cell, restore preserves and +reports those tenancy rows and directs the operator to resume attach. A serving cell, +a later cell, or a cell with another registered cell refuses an unexpected local tenancy +row. It neither creates a placement from restored content nor deletes an extra row. + +Run the ordinary [restore reconciliation](../operations/backup-and-recovery.md#restoring-one-store) +after this command, before serving resumes. Cell metadata repair also works after both +stores are restored; reconciliation still refuses when both lost the evidence it needs. +Restored signing keys, revocations, sessions, bots and SCIM rows retain the existing +backup limitations. Activity checks can detect a running process or a recent worker +pass; heartbeat expiry cannot prove that every process is stopped. + ## Operations - `loom cell restore`, after any store restore and before `loom reconcile`, writes the diff --git a/docs/operations/backup-and-recovery.md b/docs/operations/backup-and-recovery.md index 0a47670b4f..baffdb1a1b 100644 --- a/docs/operations/backup-and-recovery.md +++ b/docs/operations/backup-and-recovery.md @@ -242,17 +242,21 @@ which runs again what the restored store lost of the other's record. Before a ClickHouse restore with `LOOM_PG_PAYLOAD_CLEAR_ENABLED` on, stop the payload clearing and choose the backup as [the Postgres journal and a ClickHouse restore](#the-postgres-journal-and-a-clickhouse-restore) -says. Once a clearing pass has run, run step 3's `loom reconcile` with a build that -carries `010_revision_delivered_at.sql`, and start the deployment on that build in step 4 +says. Once a clearing pass has run, run step 4's `loom reconcile` with a build that +carries `010_revision_delivered_at.sql`, and start the deployment on that build in step 5 (rolling-upgrade [rule 43](../architecture/rolling-upgrades.md#the-rules)); then: 1. Stop every server and worker of the deployment. 2. Restore the store from its backup, and note the backup's time from the backup's own metadata (for ClickHouse, the start of `BACKUP`; for Postgres, the recovery target). -3. Run `loom reconcile --restored clickhouse --point