Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
256 commits
Select commit Hold shift + click to select a range
dcf7996
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
6229357
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
a64cbb2
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
84e6460
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
5615d41
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
b0f694b
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
fefe1e0
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
bd4cda3
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
3ba813c
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
98fc333
Operator placement durably records its directory winner and local ten…
zlareb1 Nov 16, 2026
042dd5e
Prove cell placement retries survive standby promotion
zlareb1 Nov 16, 2026
bb8e041
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
1f05935
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
55c0918
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
4a7bc88
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
4193ffd
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
c650119
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
5884b04
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
691f4f0
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
4a25c41
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
0c37c68
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
07f9ea1
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
78c2808
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
62c6b47
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
95bb74c
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
d5ec3b9
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
20681aa
Check member-role privileges and repeat attach checks after placement
zlareb1 Nov 16, 2026
cb95fff
Require effective predefined role privileges before attach
zlareb1 Nov 16, 2026
fde3798
Describe the effective owner privileges attach requires
zlareb1 Nov 16, 2026
cad839b
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
bc7246d
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
cc4ed54
Fence cell control mutations across local lock loss
zlareb1 Nov 16, 2026
e0f756b
Index cell control recovery and its upgrade refusal
zlareb1 Nov 16, 2026
3f62dfd
Validate the cell name before recovery requests
zlareb1 Nov 16, 2026
d31a08d
WIP: Preserve stopped-cell restore implementation
zlareb1 Nov 16, 2026
84ca4c8
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
db89cb2
Keep attaching cells closed through state cleanup
zlareb1 Nov 16, 2026
4380186
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
417712d
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
c601646
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
a72ce93
Restore closes owned sessions before cancelling work
zlareb1 Nov 16, 2026
51a8ff6
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
1f69c41
Supervise native cell control cleanup before cancelling work
zlareb1 Nov 16, 2026
5f76257
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
946ef2f
Validate prospective cell settings without changing process configura…
zlareb1 Nov 16, 2026
16c9aaf
Restore reports interrupted and retained configuration outcomes
zlareb1 Nov 16, 2026
c9d1f0a
Check cell fingerprint reservations with shared org pattern rules
zlareb1 Nov 16, 2026
9ac5f0e
Keep target settings values out of update errors
zlareb1 Nov 16, 2026
f4702f6
Use one owner for native cell control sessions
zlareb1 Nov 16, 2026
789b25c
Refuse tagged YAML parser failures without quoting values
zlareb1 Nov 16, 2026
f9986b5
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5a62070
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
a4e5906
Compose cell control fencing with stopped metadata restore
zlareb1 Nov 17, 2026
5809472
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
8a76924
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
836d056
Leave failed restore trigger sessions to their cleanup owner
zlareb1 Nov 17, 2026
deacbaa
Close directory control sessions before cancelling their work
zlareb1 Nov 17, 2026
0c10636
Measure cell triggers with the original API and delivery workload
zlareb1 Nov 17, 2026
2e1e1ae
Validate cell update targets against their registration
zlareb1 Nov 17, 2026
02b9f02
Fence restore on its original operator session
zlareb1 Nov 17, 2026
baaa7cc
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
459dc34
Fix control batch bounds and schema coexistence checks
zlareb1 Nov 17, 2026
9caeb23
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
0d7929a
Observe active initialization before interrupting restore
zlareb1 Nov 17, 2026
d71bfc9
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
92d6d9b
Use common fatal transport for directory session close
zlareb1 Nov 17, 2026
f1e8231
Compose cell update targets with guarded participant retirement
zlareb1 Nov 17, 2026
1fbc8a5
Close directory control sessions before cancelling their work
zlareb1 Nov 17, 2026
e39dbe2
Fix control batch bounds and schema coexistence checks
zlareb1 Nov 17, 2026
8f4785a
Use common fatal transport for directory session close
zlareb1 Nov 17, 2026
c53f7e9
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
a13eb96
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
0410333
Validate serving configuration and serialize fresh delivery rows
zlareb1 Nov 17, 2026
8c03200
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
b429beb
Compose cell control fencing with configuration updates
zlareb1 Nov 17, 2026
da5ef76
Compose native session ownership with operator placement
zlareb1 Nov 17, 2026
c4abfc0
Reuse directory session ownership for restore pages
zlareb1 Nov 17, 2026
d27ff2e
Close placement sessions before cancelling their work
zlareb1 Nov 17, 2026
8018e30
Keep attach cleanup inside its owning deadline
zlareb1 Nov 17, 2026
40be106
Reserve cell configuration and configured orgs atomically
zlareb1 Nov 17, 2026
ea49879
Directory codec failures retain their foreground owner
zlareb1 Nov 17, 2026
8335af4
Describe configuration placement concurrency
zlareb1 Nov 17, 2026
7aa65d6
Directory close waits through repeated cancellation signals
zlareb1 Nov 17, 2026
2a702ab
Directory abort failures preserve the first cleanup error
zlareb1 Nov 17, 2026
46a0f57
Compose operator placement with cell configuration updates
zlareb1 Nov 17, 2026
c66786e
Preserve native owner errors when fallback abort fails
zlareb1 Nov 17, 2026
6a1d018
Compose shared directory cleanup and foreground pool initialization
zlareb1 Nov 17, 2026
cdce8ec
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
70c21f2
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
9d18614
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
0835f33
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
5bef195
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
6fca7d9
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
723109d
Compose native control ownership with bounded abort cleanup
zlareb1 Nov 17, 2026
914d239
Compose placement with shared directory and migration cleanup
zlareb1 Nov 17, 2026
e27d47a
Compose stopped restore with shared native control cleanup
zlareb1 Nov 17, 2026
f378a28
Match persisted vector precision and preserve attach cleanup errors
zlareb1 Nov 17, 2026
8b2d1eb
WIP: coordinate cell configuration updates and address proofs
zlareb1 Nov 17, 2026
9dd0934
WIP: Preserve selected driver errors through artifact diagnostics
zlareb1 Nov 17, 2026
faf7a05
Preserve cell updates through shared cleanup failures
zlareb1 Nov 17, 2026
15f842a
Preserve artifact sync failures through file close
zlareb1 Nov 17, 2026
361bbbb
Bound placement exchanges and reserve both request bodies
zlareb1 Nov 17, 2026
a9580c1
The cell trigger driver retains its first failure through cleanup
zlareb1 Nov 17, 2026
2ff8931
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
ef7e606
Reserve native fingerprint defaults in placement bodies
zlareb1 Nov 17, 2026
d6e5eec
Scope SCIM credentials to the configured cell
zlareb1 Nov 17, 2026
c8ec1d6
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
e8b9d6a
WIP: Compose attach checks with native cell control
zlareb1 Nov 17, 2026
69e9f2d
Exercise attach cleanup with a fenced directory snapshot
zlareb1 Nov 17, 2026
01621af
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
5d04387
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
17eee33
Keep native HTTP failures through client cleanup
zlareb1 Nov 17, 2026
dfa83dd
Preserve native control request failures through HTTP cleanup
zlareb1 Nov 17, 2026
6f61b36
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
5db501c
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
6badd7d
Check the participant identity in cell heartbeat tests
zlareb1 Nov 17, 2026
ce81164
Native nonce cleanup keeps the first failure
zlareb1 Nov 17, 2026
06355a3
Bind trigger timing receipts to distinct API and CLI configurations
zlareb1 Nov 17, 2026
f485b25
Retain the control deadline during cell configuration updates
zlareb1 Nov 17, 2026
c884dd1
Preserve native errors through nonce cleanup
zlareb1 Nov 17, 2026
46aa499
Compose permanent retirement with fenced cell controls
zlareb1 Nov 17, 2026
d62fdaa
Merge certified participant retirement into cell attachment
zlareb1 Nov 17, 2026
0a94de8
Native commands preserve their first failure through cleanup
zlareb1 Nov 17, 2026
c11f8ea
Native commands retain failures through participant cleanup
zlareb1 Nov 17, 2026
a7dda54
Compose current main with native cell attachment
zlareb1 Nov 17, 2026
b253487
WIP: preserve reviewed attach lifecycle corrections
zlareb1 Nov 17, 2026
d182c26
WIP: preserve attach registration and error regressions
zlareb1 Nov 17, 2026
858c90d
WIP: validate serving cell registration and retain wire error codes
zlareb1 Nov 17, 2026
9c02c28
WIP: compose cell attachment with the web listener
zlareb1 Nov 17, 2026
2a379d0
Native command cleanup has a configured deadline
zlareb1 Nov 17, 2026
300c665
Check repeated cancellation against the original clock deadline
zlareb1 Nov 17, 2026
fd3bc09
Compose cell updates with the attach foundation
zlareb1 Nov 17, 2026
98afee1
Retain the deterministic cell cleanup deadline check
zlareb1 Nov 17, 2026
c7efa71
Format the cell cleanup deadline test
zlareb1 Nov 17, 2026
8444e8d
Check that native participant censuses exclude certified history
zlareb1 Nov 17, 2026
30cce29
Compose the merged Postgres batching measurement repair
zlareb1 Nov 17, 2026
11f7b46
Document update control writers and nonce cleanup limits
zlareb1 Nov 17, 2026
77b0638
SCIM person authorization checks active identities across requests
zlareb1 Nov 17, 2026
2397af5
Describe the fixture owner's recorded ownership digest
zlareb1 Nov 17, 2026
8e9c013
Compose SCIM authorization with cell attachment
zlareb1 Nov 17, 2026
a49d33b
Document SCIM person checks and expose generated schema on CI mismatch
zlareb1 Nov 17, 2026
55db6b1
Name the cell actor in placement fence tests
zlareb1 Nov 17, 2026
9f84a2b
Exclude certified retirement history from participant reads
zlareb1 Nov 17, 2026
17f8d53
Compare declaration work after parser initialization
zlareb1 Nov 17, 2026
295ea3c
Personal bearer subjects cannot be impersonated without a bot
zlareb1 Nov 17, 2026
9254790
Prove worker crash recovery within the fixture process namespace
zlareb1 Nov 17, 2026
74f1d6b
Workers enroll before their cell acceptance read
zlareb1 Nov 17, 2026
a7aba18
Worker service loops finish before the final drain
zlareb1 Nov 17, 2026
724299b
Quickstart matches live workers to their registrations
zlareb1 Nov 17, 2026
824870f
Caller routes check orgless bearers in the configured signup org
zlareb1 Nov 17, 2026
727e601
Trigger acceptance runs on isolated CI stores
zlareb1 Nov 17, 2026
a881cc9
SCIM fixtures clean only the identities they create
zlareb1 Nov 17, 2026
1e71631
Trigger acceptance waits for TCP and reports owned startup state
zlareb1 Nov 17, 2026
5762c94
Placement fence tests name the cell actor
zlareb1 Nov 17, 2026
79cd546
Trigger stores use the CI server configuration
zlareb1 Nov 17, 2026
c975843
Trigger ClickHouse runs as its data-directory owner
zlareb1 Nov 17, 2026
d9b6821
Prepare unit tests use their current control owner
zlareb1 Nov 17, 2026
bc6bcd6
Trigger setup verifies the owned Keeper configuration
zlareb1 Nov 17, 2026
30d57c3
Worker crash checks report the unproved predicate
zlareb1 Nov 17, 2026
f32c6f1
Attach fixtures isolate authority records and preserve failure causes
zlareb1 Nov 17, 2026
7ad45ce
Attach acceptance verifies two genuine API replicas
zlareb1 Nov 18, 2026
69b6efe
Worker crash checks read captured identities before signalling
zlareb1 Nov 18, 2026
eadf2bf
Standby drill checks ledger visibility after interrupted commit
zlareb1 Nov 18, 2026
bd11af3
Trigger acceptance keeps independent write heads stable
zlareb1 Nov 18, 2026
449769e
Retirement readers filter the native participant UUID
zlareb1 Nov 18, 2026
644f653
The worker census skips exited supervisor children
zlareb1 Nov 18, 2026
88165da
Trigger setup refusals retain bounded child diagnostics
zlareb1 Nov 18, 2026
1a1cee2
Diagnostic read failures preserve the original command refusal
zlareb1 Nov 18, 2026
1fcd50a
The replica fixture matches its migrated vector width
zlareb1 Nov 18, 2026
4e49edf
Keep attach and document source checks in the Postgres workflow
zlareb1 Nov 18, 2026
6521d1b
Wait for supervised replacement enrollment after retirement
zlareb1 Nov 18, 2026
3e82257
Attach deadline tests use an owned cleanup clock
zlareb1 Nov 18, 2026
d5e1411
Resource samples require a complete process forest
zlareb1 Nov 18, 2026
3f69214
Trigger protocol metadata describes its implemented runtime
zlareb1 Nov 18, 2026
c9685ba
Phone citation geometry waits for the selected Ask option
zlareb1 Nov 18, 2026
a400ba3
Native trigger failures retain bounded driver diagnostics
zlareb1 Nov 18, 2026
24a159f
Provider smoke retains bounded reset failure evidence
zlareb1 Nov 18, 2026
09c4f8b
Merge privacy, summary and probe changes into cell attachment
zlareb1 Nov 18, 2026
2460fa2
Trigger acceptance retains bounded Postgres failure evidence
zlareb1 Nov 18, 2026
e9bd5b1
Retirement census distinguishes method calls from local cleanup
zlareb1 Nov 18, 2026
79943b7
Worker parents retire verified stopped children
zlareb1 Nov 18, 2026
ba55790
Worker child retirement follows the application boundary
zlareb1 Nov 18, 2026
23bd673
Verify captured worker subtrees in native recovery checks
zlareb1 Nov 18, 2026
74d59a4
Test child retirement limits and clarify native token minting
zlareb1 Nov 18, 2026
9d6d586
Prove exact exit of single-thread zombie participants
zlareb1 Nov 18, 2026
fa65ac4
Compose cell attachment with the session-protection heartbeat
zlareb1 Nov 18, 2026
da95f2d
Match subtree diagnostic controls to kernel observations
zlareb1 Nov 18, 2026
a1ebaad
Distinguish store guard failures and reap interrupted test children
zlareb1 Nov 18, 2026
3d86ef4
Wait for the published ClickHouse HTTP listener
zlareb1 Nov 18, 2026
00ceb04
Wait for the complete single-thread exit witness
zlareb1 Nov 18, 2026
7112196
Reap the retirement fixture child after interrupted waits
zlareb1 Nov 18, 2026
1d752a7
Collect the attach lock fixture's delayed rollback task
zlareb1 Nov 18, 2026
efcfc40
Isolate retirement gates between integration scenarios
zlareb1 Nov 18, 2026
96bd15e
Trigger acceptance retains bounded staging failure evidence
zlareb1 Nov 18, 2026
1841255
Exercise absent head locks at the native staging boundary
zlareb1 Nov 18, 2026
d9977f7
Compose cell attach with current CLI and settings surfaces
zlareb1 Nov 18, 2026
310c407
Tracker feed holds native admission through owned cleanup
zlareb1 Nov 18, 2026
81bf248
Lock memory heads through ordered exact-key probes
zlareb1 Nov 18, 2026
5278b3d
Collect native fixture resources after setup failures
zlareb1 Nov 18, 2026
32d6790
Keep attach tenancy writes on their owning backend
zlareb1 Nov 18, 2026
00fafb2
Document unproved attach lock cleanup
zlareb1 Nov 18, 2026
b0c7dd6
Align attach cleanup and pairing descriptions
zlareb1 Nov 18, 2026
1140e9b
SCIM authorization retains token and SSO evidence
zlareb1 Nov 18, 2026
7589246
SCIM fixtures attempt every owned cleanup
zlareb1 Nov 18, 2026
cea20e1
Cell updates retain native attachment ownership
zlareb1 Nov 18, 2026
5c6913e
Cell configuration bounds include operator placements
zlareb1 Nov 18, 2026
d45c18a
Cell controls retain bounded placement exchanges
zlareb1 Nov 18, 2026
f14e39e
Placement fixtures preserve initialization failures
zlareb1 Nov 18, 2026
ce16e7b
Restore composes with current cell control and placement
zlareb1 Nov 18, 2026
86c4510
Restore fixtures finish independent resource cleanup
zlareb1 Nov 18, 2026
9a6bd2e
Restore refuses non-UTF8 databases before control writes
zlareb1 Nov 18, 2026
5c989f4
Cell attach checks fresh SCIM provisioning before serving
zlareb1 Nov 18, 2026
bdfe9ff
SCIM admits provisioned account aliases and ownerless resources
zlareb1 Nov 18, 2026
ca069f1
SSO refuses ambiguous accounts before JIT repair
zlareb1 Nov 18, 2026
f110384
Configuration updates verify target SCIM provisioning
zlareb1 Nov 18, 2026
6b1f82c
Merge SCIM callback ownership correction
zlareb1 Nov 18, 2026
c44c9ab
Merge the cell configuration SCIM foundation
zlareb1 Nov 18, 2026
8dcfd39
Cell control tests bound cancellation and isolate provisioning rows
zlareb1 Nov 18, 2026
4c14d59
Merge cell control test isolation and cancellation fixes
zlareb1 Nov 18, 2026
9ac2980
Restore cancellation tests bound readiness and clean owned tasks
zlareb1 Nov 18, 2026
6f704fd
Participant census rejects conflicts before selecting the newest row
zlareb1 Nov 18, 2026
a32df3c
Enforced cells require active provisioning for people
zlareb1 Nov 19, 2026
021e29c
Merge main for provisioning integration checks
zlareb1 Nov 19, 2026
0518d08
Keep provisioning census scoped to active attachment
zlareb1 Nov 19, 2026
7bb1a93
Standalone authorization keeps its provisioning policy
zlareb1 Nov 19, 2026
49436bf
Native doctor checks retain the first-cell identity
zlareb1 Nov 19, 2026
9a6b2b8
Cell update checks the selected provisioning policy
zlareb1 Nov 19, 2026
eda7df8
Cell restore uses the fenced configuration foundation
zlareb1 Nov 19, 2026
e9a5974
Keep raw cell sessions under their cleanup owner
zlareb1 Nov 19, 2026
86d7c2d
Restore owns the public preflight state read
zlareb1 Nov 19, 2026
a3ffc15
Cell control owns raw sessions before initialization
zlareb1 Nov 19, 2026
3d2999a
Merge shared connection ownership into cell restore
zlareb1 Nov 19, 2026
dc5b8af
Placement durability observes the participant session
zlareb1 Nov 19, 2026
9ae41a2
Merge provisioning prerequisite into cell update
zlareb1 Nov 19, 2026
8439773
Merge reviewed cell update prerequisite into restore
zlareb1 Nov 19, 2026
8609701
Restore uses the merged cell control foundation
zlareb1 Nov 20, 2026
4ef905a
Restore durability uses the canonical ASGI origin
zlareb1 Nov 20, 2026
f0612d3
Keep placement test setup outside the HTTP deadline
zlareb1 Nov 20, 2026
5b0e16f
Pause rollup refreshes while seeding equal-timestamp rows
zlareb1 Nov 20, 2026
2b3826d
Compose the rollup fixture refresh repair
zlareb1 Nov 20, 2026
ad51e08
Report native integration nodes during restore validation
zlareb1 Nov 20, 2026
0b79e10
Record restore test phases before cancellation
zlareb1 Nov 20, 2026
396db8b
Format the restore phase evidence controls
zlareb1 Nov 20, 2026
94740aa
Separate restore phase records from pytest progress
zlareb1 Nov 20, 2026
f42b392
Isolate restore fixture cleanup and activity rows
zlareb1 Nov 20, 2026
84fb0db
Preserve deadline fixture isolation during restore integration
zlareb1 Nov 20, 2026
19f8c77
Test placement identity refusal with a manual clock
zlareb1 Nov 20, 2026
8b7124b
Keep restore reconciliation heartbeats in a private table
zlareb1 Nov 20, 2026
7e44a8d
Keep the first restore probe error during nonce cleanup
zlareb1 Nov 20, 2026
6267de0
Adopt restore fixture database before pooled reads
zlareb1 Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -963,7 +963,7 @@ jobs:
exit 1
- name: pytest (Postgres authority, memory, knowledge, isolation and the SQL surface)
run: >-
pytest -q
pytest -vv
tests/integration/test_pg_authority.py
tests/integration/test_pg_durable_publish.py
tests/integration/test_pg_pipeline_metrics.py
Expand Down Expand Up @@ -1011,6 +1011,9 @@ jobs:
tests/integration/test_cell_prepare.py
tests/integration/test_cell_attach.py
tests/integration/test_cell_place.py
tests/integration/test_cell_restore.py
tests/integration/test_cell_restore_directory.py
tests/integration/test_cell_restore_reconcile.py
tests/integration/test_cell_update_directory.py
tests/integration/test_cell_update_addresses.py
tests/integration/test_cell_update_run_pg.py
Expand Down Expand Up @@ -1108,6 +1111,34 @@ jobs:
export LOOM_CELL_PLACE_DRILL_DSN="postgres://loom:loomdev@127.0.0.1:$port/loom"
docker update --memory 1g --memory-swap 1g loom-scale-place-drill-pg loom-scale-place-drill-pg-standby
pytest -q tests/integration/test_cell_place_durability.py
- name: pytest restore durability on a private physical standby
timeout-minutes: 8
env:
LOOM_PG_IMAGE: postgres:${{ matrix.postgres }}
LOOM_PG_CONTAINER: loom-scale-restore-drill-pg
LOOM_PG_HOST_PORT: "55432"
LOOM_CELL_RESTORE_DRILL: "1"
LOOM_CELL_RESTORE_DRILL_DSN: postgres://loom:loomdev@localhost:55432/loom
run: |
set -euo pipefail
for name in loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby; do
if docker container inspect "$name" >/dev/null 2>&1; then
echo "drill container already exists: $name" >&2
exit 1
fi
done
if docker network inspect loom-scale-restore-drill-pg-net >/dev/null 2>&1; then
echo "drill network already exists" >&2
exit 1
fi
cleanup() {
docker rm -f -v loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby || true
docker network rm loom-scale-restore-drill-pg-net || true
}
trap cleanup EXIT
bash dev/ci_start_postgres.sh
docker update --memory 1g --memory-swap 1g loom-scale-restore-drill-pg loom-scale-restore-drill-pg-standby
pytest -q tests/integration/test_cell_restore_durability.py
# The tenant-isolation gates again, against a second server that enforces auth:
# the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens.
# They run whether or not the memory suites passed (`!cancelled()`).
Expand Down
86 changes: 86 additions & 0 deletions docs/architecture/cells.md
Original file line number Diff line number Diff line change
Expand Up @@ -1490,6 +1490,7 @@ entry points and the named exceptions:
| --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cell_prepare._write_rows` | Fresh claim, observed local rows, exact lock backend |
| `cell_attach_pg._mode_transaction` | Attach's claim, supervised mode try, recording/first/fingerprint comparison, exact lock backend |
| `cell_restore` schema, tenancy, state and trigger repair | Fresh claim before repair, same lock backend, retained generation at restart and final acknowledgement |
| `cell_control.backout` | Fresh claim, fenced close and zero-placement read, observed local rows, exact lock backend |
| `cell_update_pg.write.change` | Fresh directory claim; owner's bounded transaction on the exact ATTACH backend, paired registration and exact observed local identity/pair before tenancy repair and acceptance write |
| `directory.attach_begin`, `attach_place`, `attach_enforced`, `attach_serving`, `remove` | Claimed generation, full registration and exact pair, including retries/no-ops |
Expand All @@ -1511,6 +1512,91 @@ required fleet turnover and offline restore of generations. N-1 data-plane build
read and insert on the additive schema; old control clients and router handlers cannot
participate in this protocol.

## Restoring a cell

Stop every server, worker and CLI writer before restoring a cell's stores. Keep old
endpoints and restored copies stopped too. Run `loom router migrate` with the router's
configuration, and apply pending ClickHouse schema with `loom migrate` while the cell
is stopped. Then, with the restored cell's own configuration and the router operator
credential, run:

```bash
loom cell restore --cell <cell> --wait 300
```

The command verifies the recorded Postgres and ClickHouse deployment identities and
epochs against each other and the directory, checks the bucket name and prefix, and
probes the configured stores directly. A different address is allowed when the store
identity matches. It applies pending cell Postgres migrations on its marked operator
session, then copies every permanent directory placement back into `loom.tenancy`,
repairs first/mode/own rows and verifies the complete trigger set. A closed cell or a
placement quota does not exclude its existing orgs. The directory's placement count
and limits stay unchanged. The command holds the same operator lock as prepare, attach
and place; another operation on that cell must finish first.

Before Postgres migration, probes or metadata repair, restore reads the full operator
registration and claims a fresh control generation on that exact lock backend. It
compares the registration and exact fingerprint pair with the bounded restore
descriptor and recorded stores. A count-only traversal restart retains that claim;
a fresh full operator read at restart and final acknowledgement must match its
registration, fingerprints and generation. A delayed claim from an earlier controller
can invalidate the attempt even while restore holds its local lock. Restore refuses
that change and requires a new attempt; it never adopts another generation mid-run.
A lost claim acknowledgement or local session also ends the attempt.

The single `--wait` deadline includes connection setup, every repair phase and cleanup.
The raw connection belongs to its cleanup owner before deployment checks or codec
initialization can wait. Cleanup sends public connection close before cancelling work;
a second signal does not cancel that close early or extend the original deadline.
A late connection cannot become usable after the attempt ends. A transport abort
leaves an unknown outcome rather than proving that the backend released its locks.
Restore, placement and cell control bind their cleanup owner synchronously inside the
shared Postgres connector, before its deployment or codec waits. A closed owner refuses
initialization and closes a late session within its original deadline. The connector
still enforces process admission, participant attribution and retirement seals; callers
without an explicit owner retain connector cleanup until initialization finishes.
After verifying the ClickHouse authority, restore checks the first-cell write policy
through that owned Postgres session before probes or repair. This preflight uses
the command's existing deadline and close path.

Each directory page uses the shared callback session owner with its own seven-statement
budget. The owner begins and commits one audited transaction, closes that proxy before
pool release and reserves cleanup inside the same deadline. A page locks its named cell
row; it does not take the directory-wide control lock.

Pages have fixed row and byte bounds and use the directory's `(cell, org)` index. A
placement added during the scan restarts it; a configuration change refuses the run.
There is no maximum number of orgs. Select a longer `--wait` for a larger cell. Every
completed batch, including a no-op retry, acknowledges new WAL before success. A
failure or interrupted commit can leave completed batches in place: repeat the same
command while the cell stays stopped.

Mode rises from `recording` to `enforced` when either store record requires it. The
immutable first row must match the directory. After attach records fingerprints, the
directory's current and next values replace the restored own row exactly. Before that,
the own row stays as restored, including its absence. The command creates all org and
tenancy guards even with that approval absent; the cell remains gated. Its result names
whether prepare is still required. Complete reconciliation first, then start gated
processes and rerun prepare before serving if the own row is absent. A retained own
fingerprint that differs from the configured values is reported separately as
`configuration_recovery_required`; the own row stays unchanged. Restore a matching
configuration or follow the normal configuration update procedure before starting
processes. `prepare_required` describes an absent own row and is not a readiness claim.

The first cell can still hold orgs that attach has not placed yet, even after it records
`enforced`: attach records directory enforcement before completing the placement
batches. While it is `attaching` and the sole directory cell, restore preserves and
reports those tenancy rows and directs the operator to resume attach. A serving cell,
a later cell, or a cell with another registered cell refuses an unexpected local tenancy
row. It neither creates a placement from restored content nor deletes an extra row.

Run the ordinary [restore reconciliation](../operations/backup-and-recovery.md#restoring-one-store)
after this command, before serving resumes. Cell metadata repair also works after both
stores are restored; reconciliation still refuses when both lost the evidence it needs.
Restored signing keys, revocations, sessions, bots and SCIM rows retain the existing
backup limitations. Activity checks can detect a running process or a recent worker
pass; heartbeat expiry cannot prove that every process is stopped.

## Operations

- `loom cell restore`, after any store restore and before `loom reconcile`, writes the
Expand Down
33 changes: 20 additions & 13 deletions docs/operations/backup-and-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,29 +242,34 @@ which runs again what the restored store lost of the other's record.
Before a ClickHouse restore with `LOOM_PG_PAYLOAD_CLEAR_ENABLED` on, stop the payload
clearing and choose the backup as
[the Postgres journal and a ClickHouse restore](#the-postgres-journal-and-a-clickhouse-restore)
says. Once a clearing pass has run, run step 3's `loom reconcile` with a build that
carries `010_revision_delivered_at.sql`, and start the deployment on that build in step 4
says. Once a clearing pass has run, run step 4's `loom reconcile` with a build that
carries `010_revision_delivered_at.sql`, and start the deployment on that build in step 5
(rolling-upgrade [rule 43](../architecture/rolling-upgrades.md#the-rules)); then:

1. Stop every server and worker of the deployment.
2. Restore the store from its backup, and note the backup's time from the backup's own
metadata (for ClickHouse, the start of `BACKUP`; for Postgres, the recovery target).
3. Run `loom reconcile --restored clickhouse --point <time>` (or `--restored postgres`),
3. For a cell, run `loom cell restore --cell <cell>` with its configuration and the
router operator credential, keeping every process stopped. It repairs cell metadata
and pending cell Postgres schema before reconciliation
([cell restore](../architecture/cells.md#restoring-a-cell)).
4. Run `loom reconcile --restored clickhouse --point <time>` (or `--restored postgres`),
which prints what it would run again; then the same with `--apply`.
4. Start the deployment.
5. After a ClickHouse restore, run each `loom rechunk` the reconcile printed, and
5. Start the deployment.
6. After a ClickHouse restore, run each `loom rechunk` the reconcile printed, and
upload and ingest again each source it names under "to ingest again": a knowledge
document's chunks and a dataset's rows live in ClickHouse alone.

A [cell](../architecture/cells.md)'s Postgres carries org triggers that refuse a row for an
org the cell holds no tenancy row for, and a `loom.tenancy` guard that refuses a tenancy
row, from a session that does not mark itself. A logical restore into a cell is such a
session: with every process of the cell stopped, run it as one command with the marker,
`PGOPTIONS='-c loom.tenancy_checked=on' pg_restore ...`, then start one pod with
`LOOM_BOOT_MIGRATE` on before the others. A data-only `pg_restore --disable-triggers` into
`PGOPTIONS='-c loom.tenancy_checked=on' pg_restore ...`, then run `loom cell restore`
while it stays stopped. That command applies pending Postgres schema and verifies the
triggers before reconciliation and startup. A data-only `pg_restore --disable-triggers` into
an existing schema ends by enabling every trigger as an origin trigger, which leaves
`loom.cell_first`'s two triggers off under `session_replication_role = replica`; that pod's
boot sets them back to `ALWAYS`, and `loom doctor` names them until it does.
`loom.cell_first`'s two triggers off under `session_replication_role = replica`; `loom cell restore`
sets them back to `ALWAYS`, and `loom doctor` names them until it does.

On the Helm chart ([Kubernetes](kubernetes.md)), with release `loom`:

Expand All @@ -279,8 +284,10 @@ On the Helm chart ([Kubernetes](kubernetes.md)), with release `loom`:
is on a build without `010_revision_delivered_at.sql`, first
`helm upgrade ... --set image.tag=<a build that carries it>` (or `image.digest`, which
wins when set) with step 2's new host and its three `--set` overrides, so no pod
starts, and keep that image in step 4. Run the reconcile in a Job made from the web
pod's own spec, so it has the release's environment, Secret and Postgres CA, its own
starts, and keep that image in step 4. For a cell, first run `loom cell restore`
in a Job with the cell's configuration and router operator credential, and wait for
it to complete while every process stays stopped. Run the reconcile in a Job made
from the web pod's own spec, so it has the release's environment, Secret and Postgres CA, its own
component label (the NetworkPolicy selects it, the Service does not), and no probes:

```bash
Expand Down Expand Up @@ -454,15 +461,15 @@ ClickHouse may still hold versions the backup lacks:
last horizon. With the default horizon that allows the newest backup or the one
before, if they started on schedule. An older backup may lack a version whose payload
is already cleared, and that version's content is then lost.
5. Turn the switch back on when the deployment starts again at step 4 of
5. Turn the switch back on when the deployment starts again at step 5 of
[restoring one store](#restoring-one-store), and not before: by then
`loom reconcile --apply` has owed a delivery again for every revision committed after
the point, and a revision keeps its payload while it owes a delivery. First check
`LOOM_CH_BACKUP_INTERVAL_MINUTES` and `LOOM_PG_PAYLOAD_RESTORE_HORIZON_MINUTES`
against the backup schedule: the restore brought back their overrides from the
backup's time, so re-apply any changed since. A version ClickHouse still lacks after
the restore keeps its payload, and each pass that reaches it counts it in
`missing_in_clickhouse`, which can stay above zero. When step 4 of
`missing_in_clickhouse`, which can stay above zero. When step 5 of
[restoring one store](#restoring-one-store) starts a build that carries
`010_revision_delivered_at.sql` on a deployment that served one without it,
the switch stays off until one clearing horizon after that build serves alone
Expand Down
1 change: 1 addition & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,7 @@ python scripts/codegen/docs_reference.py
| `loom cell place` | Place an org in this cell and record its tenancy. | `--cell`, `--org`, `--router` |
| `loom cell prepare` | Register this cell in the router's directory and write its rows. | `--cell`, `--host`, `--credential-sha256`, `--probe-url`, `--wait`, `--router` |
| `loom cell attach` | Attach the prepared first cell: place its orgs, serve. | `--cell`, `--wait`, `--scan-budget`, `--dry-run`, `--ch-allow-user`, `--ch-allow-user-directory`, `--ch-allow-object`, `--ch-allow-replica`, `--pg-allow-role`, `--revoke-unprovisioned`, `--router` |
| `loom cell restore` | Repair a stopped cell's metadata before reconciliation. | `--cell`, `--router`, `--wait` |
| `loom cell update` | Reserve configuration, finish or revert its rollout, or refresh verified addresses. | `--cell`, `--config`, `--finish`, `--revert`, `--probe-url`, `--revoke-unprovisioned`, `--router`, `--wait` |
| `loom tracker-probe` | Read-only first contact with the tracker source: counts and newest timestamps per upstream table per credential, breaker states, the configured tracker namespaces. | |
| `loom tracker-contract-check` | Read-only: does each tracker source's schema meet the record contract (docs/features/tracker/record-contract.md)? | `--source`, `--namespace`, `--json` |
Expand Down
18 changes: 15 additions & 3 deletions server/cli/_ch.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,19 +155,28 @@ def _outage(e: BaseException) -> bool:
return isinstance(e, pg.PostgresUnavailableError) or pg.is_outage(e)


async def check_org_free(free: OrgFree) -> None:
async def check_org_free(
free: OrgFree,
*,
state_reader: Callable[[], Awaitable[tuple[str, str]]] | None = None,
) -> None:
"""With `LOOM_CELL` set, refuse an org-free `write` or `revocation` command whose
stores belong to another cell: their `first` row names it. A CLI pointed at another
cell's stores always finds one there; an absent row, or state tables not migrated
yet, passes, so a later cell runs these before it is prepared. A failed read refuses,
but a revocation runs through an outage (docs/architecture/cells.md, "Invariants")."""
but a revocation runs through an outage (docs/architecture/cells.md, "Invariants").
An owned reader keeps its session and total deadline; its timeout escapes to its
owner with a fixed diagnostic rather than the generic ten-second read refusal.
"""
from server.core import pg
from server.settings import settings

if free.kind == "read" or not settings.cell or not pg.enabled():
return
try:
_found, name = await asyncio.wait_for(_read_state(), timeout=_STATE_READ_S)
_found, name = await asyncio.wait_for(
(state_reader or _read_state)(), timeout=_STATE_READ_S
)
except pg.AuthorityMismatch:
raise # another deployment's Postgres: refused as `check_configured` refuses it
except Exception as e: # noqa: BLE001 — an unread cell state refuses a write
Expand All @@ -178,6 +187,9 @@ async def check_org_free(free: OrgFree) -> None:
file=sys.stderr,
)
return
if state_reader is not None and isinstance(e, TimeoutError):
# The owner's shorter deadline may have expired; never expose driver text.
raise TimeoutError("cell state read reached its deadline") from None
raise OrgNotHeld(
f"refusing: this cell's state could not be read "
f"({type(e).__name__}: {_read_failure(e)})"
Expand Down
Loading
Loading