Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
211 commits
Select commit Hold shift + click to select a range
bc35583
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
6cf53b3
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
7382921
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
2575979
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
79c97d2
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
f2fad61
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
dcf7996
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
6229357
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
a64cbb2
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
84e6460
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
5615d41
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
b0f694b
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
fefe1e0
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
bd4cda3
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
3ba813c
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
98fc333
Operator placement durably records its directory winner and local ten…
zlareb1 Nov 16, 2026
042dd5e
Prove cell placement retries survive standby promotion
zlareb1 Nov 16, 2026
bb8e041
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
1f05935
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
55c0918
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
4a7bc88
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
4193ffd
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
c650119
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
5884b04
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
691f4f0
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
4a25c41
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
0c37c68
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
07f9ea1
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
78c2808
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
62c6b47
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
95bb74c
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
d5ec3b9
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
20681aa
Check member-role privileges and repeat attach checks after placement
zlareb1 Nov 16, 2026
cb95fff
Require effective predefined role privileges before attach
zlareb1 Nov 16, 2026
fde3798
Describe the effective owner privileges attach requires
zlareb1 Nov 16, 2026
cad839b
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
cc4ed54
Fence cell control mutations across local lock loss
zlareb1 Nov 16, 2026
e0f756b
Index cell control recovery and its upgrade refusal
zlareb1 Nov 16, 2026
3f62dfd
Validate the cell name before recovery requests
zlareb1 Nov 16, 2026
db89cb2
Keep attaching cells closed through state cleanup
zlareb1 Nov 16, 2026
4380186
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
417712d
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
c601646
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
51a8ff6
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
1f69c41
Supervise native cell control cleanup before cancelling work
zlareb1 Nov 16, 2026
5f76257
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
946ef2f
Validate prospective cell settings without changing process configura…
zlareb1 Nov 16, 2026
c9d1f0a
Check cell fingerprint reservations with shared org pattern rules
zlareb1 Nov 16, 2026
9ac5f0e
Keep target settings values out of update errors
zlareb1 Nov 16, 2026
f4702f6
Use one owner for native cell control sessions
zlareb1 Nov 16, 2026
789b25c
Refuse tagged YAML parser failures without quoting values
zlareb1 Nov 16, 2026
f9986b5
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5a62070
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5809472
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
8a76924
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
deacbaa
Close directory control sessions before cancelling their work
zlareb1 Nov 17, 2026
0c10636
Measure cell triggers with the original API and delivery workload
zlareb1 Nov 17, 2026
2e1e1ae
Validate cell update targets against their registration
zlareb1 Nov 17, 2026
459dc34
Fix control batch bounds and schema coexistence checks
zlareb1 Nov 17, 2026
9caeb23
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
d71bfc9
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
92d6d9b
Use common fatal transport for directory session close
zlareb1 Nov 17, 2026
f1e8231
Compose cell update targets with guarded participant retirement
zlareb1 Nov 17, 2026
a13eb96
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
0410333
Validate serving configuration and serialize fresh delivery rows
zlareb1 Nov 17, 2026
8c03200
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
b429beb
Compose cell control fencing with configuration updates
zlareb1 Nov 17, 2026
da5ef76
Compose native session ownership with operator placement
zlareb1 Nov 17, 2026
d27ff2e
Close placement sessions before cancelling their work
zlareb1 Nov 17, 2026
8018e30
Keep attach cleanup inside its owning deadline
zlareb1 Nov 17, 2026
40be106
Reserve cell configuration and configured orgs atomically
zlareb1 Nov 17, 2026
8335af4
Describe configuration placement concurrency
zlareb1 Nov 17, 2026
46a0f57
Compose operator placement with cell configuration updates
zlareb1 Nov 17, 2026
c66786e
Preserve native owner errors when fallback abort fails
zlareb1 Nov 17, 2026
6a1d018
Compose shared directory cleanup and foreground pool initialization
zlareb1 Nov 17, 2026
cdce8ec
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
70c21f2
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
9d18614
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
0835f33
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
723109d
Compose native control ownership with bounded abort cleanup
zlareb1 Nov 17, 2026
914d239
Compose placement with shared directory and migration cleanup
zlareb1 Nov 17, 2026
f378a28
Match persisted vector precision and preserve attach cleanup errors
zlareb1 Nov 17, 2026
8b2d1eb
WIP: coordinate cell configuration updates and address proofs
zlareb1 Nov 17, 2026
9dd0934
WIP: Preserve selected driver errors through artifact diagnostics
zlareb1 Nov 17, 2026
faf7a05
Preserve cell updates through shared cleanup failures
zlareb1 Nov 17, 2026
15f842a
Preserve artifact sync failures through file close
zlareb1 Nov 17, 2026
361bbbb
Bound placement exchanges and reserve both request bodies
zlareb1 Nov 17, 2026
a9580c1
The cell trigger driver retains its first failure through cleanup
zlareb1 Nov 17, 2026
2ff8931
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
ef7e606
Reserve native fingerprint defaults in placement bodies
zlareb1 Nov 17, 2026
d6e5eec
Scope SCIM credentials to the configured cell
zlareb1 Nov 17, 2026
c8ec1d6
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
e8b9d6a
WIP: Compose attach checks with native cell control
zlareb1 Nov 17, 2026
69e9f2d
Exercise attach cleanup with a fenced directory snapshot
zlareb1 Nov 17, 2026
01621af
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
5d04387
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
17eee33
Keep native HTTP failures through client cleanup
zlareb1 Nov 17, 2026
dfa83dd
Preserve native control request failures through HTTP cleanup
zlareb1 Nov 17, 2026
6f61b36
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
5db501c
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
6badd7d
Check the participant identity in cell heartbeat tests
zlareb1 Nov 17, 2026
ce81164
Native nonce cleanup keeps the first failure
zlareb1 Nov 17, 2026
06355a3
Bind trigger timing receipts to distinct API and CLI configurations
zlareb1 Nov 17, 2026
f485b25
Retain the control deadline during cell configuration updates
zlareb1 Nov 17, 2026
c884dd1
Preserve native errors through nonce cleanup
zlareb1 Nov 17, 2026
46aa499
Compose permanent retirement with fenced cell controls
zlareb1 Nov 17, 2026
d62fdaa
Merge certified participant retirement into cell attachment
zlareb1 Nov 17, 2026
0a94de8
Native commands preserve their first failure through cleanup
zlareb1 Nov 17, 2026
c11f8ea
Native commands retain failures through participant cleanup
zlareb1 Nov 17, 2026
a7dda54
Compose current main with native cell attachment
zlareb1 Nov 17, 2026
b253487
WIP: preserve reviewed attach lifecycle corrections
zlareb1 Nov 17, 2026
d182c26
WIP: preserve attach registration and error regressions
zlareb1 Nov 17, 2026
858c90d
WIP: validate serving cell registration and retain wire error codes
zlareb1 Nov 17, 2026
9c02c28
WIP: compose cell attachment with the web listener
zlareb1 Nov 17, 2026
2a379d0
Native command cleanup has a configured deadline
zlareb1 Nov 17, 2026
300c665
Check repeated cancellation against the original clock deadline
zlareb1 Nov 17, 2026
fd3bc09
Compose cell updates with the attach foundation
zlareb1 Nov 17, 2026
98afee1
Retain the deterministic cell cleanup deadline check
zlareb1 Nov 17, 2026
c7efa71
Format the cell cleanup deadline test
zlareb1 Nov 17, 2026
8444e8d
Check that native participant censuses exclude certified history
zlareb1 Nov 17, 2026
30cce29
Compose the merged Postgres batching measurement repair
zlareb1 Nov 17, 2026
11f7b46
Document update control writers and nonce cleanup limits
zlareb1 Nov 17, 2026
77b0638
SCIM person authorization checks active identities across requests
zlareb1 Nov 17, 2026
2397af5
Describe the fixture owner's recorded ownership digest
zlareb1 Nov 17, 2026
8e9c013
Compose SCIM authorization with cell attachment
zlareb1 Nov 17, 2026
a49d33b
Document SCIM person checks and expose generated schema on CI mismatch
zlareb1 Nov 17, 2026
55db6b1
Name the cell actor in placement fence tests
zlareb1 Nov 17, 2026
9f84a2b
Exclude certified retirement history from participant reads
zlareb1 Nov 17, 2026
17f8d53
Compare declaration work after parser initialization
zlareb1 Nov 17, 2026
295ea3c
Personal bearer subjects cannot be impersonated without a bot
zlareb1 Nov 17, 2026
9254790
Prove worker crash recovery within the fixture process namespace
zlareb1 Nov 17, 2026
74f1d6b
Workers enroll before their cell acceptance read
zlareb1 Nov 17, 2026
a7aba18
Worker service loops finish before the final drain
zlareb1 Nov 17, 2026
724299b
Quickstart matches live workers to their registrations
zlareb1 Nov 17, 2026
824870f
Caller routes check orgless bearers in the configured signup org
zlareb1 Nov 17, 2026
727e601
Trigger acceptance runs on isolated CI stores
zlareb1 Nov 17, 2026
a881cc9
SCIM fixtures clean only the identities they create
zlareb1 Nov 17, 2026
1e71631
Trigger acceptance waits for TCP and reports owned startup state
zlareb1 Nov 17, 2026
5762c94
Placement fence tests name the cell actor
zlareb1 Nov 17, 2026
79cd546
Trigger stores use the CI server configuration
zlareb1 Nov 17, 2026
c975843
Trigger ClickHouse runs as its data-directory owner
zlareb1 Nov 17, 2026
bc6bcd6
Trigger setup verifies the owned Keeper configuration
zlareb1 Nov 17, 2026
30d57c3
Worker crash checks report the unproved predicate
zlareb1 Nov 17, 2026
f32c6f1
Attach fixtures isolate authority records and preserve failure causes
zlareb1 Nov 17, 2026
7ad45ce
Attach acceptance verifies two genuine API replicas
zlareb1 Nov 18, 2026
69b6efe
Worker crash checks read captured identities before signalling
zlareb1 Nov 18, 2026
eadf2bf
Standby drill checks ledger visibility after interrupted commit
zlareb1 Nov 18, 2026
bd11af3
Trigger acceptance keeps independent write heads stable
zlareb1 Nov 18, 2026
449769e
Retirement readers filter the native participant UUID
zlareb1 Nov 18, 2026
644f653
The worker census skips exited supervisor children
zlareb1 Nov 18, 2026
88165da
Trigger setup refusals retain bounded child diagnostics
zlareb1 Nov 18, 2026
1a1cee2
Diagnostic read failures preserve the original command refusal
zlareb1 Nov 18, 2026
1fcd50a
The replica fixture matches its migrated vector width
zlareb1 Nov 18, 2026
4e49edf
Keep attach and document source checks in the Postgres workflow
zlareb1 Nov 18, 2026
6521d1b
Wait for supervised replacement enrollment after retirement
zlareb1 Nov 18, 2026
3e82257
Attach deadline tests use an owned cleanup clock
zlareb1 Nov 18, 2026
d5e1411
Resource samples require a complete process forest
zlareb1 Nov 18, 2026
3f69214
Trigger protocol metadata describes its implemented runtime
zlareb1 Nov 18, 2026
c9685ba
Phone citation geometry waits for the selected Ask option
zlareb1 Nov 18, 2026
a400ba3
Native trigger failures retain bounded driver diagnostics
zlareb1 Nov 18, 2026
24a159f
Provider smoke retains bounded reset failure evidence
zlareb1 Nov 18, 2026
09c4f8b
Merge privacy, summary and probe changes into cell attachment
zlareb1 Nov 18, 2026
2460fa2
Trigger acceptance retains bounded Postgres failure evidence
zlareb1 Nov 18, 2026
e9bd5b1
Retirement census distinguishes method calls from local cleanup
zlareb1 Nov 18, 2026
79943b7
Worker parents retire verified stopped children
zlareb1 Nov 18, 2026
ba55790
Worker child retirement follows the application boundary
zlareb1 Nov 18, 2026
23bd673
Verify captured worker subtrees in native recovery checks
zlareb1 Nov 18, 2026
74d59a4
Test child retirement limits and clarify native token minting
zlareb1 Nov 18, 2026
9d6d586
Prove exact exit of single-thread zombie participants
zlareb1 Nov 18, 2026
fa65ac4
Compose cell attachment with the session-protection heartbeat
zlareb1 Nov 18, 2026
da95f2d
Match subtree diagnostic controls to kernel observations
zlareb1 Nov 18, 2026
a1ebaad
Distinguish store guard failures and reap interrupted test children
zlareb1 Nov 18, 2026
3d86ef4
Wait for the published ClickHouse HTTP listener
zlareb1 Nov 18, 2026
00ceb04
Wait for the complete single-thread exit witness
zlareb1 Nov 18, 2026
7112196
Reap the retirement fixture child after interrupted waits
zlareb1 Nov 18, 2026
1d752a7
Collect the attach lock fixture's delayed rollback task
zlareb1 Nov 18, 2026
efcfc40
Isolate retirement gates between integration scenarios
zlareb1 Nov 18, 2026
96bd15e
Trigger acceptance retains bounded staging failure evidence
zlareb1 Nov 18, 2026
1841255
Exercise absent head locks at the native staging boundary
zlareb1 Nov 18, 2026
d9977f7
Compose cell attach with current CLI and settings surfaces
zlareb1 Nov 18, 2026
310c407
Tracker feed holds native admission through owned cleanup
zlareb1 Nov 18, 2026
81bf248
Lock memory heads through ordered exact-key probes
zlareb1 Nov 18, 2026
5278b3d
Collect native fixture resources after setup failures
zlareb1 Nov 18, 2026
32d6790
Keep attach tenancy writes on their owning backend
zlareb1 Nov 18, 2026
00fafb2
Document unproved attach lock cleanup
zlareb1 Nov 18, 2026
b0c7dd6
Align attach cleanup and pairing descriptions
zlareb1 Nov 18, 2026
1140e9b
SCIM authorization retains token and SSO evidence
zlareb1 Nov 18, 2026
7589246
SCIM fixtures attempt every owned cleanup
zlareb1 Nov 18, 2026
cea20e1
Cell updates retain native attachment ownership
zlareb1 Nov 18, 2026
5c6913e
Cell configuration bounds include operator placements
zlareb1 Nov 18, 2026
d45c18a
Cell controls retain bounded placement exchanges
zlareb1 Nov 18, 2026
f14e39e
Placement fixtures preserve initialization failures
zlareb1 Nov 18, 2026
5c989f4
Cell attach checks fresh SCIM provisioning before serving
zlareb1 Nov 18, 2026
bdfe9ff
SCIM admits provisioned account aliases and ownerless resources
zlareb1 Nov 18, 2026
ca069f1
SSO refuses ambiguous accounts before JIT repair
zlareb1 Nov 18, 2026
f110384
Configuration updates verify target SCIM provisioning
zlareb1 Nov 18, 2026
6b1f82c
Merge SCIM callback ownership correction
zlareb1 Nov 18, 2026
8dcfd39
Cell control tests bound cancellation and isolate provisioning rows
zlareb1 Nov 18, 2026
6f704fd
Participant census rejects conflicts before selecting the newest row
zlareb1 Nov 18, 2026
a32df3c
Enforced cells require active provisioning for people
zlareb1 Nov 19, 2026
021e29c
Merge main for provisioning integration checks
zlareb1 Nov 19, 2026
0518d08
Keep provisioning census scoped to active attachment
zlareb1 Nov 19, 2026
7bb1a93
Standalone authorization keeps its provisioning policy
zlareb1 Nov 19, 2026
49436bf
Native doctor checks retain the first-cell identity
zlareb1 Nov 19, 2026
9a6b2b8
Cell update checks the selected provisioning policy
zlareb1 Nov 19, 2026
a3ffc15
Cell control owns raw sessions before initialization
zlareb1 Nov 19, 2026
dc5b8af
Placement durability observes the participant session
zlareb1 Nov 19, 2026
9ae41a2
Merge provisioning prerequisite into cell update
zlareb1 Nov 19, 2026
1d71737
Keep configuration refusal reasons explicit and integration origins l…
zlareb1 Nov 19, 2026
dd5f3bb
Wait for placement seed cancellation to finish
zlareb1 Nov 20, 2026
36938b4
Await directory disconnect after codec setup failure
zlareb1 Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1010,9 +1010,16 @@ jobs:
tests/integration/test_cell_unset_gate_pg.py
tests/integration/test_cell_prepare.py
tests/integration/test_cell_attach.py
tests/integration/test_cell_place.py
tests/integration/test_cell_update_directory.py
tests/integration/test_cell_update_addresses.py
tests/integration/test_cell_update_run_pg.py
tests/integration/test_cell_update_participants_ch.py
tests/integration/test_cell_place_cleanup.py
tests/integration/test_cell_attach_two_replicas.py
tests/integration/test_cell_attach_cleanup.py
tests/integration/test_cell_scim_census_live_ch.py
tests/integration/test_cell_update_scim_live_ch.py
tests/integration/test_cell_control_cleanup.py
tests/integration/test_cell_control_fence.py
tests/integration/test_directory_control_cleanup.py
Expand Down Expand Up @@ -1062,6 +1069,34 @@ jobs:
bash dev/ci_start_postgres.sh
docker update --memory 1g --memory-swap 1g loom-scale-index-drill-pg loom-scale-index-drill-pg-standby
pytest -q tests/integration/test_pg_online_index_durability.py
- name: pytest placement durability on a private physical standby
timeout-minutes: 8
env:
LOOM_PG_IMAGE: postgres:${{ matrix.postgres }}
LOOM_PG_CONTAINER: loom-scale-place-drill-pg
LOOM_PG_HOST_PORT: "55432"
LOOM_CELL_PLACE_DRILL: "1"
LOOM_CELL_PLACE_DRILL_DSN: postgres://loom:loomdev@localhost:55432/loom
run: |
set -euo pipefail
for name in loom-scale-place-drill-pg loom-scale-place-drill-pg-standby; do
if docker container inspect "$name" >/dev/null 2>&1; then
echo "drill container already exists: $name" >&2
exit 1
fi
done
if docker network inspect loom-scale-place-drill-pg-net >/dev/null 2>&1; then
echo "drill network already exists" >&2
exit 1
fi
cleanup() {
docker rm -f -v loom-scale-place-drill-pg loom-scale-place-drill-pg-standby || true
docker network rm loom-scale-place-drill-pg-net || true
}
trap cleanup EXIT
bash dev/ci_start_postgres.sh
docker update --memory 1g --memory-swap 1g loom-scale-place-drill-pg loom-scale-place-drill-pg-standby
pytest -q tests/integration/test_cell_place_durability.py
# The tenant-isolation gates again, against a second server that enforces auth:
# the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens.
# They run whether or not the memory suites passed (`!cancelled()`).
Expand Down
155 changes: 104 additions & 51 deletions docs/architecture/cells.md

Large diffs are not rendered by default.

161 changes: 161 additions & 0 deletions docs/operations/cell-update.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Update a cell's configuration or addresses

`loom cell update` reserves a configuration before the cell accepts it. Run it with
that cell's stores, `LOOM_CELL`, and the router operator credential configured. It
holds the cell's ATTACH lock on one direct Postgres session. Prepare, attach,
restore, another update, and the documented state cleanup cannot run concurrently.
An interrupted command exits unsuccessfully; rerun the same action after correcting
the reported cause.

## Reserve a configuration

Write a nonsecret settings file:

```yaml
version: 1
settings:
tracker_org: operations
```

```bash
loom cell update --cell first --config target.yaml
```

The file overlays the CLI's effective settings. Every supplied setting is validated,
including values outside the fingerprint. Secret fields are refused. The command
neither installs the file nor changes the running processes' configuration. It
prints fingerprint digests and changed field names.

The deployment id, registered public and OAuth host, and bucket and prefix cannot
change. The root-secret fingerprint must match the directory's locked value.

A SCIM-on target requires this cell's valid SCIM token, OIDC login and required
authentication. The CLI freshly checks active accounts, live refresh sessions and
personal service-account creators against the target provisioning policy before
claiming the directory generation, after durable local acceptance, and after
participant turnover before releasing the reservation. It uses the same bounded
[provisioning census as attach](cell-attach.md); a cached admission decision does
not establish this precondition. Run the CLI with the target's SCIM credentials
and authentication settings configured.

Provision every required identity before updating. Alternatively, add
`--revoke-unprovisioned` to `--config`, `--finish` or `--revert` to end tokens
issued before the update for subjects that cannot meet its SCIM policy. Accepted
SCIM cutoffs are reported immediately and remain if a later check fails; the
command does not roll them back. Correct the reported cause and rerun the same
action. This flag does not apply to address refresh. Reverting to SCIM-off current
does not require provisioning.

After attach has begun, the directory reserves both current and next values and
places newly configured orgs before the cell's Postgres accepts next. Overlap checks
include both values of every other attached or attaching cell. A different pending
value refuses. Repeating the identical candidate repairs an interrupted local
write without placing an org twice.

New configured placements obey the cell's closed flag and quota. Existing winners
in that cell remain valid while closed or full. An org held elsewhere refuses the
whole reservation. A later cell still attaching refuses configured-org changes;
a first cell attaching after begin accepts them through the normal placement gate.

Before attach/begin has recorded the directory fingerprint, update claims the
registration's generation and changes only the local pair. It places no org and
reserves no issuer or pattern. Attach checks and reserves both local values later.

## Finish or revert

Install and roll out the reserved configuration, then run:

```bash
loom cell update --cell first --finish --wait 600
```

To keep current instead:

```bash
loom cell update --cell first --revert --wait 600
```

These forms first repair local tenancy for actual same-cell configured placements
from both directory values. Their operator read compares the exact generation,
registration, pair, and root-secret fingerprint, writes an audit and WAL fence,
and waits for synchronous commit. Public lookup's first-cell fallback is not proof
that an immutable placement exists. A missing placement outside the attaching
first cell requires directory metadata recovery.

Finalization durably narrows local acceptance first. The directory keeps both
reservations until every participant captured by a complete post-cutoff census
has permanently retired or its exact process is proved dead with no outstanding
ClickHouse query. Captured target processes also turn over. A target-serving
replacement must join a live instance, a marked cell heartbeat, and a guarded
participant outside the captured set. A workers-only process cannot satisfy this.

Every configured ClickHouse replica must answer. Replicated participant tables
synchronize after the local cutoff; the census unions their newest rows without
age filters, sampling, caches, or partial-result limits. Heartbeat expiry,
`drained`, a missing row, an empty query list alone, a request deadline, or deletion
of a pod object does not prove permanent retirement. An uninspectable process or
unreadable topology refuses finalization.

Online finalization requires the [participant lifecycle baseline](../architecture/cells.md#participant-lifecycle-baseline)
and [controller fencing](../architecture/cells.md#control-mutation-fencing). Positively
terminate or fence every legacy server, worker, one-off command and control writer,
including paused boots, before using this protocol. Prevent old binaries from
returning. Inventory actual process/container termination or independent node
fencing; an expired heartbeat or missing API object is insufficient. The CLI
cannot establish this external inventory from its census. Unknown legacy identities
remain blockers. No command flag asserts or bypasses this prerequisite.

`--wait` is finite and between 0 and 86400 seconds. Zero still makes a bounded
initial attempt. Local transactions and complete census reads have their own
bounded attempts. If turnover times out, local acceptance stays single and the
directory stays dual. Keep the target replacement serving, resolve the named
participants and rerun the same action. Each rerun takes a fresh census. Retrying
`--config` can widen local acceptance again and requires another cutoff and census
before finalization. All directory placements and local tenancy rows remain after
revert, including orgs introduced by the pending value.

| Interruption | Recorded state | Recovery |
| -------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------- |
| Directory reservation commits before local write | Directory dual; local pair may be old | Repeat config, finish, or revert explicitly |
| Local cutoff commits before turnover or final directory commit | Directory dual; local single | Repeat the selected finish or revert |
| Generation or ATTACH backend changes | An in-flight directory reply may be stale | Stop; a fresh invocation claims again and recomputes |
| Final directory singleton already matches local | Both single | The no-op still proves synchronous local durability |

## Refresh addresses

For a later cell:

```bash
loom cell update --cell second --probe-url https://second-probe.example.test
```

For the first cell, roll out the intended `LOOM_ROUTER_UPSTREAM_URL` and
`LOOM_ROUTER_UPSTREAM_PROBE_URL` to **every router first**, then run without an
action flag:

```bash
loom cell update --cell first
```

The contacted router supplies its normalized first-cell addresses. The probe must
echo fresh Postgres, ClickHouse, and configured bucket nonces and the registered
deployment id, bucket and prefix. A main listener cannot substitute for the probe.
The first cell's main upstream must independently report that deployment id. The
router rechecks its configured addresses at commit. A later refresh changes only
its probe URL; it cannot move the registered host, stores or upstream.

A generation and address comparison excludes stale requests; it does not establish
fleet-wide configuration agreement. An old router can read the current expected
row and reverse the addresses if it still runs the old configuration. Keep the
old endpoints reachable until every router demonstrably refreshes its routing
table or restarts **after** the directory change. A configuration rollout alone
does not flush cached routes. During a directory outage a router can keep its last
routing table indefinitely. The command cannot certify all-router agreement.

Configuration actions, finish, revert and probe refresh are mutually exclusive.
Probe readiness retries use `--wait`; invalid proofs refuse immediately. Nonce
cleanup uses a value-qualified Postgres delete and a Keeper compare-and-delete.
Bucket cleanup reads and compares this run's value before an unconditional delete.
If the ATTACH backend is lost, a successor can replace the object between that read
and delete and lose its nonce. Rerun the affected proof; bucket cleanup does not
guarantee preservation of a successor's nonce.
2 changes: 2 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,8 +334,10 @@ python scripts/codegen/docs_reference.py
| `loom cell limit` | Close, reopen or bound a cell to new orgs. | `--cell`, `--closed`, `--open`, `--placements-max`, `--router` |
| `loom cell which` | Print the lookup's cell and host for an org (the first cell if not placed). | `--org`, `--router` |
| `loom cell remove` | Remove the directory row of a cell attaching with no placement. | `--cell`, `--router` |
| `loom cell place` | Place an org in this cell and record its tenancy. | `--cell`, `--org`, `--router` |
| `loom cell prepare` | Register this cell in the router's directory and write its rows. | `--cell`, `--host`, `--credential-sha256`, `--probe-url`, `--wait`, `--router` |
| `loom cell attach` | Attach the prepared first cell: place its orgs, serve. | `--cell`, `--wait`, `--scan-budget`, `--dry-run`, `--ch-allow-user`, `--ch-allow-user-directory`, `--ch-allow-object`, `--ch-allow-replica`, `--pg-allow-role`, `--revoke-unprovisioned`, `--router` |
| `loom cell update` | Reserve configuration, finish or revert its rollout, or refresh verified addresses. | `--cell`, `--config`, `--finish`, `--revert`, `--probe-url`, `--revoke-unprovisioned`, `--router`, `--wait` |
| `loom tracker-probe` | Read-only first contact with the tracker source: counts and newest timestamps per upstream table per credential, breaker states, the configured tracker namespaces. | |
| `loom tracker-contract-check` | Read-only: does each tracker source's schema meet the record contract (docs/features/tracker/record-contract.md)? | `--source`, `--namespace`, `--json` |
| `loom tracker-dupcheck-fit` | Refit tracker.duplicate_check's calibrated confidence for one tracker namespace now: replays the duplicate gold at filing time, fits, stores the fit. | `--namespace`, `--org`, `--negatives`, `--dry-run`, `--out` |
Expand Down
1 change: 1 addition & 0 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads
- [Backup and recovery](docs/operations/backup-and-recovery.md): A Loom backup is a ClickHouse backup, plus the document bucket while any catalog row names an object in it, and under the Postgres authority a backup of Postgres too.
- [Attaching the first cell](docs/operations/cell-attach.md): `loom cell attach` turns a prepared first cell (`loom cell prepare`, mode `recording`, directory row `attaching`) into a `serving` one: it places every org the cell holds in the router's directory, raises the cell's mode to `enforced` and…
- [Cell control upgrades and recovery](docs/operations/cell-control.md): Cell control mutations require generation-aware clients and router handlers, and direct access to the cell's Postgres.
- [Update a cell's configuration or addresses](docs/operations/cell-update.md): `loom cell update` reserves a configuration before the cell accepts it.
- [ClickHouse Cloud and Managed Postgres](docs/operations/clickhouse-cloud.md): This guide runs Loom on Kubernetes over two managed services: ClickHouse Cloud for projections, vectors, history and analytics, and ClickHouse Managed Postgres as the memory authority.
- [Code index operations](docs/operations/code-index-operations.md): This page covers how to enable the code index workers, configure repositories and keep the index current; the code index runs only when an operator enables its workers and configures repositories.
- [Deployment](docs/operations/deployment.md): Loom is one server image plus ClickHouse (and Postgres, under `LOOM_STORAGE_AUTHORITY=postgres`), with the document extractor service as an image of its own (container images).
Expand Down
27 changes: 27 additions & 0 deletions server/cli/_ch.py
Original file line number Diff line number Diff line change
Expand Up @@ -427,3 +427,30 @@ async def _go() -> T:
return result

return asyncio.run(_go())


def run_control_connected[T](run: Callable[[CHClient], Awaitable[T]], *, org_free: OrgFree) -> T:
"""Connect an operator-only coordinator without admitting model or org work.

Such a coordinator cannot join the participant set whose retirement it waits
for. It still verifies paired stores and the explicit org-free write gate.
"""
from server.core.ch import get_client

_declared(None, org_free)

async def go() -> T:
ch = get_client()
await connect_verified(ch)
try:
await check_acceptance(org_free=org_free)
result = await run(ch)
except BaseException: # Keep the coordinator failure while closing its clients.
with suppress(BaseException):
await disconnect(ch)
raise
else:
await disconnect(ch)
return result

return asyncio.run(go())
4 changes: 4 additions & 0 deletions server/cli/cell.py
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,11 @@ def _add_cell_parser(sub: Any) -> None:
remove.set_defaults(func=_cmd_remove)

from server.cli.cell_attach import add_attach_parser
from server.cli.cell_place import add_place_parser
from server.cli.cell_prepare import add_prepare_parser
from server.cli.cell_update import add_update_parser

add_place_parser(csub)
add_prepare_parser(csub)
add_attach_parser(csub)
add_update_parser(csub)
20 changes: 15 additions & 5 deletions server/cli/cell_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,12 +143,22 @@ async def admitted() -> T | _Fatal:
task.add_done_callback(pg._retrieve_task)
return task

def _bind(self, conn: Any) -> bool:
self.conn = conn
return not self.closed

async def connect(self, *, statement_timeout_ms: int) -> None:
if self.conn is not None or self.closed:
raise _refused("a control attempt cannot open another session")
try:
await pg._connect_dedicated(statement_timeout_ms=statement_timeout_ms, bind=self._bind)
except BaseException:
with contextlib.suppress(BaseException):
await self.close()
raise

async def _open(self, what: str) -> None:
self.conn = await pg._connect_dedicated(statement_timeout_ms=60_000)
if self.closed:
# A connector returning after cancellation cannot publish a usable session.
await self.close()
raise _refused("connection arrived after the control attempt ended")
await self.connect(statement_timeout_ms=60_000)
refused = await apg.direct_refusals(self.conn, what)
refused += await apg.encoding_refusals(self.conn)
if refused:
Expand Down
Loading