Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
152 commits
Select commit Hold shift + click to select a range
bb8e041
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
1f05935
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
55c0918
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
4a7bc88
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
4193ffd
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
c650119
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
5884b04
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
691f4f0
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
4a25c41
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
0c37c68
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
07f9ea1
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
78c2808
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
62c6b47
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
95bb74c
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
d5ec3b9
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
20681aa
Check member-role privileges and repeat attach checks after placement
zlareb1 Nov 16, 2026
cb95fff
Require effective predefined role privileges before attach
zlareb1 Nov 16, 2026
fde3798
Describe the effective owner privileges attach requires
zlareb1 Nov 16, 2026
cad839b
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
cc4ed54
Fence cell control mutations across local lock loss
zlareb1 Nov 16, 2026
e0f756b
Index cell control recovery and its upgrade refusal
zlareb1 Nov 16, 2026
3f62dfd
Validate the cell name before recovery requests
zlareb1 Nov 16, 2026
db89cb2
Keep attaching cells closed through state cleanup
zlareb1 Nov 16, 2026
4380186
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
417712d
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
c601646
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
51a8ff6
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
1f69c41
Supervise native cell control cleanup before cancelling work
zlareb1 Nov 16, 2026
5f76257
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
f4702f6
Use one owner for native cell control sessions
zlareb1 Nov 16, 2026
f9986b5
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5a62070
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
deacbaa
Close directory control sessions before cancelling their work
zlareb1 Nov 17, 2026
0c10636
Measure cell triggers with the original API and delivery workload
zlareb1 Nov 17, 2026
459dc34
Fix control batch bounds and schema coexistence checks
zlareb1 Nov 17, 2026
9caeb23
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
d71bfc9
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
92d6d9b
Use common fatal transport for directory session close
zlareb1 Nov 17, 2026
a13eb96
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
0410333
Validate serving configuration and serialize fresh delivery rows
zlareb1 Nov 17, 2026
8c03200
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
8018e30
Keep attach cleanup inside its owning deadline
zlareb1 Nov 17, 2026
6a1d018
Compose shared directory cleanup and foreground pool initialization
zlareb1 Nov 17, 2026
cdce8ec
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
70c21f2
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
9d18614
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
0835f33
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
723109d
Compose native control ownership with bounded abort cleanup
zlareb1 Nov 17, 2026
f378a28
Match persisted vector precision and preserve attach cleanup errors
zlareb1 Nov 17, 2026
9dd0934
WIP: Preserve selected driver errors through artifact diagnostics
zlareb1 Nov 17, 2026
15f842a
Preserve artifact sync failures through file close
zlareb1 Nov 17, 2026
a9580c1
The cell trigger driver retains its first failure through cleanup
zlareb1 Nov 17, 2026
2ff8931
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
e8b9d6a
WIP: Compose attach checks with native cell control
zlareb1 Nov 17, 2026
69e9f2d
Exercise attach cleanup with a fenced directory snapshot
zlareb1 Nov 17, 2026
01621af
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
5d04387
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
dfa83dd
Preserve native control request failures through HTTP cleanup
zlareb1 Nov 17, 2026
6f61b36
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
5db501c
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
6badd7d
Check the participant identity in cell heartbeat tests
zlareb1 Nov 17, 2026
06355a3
Bind trigger timing receipts to distinct API and CLI configurations
zlareb1 Nov 17, 2026
c884dd1
Preserve native errors through nonce cleanup
zlareb1 Nov 17, 2026
46aa499
Compose permanent retirement with fenced cell controls
zlareb1 Nov 17, 2026
d62fdaa
Merge certified participant retirement into cell attachment
zlareb1 Nov 17, 2026
0a94de8
Native commands preserve their first failure through cleanup
zlareb1 Nov 17, 2026
c11f8ea
Native commands retain failures through participant cleanup
zlareb1 Nov 17, 2026
a7dda54
Compose current main with native cell attachment
zlareb1 Nov 17, 2026
70b48c7
Preserve cell administration contracts and body limits
zlareb1 Nov 17, 2026
888a9a7
Merge remote-tracking branch 'origin/main' into feat/cell-each
zlareb1 Nov 17, 2026
b253487
WIP: preserve reviewed attach lifecycle corrections
zlareb1 Nov 17, 2026
d182c26
WIP: preserve attach registration and error regressions
zlareb1 Nov 17, 2026
858c90d
WIP: validate serving cell registration and retain wire error codes
zlareb1 Nov 17, 2026
9c02c28
WIP: compose cell attachment with the web listener
zlareb1 Nov 17, 2026
b313b39
WIP: preserve cell administration commands
zlareb1 Nov 17, 2026
8e96568
Compose cell administration with fenced cell control
zlareb1 Nov 17, 2026
2a379d0
Native command cleanup has a configured deadline
zlareb1 Nov 17, 2026
5f2b1fb
Provision cell automation identities and expose subject revocation
zlareb1 Nov 17, 2026
300c665
Check repeated cancellation against the original clock deadline
zlareb1 Nov 17, 2026
2d3c36b
Compose current cell control validation with cell administration
zlareb1 Nov 17, 2026
02d5688
Format bounded cell administration handlers
zlareb1 Nov 17, 2026
8444e8d
Check that native participant censuses exclude certified history
zlareb1 Nov 17, 2026
30cce29
Compose the merged Postgres batching measurement repair
zlareb1 Nov 17, 2026
2397af5
Describe the fixture owner's recorded ownership digest
zlareb1 Nov 17, 2026
9f84a2b
Exclude certified retirement history from participant reads
zlareb1 Nov 17, 2026
eeb5ac3
Validate cell storage and exercise native administration in CI
zlareb1 Nov 17, 2026
17f8d53
Compare declaration work after parser initialization
zlareb1 Nov 17, 2026
9254790
Prove worker crash recovery within the fixture process namespace
zlareb1 Nov 17, 2026
724299b
Quickstart matches live workers to their registrations
zlareb1 Nov 17, 2026
727e601
Trigger acceptance runs on isolated CI stores
zlareb1 Nov 17, 2026
1e71631
Trigger acceptance waits for TCP and reports owned startup state
zlareb1 Nov 17, 2026
79cd546
Trigger stores use the CI server configuration
zlareb1 Nov 17, 2026
c975843
Trigger ClickHouse runs as its data-directory owner
zlareb1 Nov 17, 2026
bc6bcd6
Trigger setup verifies the owned Keeper configuration
zlareb1 Nov 17, 2026
30d57c3
Worker crash checks report the unproved predicate
zlareb1 Nov 17, 2026
f32c6f1
Attach fixtures isolate authority records and preserve failure causes
zlareb1 Nov 17, 2026
7ad45ce
Attach acceptance verifies two genuine API replicas
zlareb1 Nov 18, 2026
69b6efe
Worker crash checks read captured identities before signalling
zlareb1 Nov 18, 2026
eadf2bf
Standby drill checks ledger visibility after interrupted commit
zlareb1 Nov 18, 2026
bd11af3
Trigger acceptance keeps independent write heads stable
zlareb1 Nov 18, 2026
449769e
Retirement readers filter the native participant UUID
zlareb1 Nov 18, 2026
644f653
The worker census skips exited supervisor children
zlareb1 Nov 18, 2026
88165da
Trigger setup refusals retain bounded child diagnostics
zlareb1 Nov 18, 2026
1a1cee2
Diagnostic read failures preserve the original command refusal
zlareb1 Nov 18, 2026
1fcd50a
The replica fixture matches its migrated vector width
zlareb1 Nov 18, 2026
4e49edf
Keep attach and document source checks in the Postgres workflow
zlareb1 Nov 18, 2026
6521d1b
Wait for supervised replacement enrollment after retirement
zlareb1 Nov 18, 2026
3e82257
Attach deadline tests use an owned cleanup clock
zlareb1 Nov 18, 2026
d5e1411
Resource samples require a complete process forest
zlareb1 Nov 18, 2026
3f69214
Trigger protocol metadata describes its implemented runtime
zlareb1 Nov 18, 2026
c9685ba
Phone citation geometry waits for the selected Ask option
zlareb1 Nov 18, 2026
a400ba3
Native trigger failures retain bounded driver diagnostics
zlareb1 Nov 18, 2026
24a159f
Provider smoke retains bounded reset failure evidence
zlareb1 Nov 18, 2026
09c4f8b
Merge privacy, summary and probe changes into cell attachment
zlareb1 Nov 18, 2026
2460fa2
Trigger acceptance retains bounded Postgres failure evidence
zlareb1 Nov 18, 2026
e9bd5b1
Retirement census distinguishes method calls from local cleanup
zlareb1 Nov 18, 2026
79943b7
Worker parents retire verified stopped children
zlareb1 Nov 18, 2026
ba55790
Worker child retirement follows the application boundary
zlareb1 Nov 18, 2026
23bd673
Verify captured worker subtrees in native recovery checks
zlareb1 Nov 18, 2026
74d59a4
Test child retirement limits and clarify native token minting
zlareb1 Nov 18, 2026
9d6d586
Prove exact exit of single-thread zombie participants
zlareb1 Nov 18, 2026
fa65ac4
Compose cell attachment with the session-protection heartbeat
zlareb1 Nov 18, 2026
da95f2d
Match subtree diagnostic controls to kernel observations
zlareb1 Nov 18, 2026
a1ebaad
Distinguish store guard failures and reap interrupted test children
zlareb1 Nov 18, 2026
3d86ef4
Wait for the published ClickHouse HTTP listener
zlareb1 Nov 18, 2026
00ceb04
Wait for the complete single-thread exit witness
zlareb1 Nov 18, 2026
7112196
Reap the retirement fixture child after interrupted waits
zlareb1 Nov 18, 2026
1d752a7
Collect the attach lock fixture's delayed rollback task
zlareb1 Nov 18, 2026
efcfc40
Isolate retirement gates between integration scenarios
zlareb1 Nov 18, 2026
96bd15e
Trigger acceptance retains bounded staging failure evidence
zlareb1 Nov 18, 2026
1841255
Exercise absent head locks at the native staging boundary
zlareb1 Nov 18, 2026
d9977f7
Compose cell attach with current CLI and settings surfaces
zlareb1 Nov 18, 2026
310c407
Tracker feed holds native admission through owned cleanup
zlareb1 Nov 18, 2026
81bf248
Lock memory heads through ordered exact-key probes
zlareb1 Nov 18, 2026
5278b3d
Collect native fixture resources after setup failures
zlareb1 Nov 18, 2026
32d6790
Keep attach tenancy writes on their owning backend
zlareb1 Nov 18, 2026
00fafb2
Document unproved attach lock cleanup
zlareb1 Nov 18, 2026
b0c7dd6
Align attach cleanup and pairing descriptions
zlareb1 Nov 18, 2026
32b625c
Cell administration composes with current attachment ownership
zlareb1 Nov 18, 2026
77b915c
Cell administration tests isolate each authority refusal
zlareb1 Nov 18, 2026
7b984eb
Abort body parsing before oversized actions execute
zlareb1 Nov 18, 2026
a3bb80c
Run bounded administration across registered cells
zlareb1 Nov 19, 2026
afbfd51
Blacklist removal writes a dominating selected-row tombstone
zlareb1 Nov 19, 2026
61f156a
Each state tests call their owning modules directly
zlareb1 Nov 19, 2026
6a2097c
Cell administration keeps one active context contract
zlareb1 Nov 19, 2026
43bff6e
Cell administration removes unused command and removal models
zlareb1 Nov 19, 2026
8e0ec86
Bounded rate changes test refusal and adoption contracts
zlareb1 Nov 19, 2026
1cc7316
Private cell fixtures use a valid document prefix
zlareb1 Nov 19, 2026
24b766f
Assert the classified cell identity refusal envelope
zlareb1 Nov 19, 2026
7e2e56f
Cell administration composes with provisioning checks
zlareb1 Nov 20, 2026
02baeb0
Await disconnect after directory codec failure
zlareb1 Nov 20, 2026
82fa76f
Retain cell administration with current router contracts
zlareb1 Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1005,6 +1005,7 @@ jobs:
tests/integration/test_sql_surface_live_ch.py
tests/integration/test_pg_clickhouse_live_ch.py
tests/integration/test_cell_tenancy_pg.py
tests/integration/test_cell_each_native.py
tests/integration/test_router_directory_pg.py
tests/integration/test_pg_online_index.py
tests/integration/test_pg_owned_cleanup.py
Expand Down
54 changes: 53 additions & 1 deletion docs/architecture/cells.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,12 +112,63 @@ What spans orgs becomes per cell:
and overrides, policy rules, rate-limit and budget overrides, signing-key rotation, a
subject's revocation. `loom cell each -- <command>` runs an org-free admin command in
every cell with a short-lived credential per cell, records each cell's outcome,
resumes the cells that failed, and fails until all succeed; `loom doctor` compares the
resumes confirmed no-action failures, and stops on an uncertain mutation;
`loom doctor` compares the
cells' settings and configuration fingerprints and names what differs;
- the code and tracker orgs (`LOOM_CODE_ORG`, `LOOM_TRACKER_ORG`) are each cell's own:
each cell indexes its own repositories, at the ingest cost of each;
- an operator's platform view and the dashboard are per cell, at the cell's host.

## Administration across cells

`loom cell each` is a serial HTTP coordinator with a closed command list. It does not
start a command in a cell or select a store. It supports live settings, signing keys,
an exact subject's revocation, namespace-name rate limits, operation and consumer
latency budgets, and global blacklist entries. Tenant policy and org-limit sweeps are
outside this command. See [cell administration](../operations/cell-administration.md).

The operator authenticates a bounded directory census once before the run, reads each
target again before its attempt, and compares a final whole census. Every registered
cell must be serving, enforced and supported; closing placements does not exclude a
cell. Identity includes the deployment, public origin, control generation, complete
configuration digests and separate automation and recovery public identities. A changed
target refuses continuation. Fingerprint digests are SHA-256 over the complete stored
Postgres JSONB text, rather than the shortened fingerprint printed for operators.

The coordinator reads an explicitly named private key for each cell and proves that
its public thumbprint matches the automation identity before signing an OAuth
assertion. Each exchange requests a sixty-second access token. Credentials travel
only to their exact HTTPS origin, with ordinary certificate verification and without
redirects, proxy discovery or automatic retries. The recovery key is never a fallback.

The cell applies its ordinary request lifecycle and authentication gates, then requires
a current registered automation platform-admin bot. The bot has one approved public
key and no alternate workload trust. A fresh read authenticated with that cell's own
router credential supplies the authoritative public pair; request text cannot select
an automation identity. Empty or malformed directory metadata refuses. This is a fresh
admission check, not an atomic transaction with later directory changes.

`loom cell automation` provisions the public pair after proving both explicit keys
against that cell's current bot registrations. It uses the existing native cell
control owner, registration comparison and directory generation claim. The directory
stores only the two purposes, names and public thumbprints in `break_glass`, with an
operator audit in the same transaction. Registration reruns preserve this pair.

Each action awaits a strict started audit, its owning business helper, and a strict
completed audit. Correlation fields do not provide idempotency. Before a mutating
request can leave the coordinator, its private durable state records `dispatching`.
An ambiguous response becomes `UNSETTLED` and stops later cells. Resume refuses this
uncertainty before reading credentials or opening a network connection. Completed
outcomes remain immutable; an explicit correlated refusal before business work may be
retried. There is no receipt lookup or automatic replay of an uncertain mutation.

The census is limited to 4,096 cells and four MiB, with one KiB per public target.
Individual action bodies are capped at 64 KiB before JSON parsing, including when the
global request limit is disabled. A run has a one-hour deadline and each server action
has thirty seconds. Local state is capped at 32 MiB, protected by one lifetime file
lock, and replaced with file and directory synchronization. These bounds refuse excess
work rather than silently truncating a fleet, result or security read.

## The router

The router is a stateless HTTP proxy, run as several replicas behind the deployment's
Expand Down Expand Up @@ -1491,6 +1542,7 @@ entry points and the named exceptions:
| `cell_prepare._write_rows` | Fresh claim, observed local rows, exact lock backend |
| `cell_attach_pg._mode_transaction` | Attach's claim, supervised mode try, recording/first/fingerprint comparison, exact lock backend |
| `cell_control.backout` | Fresh claim, fenced close and zero-placement read, observed local rows, exact lock backend |
| `cell_automation.provision`, `directory.set_administration` | Fresh claim, paired authority, two live bot proofs, fenced public metadata and operator audit |
| `cell_update_pg.write.change` | Fresh directory claim; owner's bounded transaction on the exact ATTACH backend, paired registration and exact observed local identity/pair before tenancy repair and acceptance write |
| `directory.attach_begin`, `attach_place`, `attach_enforced`, `attach_serving`, `remove` | Claimed generation, full registration and exact pair, including retries/no-ops |
| `directory.update_addresses` | Claimed generation, full registration and exact pair under directory/row locks; commit-time address check, audit and synchronous WAL acknowledgement in one bounded transaction |
Expand Down
7 changes: 4 additions & 3 deletions docs/architecture/system-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,10 @@ HTTP answers with a status and a `detail` body, MCP with a result that carries `
### HTTP

`build_router()` in `server/transport/http.py` adds one `POST /v1/<op>` route per op.
`BodyLimitMiddleware` first refuses a body over `LOOM_MAX_REQUEST_BYTES` (32 MiB) with 413;
0 disables the cap. Each request then passes these phases in order (`_make_endpoint`). A
refusal ends the request.
`BodyLimitMiddleware` refuses oversized bodies with 413. `LOOM_MAX_REQUEST_BYTES`
(32 MiB) sets the global cap; 0 disables only that cap. Fixed cell administration
caps remain. Each request then passes these phases in order (`_make_endpoint`).
A refusal ends the request.

| Phase | Code | Refusal |
| -------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand Down
107 changes: 107 additions & 0 deletions docs/operations/cell-administration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Administer every cell

`loom cell each` applies one supported administration command to every registered cell
and records confirmed outcomes in a private local file. Run it from the operator's
machine with access to the router and each cell's public HTTPS origin. It needs no
local Postgres or ClickHouse connection, server profile, or root signing secret.

Every cell must be serving and enforced. A cell closed to placements remains a target.
An attaching cell, unsupported server, changed deployment, missing key or incomplete
automation registration refuses the run.

## Prepare credentials

Each cell has two distinct key-proof platform-admin bots: one for automation and one
for recovery. Their names and public key thumbprints are registered in the directory.
Keep their private keys separate. The automation bot accepts exactly its approved
public key and has no alternate workload trust. This restriction lets the cell match a
short-lived access token's bot to the one approved key without inventing an assertion
key claim in the token.

Register both purposes only after each bot has its own approved key and platform-admin
grant. On the configured cell operator profile, run `loom cell automation --cell CELL
--automation-bot NAME --automation-thumbprint THUMBPRINT --automation-key PRIVATE_FILE
--recovery-bot NAME --recovery-thumbprint THUMBPRINT --recovery-key PRIVATE_FILE`.
The command holds the cell's control lock, claims its current directory registration,
and proves both keys through the cell's public authentication and bot-registration
endpoints before writing the public pair. Neither private key is sent to the router.

Create a private credential manifest with these fields:

| Field | Value |
| ------------------- | ----------------------------------------------------------------------------- |
| `version` | `1` |
| `router_token_file` | Absolute path to the private router operator-token file |
| `keys` | One entry per target, with `cell`, `deployment_id` and an absolute `key_file` |
| `ca_file` | Optional absolute path to an explicit CA certificate bundle |

The manifest, token and key files must be regular files owned by the current user,
with private permissions and private containing directories. Symlinks and unsafe
writable ancestors refuse. The command does not change permissions or use ambient
server credentials. It reads each automation key at use, derives its public thumbprint,
and refuses a mismatch before signing. It never tries a recovery key.

Use the router's exact HTTPS origin for `--router`. Redirects and ambient HTTP proxies
are disabled. The operator token goes only to the router; assertions and cell tokens
go only to the cell's exact public origin. Normal CA and hostname verification applies.
For private PKI, configure the cell server's operating-system CA trust for its router
connection. The server uses the normal TLS trust store, including `SSL_CERT_FILE`
when the operator supplies it, while ignoring ambient HTTP proxies. The manifest's
`ca_file` controls the command's router and cell connections; it does not change the
server's trust store. Neither side disables certificate or hostname verification.

## Run a supported command

Set `ROUTER`, `CREDENTIALS` and `STATE` to the trusted router origin, private manifest
path and a fresh private outcome-file path, then run:

```bash
loom cell each --router "$ROUTER" --credentials "$CREDENTIALS" --state "$STATE" -- settings list
```

The supported command families are closed:

| Family | Actions |
| -------------- | ------------------------------------------------------- |
| `settings` | `list`, `set`, `reset`, `history` for live settings |
| `signing-key` | `list`, `rotate`, `retire` |
| `revoke-token` | One exact global subject |
| `ratelimit` | Namespace-name limit, shared across orgs with that name |
| `budget` | Operation budget or consumer multiplier |
| `blacklist` | Global `add`, `list` |

Tenant policy, org-limit sweeps, arbitrary commands and native store operations are
not supported. Use `loom cell each --help` for coordinator options. A server lacking
the capability contract or requested action refuses before business work; the command
does not fall back to an older API.
Global blacklist removal is unavailable. A parsed removal command receives no
advertised capability.

The command runs one cell at a time. It reads the directory before the run, again for
each target, and once after all attempts. Exit zero requires every target to have a
confirmed completion and the final identities to match. Directory drift discovered
at the end fails the run while retaining the completed history.

## Resume confirmed failures

Use the existing state file without supplying another command:

```bash
loom cell each --router "$ROUTER" --credentials "$CREDENTIALS" --resume "$STATE"
```

Completed cells are not repeated. Confirmed pre-business refusals can be attempted
again with fresh credentials. The original command and router binding cannot change.
New cells require complete key coverage and are appended only while existing target
identities remain unchanged. Removed or replaced targets refuse continuation.

`dispatching` and `UNSETTLED` indicate that a mutation may have reached a cell without
a confirmed response. Resume stops before credential or network access. Preserve the
state and inspect that cell's audit and business state using the separate recovery
procedure; do not edit away uncertainty or automatically replay the command. Action
correlation is not idempotency, and an HTTP error alone does not prove no action ran.

Keep outcome files private. They contain public target identities, command arguments,
completion hashes and result digests, with no access token or private key. Command
arguments can still contain sensitive setting values or blacklist text. Preserve the
file and its sibling lock location for the lifetime of the run.
2 changes: 2 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,8 @@ python scripts/codegen/docs_reference.py
| `loom cell` | Operate a deployment's cells through the router's directory. | |
| `loom cell scim-token` | Generate a cell's SCIM token offline. | |
| `loom cell scim-token new` | Print a random token for the named cell. | `--cell` |
| `loom cell each` | Run a supported admin command in every cell | `--router`, `--credentials`, `--state`, `--resume`, `<command>` |
| `loom cell automation` | Prove and register distinct automation and recovery bot keys. | `--cell`, `--router`, `--automation-bot`, `--automation-thumbprint`, `--automation-key`, `--recovery-bot`, `--recovery-thumbprint`, `--recovery-key`, `--ca-file` |
| `loom cell credential` | Make a router credential, or rotate a cell's. | `<action>`, `--cell`, `--next`, `--finish`, `--revert`, `--operator`, `--router` |
| `loom cell limit` | Close, reopen or bound a cell to new orgs. | `--cell`, `--closed`, `--open`, `--placements-max`, `--router` |
| `loom cell which` | Print the lookup's cell and host for an org (the first cell if not placed). | `--org`, `--router` |
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Authentication, tokens and identity providers.
| `LOOM_RESTRICTED_SERVE` | `bool` | `True` | | Kill switch for restricted namespaces. True: served to a principal whose token names the namespace AND who holds a live direct grant. False: every read is refused while ingest continues. |
| `LOOM_RESTRICTED_AUDIT_INLINE` | `bool` | `True` | | Wait for the `namespace.restricted_access` audit row of each admitted restricted read before answering; false writes it beside the answer, lost if the process dies first. |
| `LOOM_CORS_ORIGINS` | `str` | `''` | | Comma-separated origin list. "*" allows all (acceptable when auth_required=True). Empty disables CORS entirely. |
| `LOOM_MAX_REQUEST_BYTES` | `int` | `33554432` | | Request-body cap, bytes, enforced by ASGI middleware before any parser (413 PAYLOAD_TOO_LARGE); uvicorn has no limit of its own. 0 disables. |
| `LOOM_MAX_REQUEST_BYTES` | `int` | `33554432` | | Global request-body cap, bytes, enforced by ASGI middleware before parsing (413 PAYLOAD_TOO_LARGE). 0 disables the global cap; fixed cell administration caps remain. |
| `LOOM_HSTS_MAX_AGE_SECONDS` | `int` | `31536000` | | Strict-Transport-Security max-age, seconds; browsers ignore it over plain HTTP. 0 sends no header. |
| `LOOM_POLICY_MODE` | `'off' \| 'shadow' \| 'enforce'` | `'off'` | | History policy over the agent trace: "off" reads/writes nothing; "shadow" records verdicts in loom.policy_decisions without blocking; "enforce" denies (403 POLICY_FORBIDDEN) and records them too. In enforce, writes fail closed on evaluation errors; reads never do. |
| `LOOM_POLICY_MAX_WINDOW_SECONDS` | `int` | `86400` | | Cap on any rule's look-back window, in seconds. |
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -752,6 +752,10 @@ HTTP 503, raised as `server.core.pg.PostgresUnavailableError`. 503 — Postgres,

HTTP 409, raised as `server.core.errors.PreconditionFailedError`. 409 — `if_match_sha256` mismatch, an `approved_sha256` that no longer matches the definition's text, or an erasure that landed while a `namespace.restore` ran. Category `input`, retryable: no. What to do: [common errors](#common-errors-and-what-to-do).

#### `PREFLIGHT_REFUSED`

Raised in `server/cli/cell_each_http.py`.

#### `PROBE_BUSY`

Raised in `server/router/app.py`. Category `capacity`, retryable: yes.
Expand Down Expand Up @@ -780,6 +784,10 @@ HTTP 429, raised as `server.core.errors.RateLimitedError`. 429 — the caller is

HTTP 409, raised as `server.core.errors.ReadsLlmFreeError`. 409 — `memory.ask` or `code.ask` asked for a model's answer while `LOOM_READS_LLM_FREE` is on. Category `input`, retryable: no. What to do: [common errors](#common-errors-and-what-to-do).

#### `READ_INCOMPLETE`

Raised in `server/cli/cell_each_state.py`.

#### `RECALL_REQUIRES_FILTER`

Raised in `server/memory/app/recall.py`.
Expand Down Expand Up @@ -928,6 +936,10 @@ Raised in `server/knowledge/app/writes.py`.

Raised in `server/knowledge/app/sources.py`, `server/tracker/app/sources.py`.

#### `SUBJECT_REVOCATION_INCOMPLETE`

Raised in `server/insights/app/cell_admin_security_actions.py`.

#### `SUBSTRING_NOT_FOUND`

Raised in `server/memory/app/profile.py`, `server/memory/app/working_memory.py`.
Expand Down
Loading
Loading