Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
bc35583
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
6cf53b3
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
7382921
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
2575979
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
79c97d2
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
f2fad61
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
dcf7996
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
6229357
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
a64cbb2
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
84e6460
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
5615d41
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
b0f694b
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
fefe1e0
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
bd4cda3
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
3ba813c
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
98fc333
Operator placement durably records its directory winner and local ten…
zlareb1 Nov 16, 2026
042dd5e
Prove cell placement retries survive standby promotion
zlareb1 Nov 16, 2026
da5ef76
Compose native session ownership with operator placement
zlareb1 Nov 17, 2026
d27ff2e
Close placement sessions before cancelling their work
zlareb1 Nov 17, 2026
c66786e
Preserve native owner errors when fallback abort fails
zlareb1 Nov 17, 2026
914d239
Compose placement with shared directory and migration cleanup
zlareb1 Nov 17, 2026
361bbbb
Bound placement exchanges and reserve both request bodies
zlareb1 Nov 17, 2026
ef7e606
Reserve native fingerprint defaults in placement bodies
zlareb1 Nov 17, 2026
c8ec1d6
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
17eee33
Keep native HTTP failures through client cleanup
zlareb1 Nov 17, 2026
ce81164
Native nonce cleanup keeps the first failure
zlareb1 Nov 17, 2026
5762c94
Placement fence tests name the cell actor
zlareb1 Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -901,6 +901,8 @@ jobs:
tests/integration/test_cell_unset_gate_pg.py
tests/integration/test_cell_prepare.py
tests/integration/test_cell_attach.py
tests/integration/test_cell_place.py
tests/integration/test_cell_place_cleanup.py
tests/integration/test_cell_attach_cleanup.py
tests/integration/test_cell_control_cleanup.py
tests/integration/test_cell_control_fence.py
Expand Down Expand Up @@ -934,6 +936,34 @@ jobs:
bash dev/ci_start_postgres.sh
docker update --memory 1g --memory-swap 1g loom-scale-index-drill-pg loom-scale-index-drill-pg-standby
pytest -q tests/integration/test_pg_online_index_durability.py
- name: pytest placement durability on a private physical standby
timeout-minutes: 8
env:
LOOM_PG_IMAGE: postgres:${{ matrix.postgres }}
LOOM_PG_CONTAINER: loom-scale-place-drill-pg
LOOM_PG_HOST_PORT: "55432"
LOOM_CELL_PLACE_DRILL: "1"
LOOM_CELL_PLACE_DRILL_DSN: postgres://loom:loomdev@localhost:55432/loom
run: |
set -euo pipefail
for name in loom-scale-place-drill-pg loom-scale-place-drill-pg-standby; do
if docker container inspect "$name" >/dev/null 2>&1; then
echo "drill container already exists: $name" >&2
exit 1
fi
done
if docker network inspect loom-scale-place-drill-pg-net >/dev/null 2>&1; then
echo "drill network already exists" >&2
exit 1
fi
cleanup() {
docker rm -f -v loom-scale-place-drill-pg loom-scale-place-drill-pg-standby || true
docker network rm loom-scale-place-drill-pg-net || true
}
trap cleanup EXIT
bash dev/ci_start_postgres.sh
docker update --memory 1g --memory-swap 1g loom-scale-place-drill-pg loom-scale-place-drill-pg-standby
pytest -q tests/integration/test_cell_place_durability.py
# The tenant-isolation gates again, against a second server that enforces auth:
# the Postgres read paths (heads by key and by id, narrowed scopes) see real tokens.
# They run whether or not the memory suites passed (`!cancelled()`).
Expand Down
34 changes: 25 additions & 9 deletions docs/architecture/cells.md
Original file line number Diff line number Diff line change
Expand Up @@ -754,13 +754,26 @@ read-only scope, lists any tenancy row the directory does not place.
it, unless the cell is the first and its mode reads `recording`, and an org-free command
never uses it; a leaked one is stopped by `loom cell limit --closed`, which needs none of
the cell's stores, and replaced by `loom cell credential` (below).
`loom cell place --org <org>` (declared org-free: it writes the org's placement and
tenancy row, which is how an org comes to be held), with the router's operator credential,
`loom cell place --cell <cell> --org <org>` (declared org-free: it writes the org's
placement and tenancy row, which is how an org comes to be held), with the router's operator credential,
places an org for a command that writes a new one (a bot, an import), through the router's
placement API and in its order, as attach's placement of configured orgs does: it inserts
the placement and reads back the winner, writes the cell's tenancy row only if the winner
is this cell (otherwise it refuses and writes nothing), and reports success only once the
row is written, so a rerun is idempotent. For an org the directory does not place yet it
row commits synchronously. The CLI holds the attach lock on a direct marked Postgres
session, requires a full enforced local cell state and the full accepted configuration
fingerprint, and checks both again before inserting. A transactional durability message
makes even a no-op insert flush an earlier primary-only tenancy row to the standby.
Opening the session, initial checks and the local transaction through COMMIT each have
a separate client deadline derived from `LOOM_PG_STATEMENT_TIMEOUT_MS`; the directory
request keeps its own transport timeout. Each local phase reserves cleanup inside its
original deadline. On failure the shared control owner closes the exact backend before
cancelling or reaping its work, deferring repeated signals within that same deadline.
Successful exit and failures outside a local phase have a separate cleanup allowance
derived from the same setting. An unknown outcome names rerunning the same command;
the permanent directory winner and idempotent local insert let that rerun finish. The
route checks directory state; a direct operator API caller is trusted to check local
Postgres mode as the CLI does. For an org the directory does not place yet it
passes the placement API's gate (below): refused while the directory is unavailable, for a
cell the gate does not accept, closed, or when the fingerprint of the CLI's own
configuration of the cell (with the deployment id read from the stores it connects to)
Expand Down Expand Up @@ -789,9 +802,9 @@ read-only scope, lists any tenancy row the directory does not place.
serves past a table whose lock it cannot take, logged with the holders of its locks, or
one the role cannot alter, logged as such (the third exception under Invariants). A
first cell is `recording` until attach sets `enforced` while it runs, so no boot can
have created them: attach creates them itself with the same pass
(`cell_triggers.ensure`) once it has set `enforced`, and the cell goes `serving` only
once `cell_triggers.missing` names no defect. A later cell is `enforced` from prepare
have created them: attach creates them with `cell_triggers.ensure_table`, passing
`recording_first=True`, before step 2 and again before setting `enforced`.
The cell goes `serving` only once `cell_triggers.missing` names no defect. A later cell is `enforced` from prepare
and gets them at its first migrating boot after prepare, which its attach requires the
same way. `loom doctor` names a disabled or altered one in any state, a missing one on
an `enforced` cell, and a partitioned table with an `org` column as unsupported, since
Expand Down Expand Up @@ -859,9 +872,12 @@ read-only scope, lists any tenancy row the directory does not place.
safe, since the read-back makes it idempotent. For an org the directory already
places, a cell's call is checked only for its credential (against the cells' hashes
the router caches for `LOOM_ROUTER_DIRECTORY_CACHE_S`, default 5 s) and the cell's
rate, and a `loom cell place` only for the operator credential; both answer the winner
whatever the cell's state, quota or fingerprint, adding no placement (the winner is
public through the lookup anyway); a request refused past its cell's rate answers 503
rate. An operator's `loom cell place` checks its deployment id and fingerprint against
the requested cell's locked row even for an org already placed there; an org placed
elsewhere answers that winner and the CLI refuses without inserting local tenancy.
Existing placements bypass closed and quota checks. Every successful call writes an
audit row before its synchronous commit, including a retry of an existing winner, so
the returned winner survives failover. A request refused past its cell's rate answers 503
with `Retry-After` (at most a minute), and its retry is admitted by the replica that
answers it once the cell's calls there are under the rate. An org the cell already
holds needs no call, so serving it never depends on the router.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,7 @@ python scripts/codegen/docs_reference.py
| `loom cell limit` | Close, reopen or bound a cell to new orgs. | `--cell`, `--closed`, `--open`, `--placements-max`, `--router` |
| `loom cell which` | Print the lookup's cell and host for an org (the first cell if not placed). | `--org`, `--router` |
| `loom cell remove` | Remove the directory row of a cell attaching with no placement. | `--cell`, `--router` |
| `loom cell place` | Place an org in this cell and record its tenancy. | `--cell`, `--org`, `--router` |
| `loom cell prepare` | Register this cell in the router's directory and write its rows. | `--cell`, `--host`, `--credential-sha256`, `--probe-url`, `--wait`, `--router` |
| `loom cell attach` | Attach the prepared first cell: place its orgs, serve. | `--cell`, `--wait`, `--scan-budget`, `--dry-run`, `--ch-allow-user`, `--ch-allow-user-directory`, `--ch-allow-object`, `--ch-allow-replica`, `--pg-allow-role`, `--router` |
| `loom tracker-probe` | Read-only first contact with the tracker source: counts and newest timestamps per upstream table per credential, breaker states, the configured tracker namespaces. | |
Expand Down
5 changes: 3 additions & 2 deletions server/cli/cell.py
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@ def _cmd_remove(args: argparse.Namespace) -> int:
print(f"loom cell remove: refusing: {reason}", file=sys.stderr)
return 1
try:
answer = asyncio.run(_remove_locked(args, router, row))
answer = asyncio.run(_remove_locked(args, router))
except Exception as e: # noqa: BLE001 — an unknown control outcome stops the run
print(f"loom cell remove: {type(e).__name__}: {e}; stop and rerun", file=sys.stderr)
return 1
Expand All @@ -360,7 +360,6 @@ def _cmd_remove(args: argparse.Namespace) -> int:
async def _remove_locked(
args: argparse.Namespace,
router: httpx.URL,
row: dict[str, Any],
transport: httpx.AsyncBaseTransport | None = None,
) -> dict[str, Any]:
from server.cli import cell_control
Expand Down Expand Up @@ -463,7 +462,9 @@ def _add_cell_parser(sub: Any) -> None:
remove.set_defaults(func=_cmd_remove)

from server.cli.cell_attach import add_attach_parser
from server.cli.cell_place import add_place_parser
from server.cli.cell_prepare import add_prepare_parser

add_place_parser(csub)
add_prepare_parser(csub)
add_attach_parser(csub)
30 changes: 24 additions & 6 deletions server/cli/cell_attach.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,19 @@ async def call(self, path: str, body: dict[str, Any] | None = None) -> dict[str,
async def health(self, host: str) -> str:
"""The deployment id `https://<host>/v1/health/deployment` answers ('' when none)."""
try:
async with httpx.AsyncClient(
timeout=_HEALTH_TIMEOUT, transport=self.transport
) as client:
manager = httpx.AsyncClient(timeout=_HEALTH_TIMEOUT, transport=self.transport)
client = await manager.__aenter__()
try:
r = await client.get(f"https://{host}/v1/health/deployment")
except BaseException: # Client exit preserves the request's first failure.
with contextlib.suppress(BaseException):
await manager.__aexit__(*sys.exc_info())
raise
except (httpx.HTTPError, ValueError):
return ""
# A required close failure after a successful read must reach the owner.
await manager.__aexit__(None, None, None)
try:
answer = r.json()
except (httpx.HTTPError, ValueError):
return ""
Expand Down Expand Up @@ -367,7 +376,11 @@ async def begin(self) -> None:
await _write_nonces(self.ch, nonces)
self.row = {**self.row, **await self.mutate("/attach/begin", body)}
self.begun_keys_fp = self.keys_fp
finally:
except BaseException: # Nonce cleanup preserves the first failure or cancellation.
with contextlib.suppress(BaseException):
await _delete_nonces(self.ch, nonces)
raise
else:
await _delete_nonces(self.ch, nonces)

# -- the scans ---------------------------------------------------------------------
Expand Down Expand Up @@ -533,11 +546,16 @@ async def serving(self) -> dict[str, Any]:
nonces = _new_nonces()
try:
await _write_nonces(self.ch, nonces)
return await self.mutate(
answer = await self.mutate(
"/attach/serving", {"deployment_id": self.row["deployment_id"], "nonces": nonces}
)
finally:
except BaseException: # Nonce cleanup preserves the first failure or cancellation.
with contextlib.suppress(BaseException):
await _delete_nonces(self.ch, nonces)
raise
else:
await _delete_nonces(self.ch, nonces)
return answer

# -- the run -----------------------------------------------------------------------

Expand Down
20 changes: 14 additions & 6 deletions server/cli/cell_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
from __future__ import annotations

import asyncio
import contextlib
import json
import sys
import time
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
Expand Down Expand Up @@ -304,16 +306,22 @@ async def request(

endpoint = cli.at(url, f"/cells/v1/operator/cells/{name}" + path)
try:
async with (
asyncio.timeout(REQUEST_S),
httpx.AsyncClient(
async with asyncio.timeout(REQUEST_S):
manager = httpx.AsyncClient(
timeout=cli._OPERATOR_TIMEOUT,
trust_env=url.scheme == "https",
transport=transport,
headers={"Authorization": f"Bearer {settings.router_operator_token}"},
) as client,
):
response = await client.request(method, endpoint, json=body)
)
client = await manager.__aenter__()
try:
response = await client.request(method, endpoint, json=body)
except BaseException: # Client exit preserves the request's first failure.
with contextlib.suppress(BaseException):
await manager.__aexit__(*sys.exc_info())
raise
else:
await manager.__aexit__(None, None, None)
except (httpx.HTTPError, TimeoutError) as e:
raise _refused(f"router reply unknown ({type(e).__name__}): stop and rerun") from None
answer = cli.router_answer(response, {"cell": name}, cli.OPERATOR_REFUSALS)
Expand Down
Loading