Skip to content

Operator placement records tenancy after the directory winner - #795

Closed
zlareb1 wants to merge 27 commits into
feat/cell-attach-clifrom
feat/cell-place
Closed

zlareb1 wants to merge 27 commits into
feat/cell-attach-clifrom
feat/cell-place

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Related: #735

Changelog category

  • New Feature

Changelog entry

loom cell place places an org through the directory and records its local tenancy only when this cell wins. Success requires an acknowledged local commit.

Why

Operator imports and bot setup need an explicit placement command. Writing local tenancy before the directory chooses a winner could let two cells accept the same org. Directory-first placement keeps one permanent winner and lets an interrupted command finish through an idempotent rerun.

Scope and risk

The command holds a direct marked Postgres session and the cell control lock. It checks the enforced cell identity and accepted fingerprint before placement and again before the local insert. The router verifies the requested cell's deployment and fingerprint, serializes placement and quota checks, and records an audit row even when returning an existing winner. A transactional durability message also covers a no-op local insert.

The change includes bounded request envelopes, whole-exchange deadlines, native cleanup that preserves the first failure, CLI snapshots and reference text, and placement integration and standby-failover test sources. It does not change retrieval ranking, memory authority defaults or erasure semantics.

This is a draft stacked on #735. Its base has advanced beyond the shared ancestor; the preserved branch has not been composed with the final accepted dependency. That composition and a fresh whole review are required before merge.

One inherited documentation finding remains: the benchmark-results paragraph attributes the delivery-pass gate to statement timings. The replacement acceptance driver and its documentation correction must arrive together through #735. Neither end-to-end timing gate is proved by this draft.

Compatibility, rollout and rollback

  • Wire: adds POST /cells/v1/operator/cells/{cell}/place with the operator credential and exactly org, deployment_id and fingerprint. It returns the winning org, cell and deployment_id; placement refusals return 409 and directory unavailability returns 503.
  • CLI: adds loom cell place --cell <cell> --org <org> [--router <url>]. It requires the cell's Postgres configuration and an enforced local cell. An uncertain outcome requires rerunning the same command.
  • Operator steps: complete the Cell attachment verifies registration and fences activation #735 rollout first, then run placement with the intended cell configuration and router operator credential. No new setting or placement backfill is introduced.
  • Rollback: reverting removes the command and route. Existing directory placements and local tenancy remain; rollback does not delete tenant data.

Tests

Current-head source checks pass: Ruff lint and format checks on the 23 Python files in the placement delta, plus git diff --check against the shared ancestor.

Historical evidence on the preserved source includes the 975-case nonce regression changing from 651 failures and 324 controls to 975 passes, and 2,089 affected unit cases passing. Those results predate this publication and are not current-head reruns or evidence for the final dependency composition.

The three control-fence placement calls name the required cell actor; their original assertions are unchanged. This correction has scoped lint and formatting checks, with no native execution claim. No tests or services were started for publication. Full required GitHub CI, final dependency composition, current placement integration, mixed-version checks, physical Postgres 16/18 drills and the end-to-end acceptance workload remain pending. A stacked draft with no checks is not CI green.

AI assistance in the code

  • AI-generated: Claude and Codex wrote and reviewed the implementation. The source and retained evidence were inspected, the existing clean branch was preserved, and scoped source checks were repeated for this publication. Fresh independent review of the public head is pending.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Required format, lint, type, unit and integration CI checks pass after the final dependency composition.
  • Generated CLI snapshots and references are checked against the final composed head.
  • Additive migrations and rolling-upgrade behavior are validated with the dependency.
  • Docs and the inherited timing-gate finding are resolved before merge.
  • Public text contains no deployment identifiers or secrets.
  • Commits are signed off.

zlareb1 and others added 26 commits November 16, 2026 20:44
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup.

Give gate batch scopes a unique prefix per run so cleanup preserves existing rows.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ancy

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise primary-visible commits, durable duplicate retries and cancellation cleanup on an isolated physical standby pair in both supported Postgres CI jobs.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Native placement callers serialize every known fingerprint field. Reserve
their defaults when a stored fingerprint omits them, while retaining unknown
stored fields in the size bound.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Client exit preserves request failures and selected registration refusals.
A failed required exit after a successful request still stops the command.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Prepare and attach attempt nonce deletion without replacing the failure
that selected cleanup. Standalone cleanup interruptions still propagate.

Remove the unused removal snapshot argument and unreachable success-test
branches.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep queued-placement and backout assertions on the required directory.place call contract.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1

zlareb1 commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Superseded by #776, which includes the placement implementation and tests. The remaining benchmark wording correction is tracked in #1063; restore validation continues in #796. The original branch and evidence are retained.

@zlareb1 zlareb1 closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant