Skip to content

Cell configuration and org placement share fenced control - #776

Merged
zlareb1 merged 211 commits into
mainfrom
feat/cell-update
Nov 20, 2026
Merged

zlareb1 merged 211 commits into
mainfrom
feat/cell-update

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Related: #735, #782, #795

Changelog category

  • New Feature

Changelog entry

loom cell update reserves configuration changes before rollout and finishes or reverts them after participant turnover. loom cell place uses the same fenced directory control to place an org without changing an existing placement.

Why

A cell must reserve new issuers and orgs before accepting a configuration, then keep both reservations until old participants can no longer serve it. Configuration updates and explicit org placement share directory identity checks, immutable placement winners and synchronous durability, so they use one control protocol. Configuration updates also fence each directory change with a generation.

Scope and risk

The CLI validates a complete candidate without installing it, reserves the directory pair, repairs local tenancy and narrows local acceptance before releasing the other reservation. Config, finish and revert compare the exact observed registration, fingerprints, root secret and generation under their transaction locks. Existing placement winners remain immutable. A lost acknowledgment ends the invocation; a retry claims a new generation and recomputes rather than compensating for an uncertain write.

A selected SCIM-on configuration requires the shared fresh provisioning census before the directory claim, after local durability and after participant turnover before reservation release. Target deployment and paired-store identity are checked. Cached admission permits do not qualify. Accepted revocation cutoffs are reported immediately and remain if a later check fails; reverting to SCIM-off current performs no provisioning read.

Participant capture retains every post-cutoff obligation. Equal-version birth or state conflicts refuse before winner selection. A captured process must have an exact guarded retirement certificate, or a positive process-death proof with a complete zero-query census. A live replacement must serve the selected configuration. Expiry, missing rows, unavailable replicas and incomplete or fallback local reads cannot prove turnover.

Operator routes authenticate before parsing their bounded request body or accessing the directory. Placement checks deployment, full fingerprint, closure, quota and immutable winners under the transaction locks. Both duplicate and new placement successes emit audit WAL and commit synchronously. Unexpected exceptions from configuration-pair selection are not exposed as response text; that boundary translates only four fixed typed refusal reasons.

The shared Postgres connector binds cleanup ownership before deployment or codec initialization can wait. Failed raw-session codec initialization awaits bounded public close while the acquisition is uncancelled, then always attempts local termination. Already-cancelling acquisition starts no new close allowance. Cleanup failures preserve the original outcome and record unknown closure where required. The directory pool opens sessions for foreground acquisition only. Complete HTTP exchanges and database cleanup retain their original owner deadlines and first-outcome rules.

The main risks are interrupted cross-store writes, concurrent controllers, incomplete participant proof and stale router caches. Interruption after a cutoff leaves local acceptance single and directory reservations dual; recovery follows the documented protocol. Store nonces and generation comparisons prove the probed deployment, not agreement across all routers. Value-qualified Postgres and Keeper cleanup remain; bucket read-then-unconditional-delete is not represented as successor-safe cleanup.

Compatibility, rollout and rollback

  • Wire: adds operator-authenticated POST /cells/v1/operator/cells/{cell}/place with org, deployment id and fingerprint; GET and POST .../addresses; and POST .../configuration/placements and POST .../configuration. Update writes carry the claimed generation and exact observed fingerprint pair. Address writes carry store nonces and proposed upstream/probe URLs; configuration writes carry action, target fingerprint, root-secret fingerprint and store nonces. No existing operation or field is removed, and no setting or default is added by this concern.
  • Provisioning: use the paired cell stores and router operator credential. Provision every live identity for a SCIM-on target, or explicitly pass --revoke-unprovisioned with config, finish or revert. The existing subject leases and SCIM cutoffs apply. The flag is refused for address refresh. The target requires its own valid SCIM, OIDC and authentication configuration.
  • Rollout: deploy the participant lifecycle and controller-fencing foundation, permanently terminate or fence legacy processes, and prevent their return before online finalization. The versioned nonsecret settings file overlays the CLI's effective settings; the command does not install it. Roll out the reserved configuration before finish. For first-cell address refresh, update every router first and retain old endpoints until every router refreshes its routing table or restarts after the directory change.
  • Rollback: rerun an interrupted action or explicitly revert a pending configuration through the documented protocol. Revert preserves immutable placements, local tenancy and accepted revocation cutoffs. No migration is added by this concern; reverting code alone does not undo committed directory or tenancy changes. Existing rolling-upgrade and schema requirements remain.

Tests

  • All five configured hosted workflows succeed on this source: 52 successful checks and two expected publication skips. All six unit shards, coverage floors, format, lint, type, applicable generated-file and public-copy checks pass. The successful synthetic CI checkout is the normal composition of this head and its recorded main parent; all 69 concern files and 32 supporting caller images are byte-identical to the reviewed head in that composition.
  • Postgres 16 and 18 authority jobs each report 906 passes, one skip and one existing xpass. Each also passes the online-index durability case, physical-standby placement case, 354 authenticated isolation cases and six reconciliation cases. The native selection includes the three update/provisioning refusal and no-mutation cases. All nineteen placement durability assertions remain.
  • Cell acceptance completes all ten arms, 20,500 successful API requests, 25,000 deliveries and its 45,500-row provenance check. Independent recalculation from every retained latency sample gives on/off median p99 values of 488.888657 / 490.608826 ms: a -1.720169 ms difference within the unchanged 10 ms limit. Delivery throughput loss is 0.210%, within the unchanged 10% limit. Every arm remains included. Intended content and delivery-audit hashes reproduce the retained provenance digest; observed database rows remain runtime attestations. Retirement, trigger restoration and owned cleanup receipts complete successfully.
  • Sixteen causal codec-cleanup cases fail on the old product. Those cases and related pool/owner controls pass on the repair: 62 passes in 1.63 seconds in this worktree. Scoped lint, Python formatting, whitespace and production-module type checks pass. Both single-file architecture formatter admissions were refused before execution and remain unrun locally; hosted formatting passes. No broad or native local suite was run for this publication.
  • A fresh different independent review covers all 69 paths, with no open source findings or nits. All 1,338 prior outer assertion ASTs remain, with 1,355 now, plus the retained native and embedded-program assertions. CLI snapshots and references match their generators; applicable hosted generator checks pass.
  • Earlier worker-loss, HTTP deadline and latency failures remain recorded. The preceding Cell result's 31.326599 ms difference failed the unchanged 10 ms limit. Its exact cause remains unknown. The later passing composition does not invalidate that result, prove flakiness or establish a performance effect from the cleanup repair. No timeout, workload, statistic, assertion or threshold was weakened.

AI assistance in the code

  • AI-generated: coding agents wrote most of the diff. Codex checked conflict resolutions, retained assertions, generated references and focused regressions. A different independent agent reviewed the entire concern; a separate reviewer qualified the retained Cell evidence.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • All required format, lint, type, unit and integration CI gates pass on the publication head.
  • Applicable generated-file checks pass; CLI snapshots and references match their generators.
  • No migration is added or changed by this concern.
  • No plugin payload or frontend source changes.
  • Architecture and operations pages describe operator steps, wire changes and recovery.
  • Public-copy CI passes on the publication head.
  • Commits are signed off.

zlareb1 and others added 30 commits November 16, 2026 20:44
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup.

Give gate batch scopes a unique prefix per run so cleanup preserves existing rows.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ancy

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise primary-visible commits, durable duplicate retries and cancellation cleanup on an isolated physical standby pair in both supported Postgres CI jobs.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
An unresolved shared contact could repair admin owner grants before refusing sign-in. Check active passwordless and linked accounts before repair or signup while retaining genuine first sign-ins.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse the cell provisioning census before claiming update ownership, after local acceptance and before final reservation release. Explicit cutoffs retain the same subject leases and immediate reporting as attachment.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 changed the base branch from feat/cell-attach-cli to feat/cell-scim-person November 18, 2026 15:29
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Compare repeated identity and version pairs independently of their order so a newer replica row cannot hide conflicting older evidence.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the complete provisioning decision for authentication, consent, token ancestry and first-cell attachment while retaining verified ownership and cell scope.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove unused policy branches, describe dry-run participation and adopt the verified generated OAuth schema. Reuse the existing authorization import to retain the invalidation guard within the function limit.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blank-person refusal inside enforced provisioning and redact tenancy refusal details. Match test I/O fakes and public refusal expectations to the shared authentication path, and regenerate the schema statement corpus.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Read the diagnostic owner's four-value result and assert the prepared
first-cell identity. Preserve the marker and trigger checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning foundation while retaining update and placement control. Check the intended target before claims and reservation release, and remove an unused local tenancy observation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind cleanup ownership before deployment and codec checks can wait, so cancellation closes the session within the control attempt deadline. Placement uses the same acquisition path once.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the connector's exact participant tag when waiting for synchronous replication, so the drill reaches cancellation and standby promotion checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 changed the base branch from feat/cell-scim-person to main November 19, 2026 22:45
@zlareb1 zlareb1 closed this Nov 19, 2026
@zlareb1 zlareb1 reopened this Nov 19, 2026
…ocal

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Drain the cancellation acknowledgement before closing the seed connection so the first lookup sees the completed transaction. Keep the standby paused and retain the durability assertions.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the failed raw session owned until bounded driver closure completes, and preserve the original initialization failure through cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 marked this pull request as ready for review November 20, 2026 08:26
@zlareb1
zlareb1 merged commit f52fcbe into main Nov 20, 2026
54 checks passed
zlareb1 added a commit that referenced this pull request Nov 20, 2026
Related: #776

### Changelog category

- Improvement

### Changelog entry

Cell checks share topology, catalog and synchronization helpers across command and server code.

### Why

Complete cell reads depend on topology validation, the caller's effective privileges, replica identity and synchronization. These implementations live in CLI modules, which server code cannot import. Shared owners let both layers use the same checks without copying them.

### Scope and risk

The complete topology module moves to `server/core/cell_ch.py`. Catalog, grant, default-role, row-policy and replica checks move to `server/insights/store/cell_catalog.py`; synchronization and budget helpers move to `server/core/cell_sync.py`. Callers and test patch points use those owners, and the old topology module is deleted.

SQL, query settings and tags, defaults, refusal conditions, exception identity, row interpretation and ordinary update behavior remain unchanged. A missed caller import or patch point is the main risk. Bounded participant and observation joins remain unimplemented; this extraction adds no later-cell activation or readiness result.

The configuration and placement foundation is merged. Its raw connection ownership, retirement admission, typed configuration refusal, deadlines and cleanup remain intact when the shared imports compose with it.

### Compatibility, rollout and rollback

- Wire and defaults: unchanged; no operations, arguments, settings, tables or stored encodings are added.
- Operator steps: none.
- Rollback: reverting restores the former internal owners and callers; there is no stored data transition.
- The English architecture page names the shared topology owner. Existing translated code references await the normal translation workflow; translated files are unchanged.

### Tests

- Static bindings retain all 37 moved definitions, all 574 audited test assertions and the single unique moved topology assertion. The merged foundation's 577 assertions in the audited test modules remain in source order, with only the sixteen existing shared-owner reference substitutions. Production assertion relocations retain their conditions and multiplicity. The older reported total counted the moved topology assertion in both source images; no assertion was removed.
- The local attempt to run sixteen directly related pure selectors is refused before a test child starts because admission detects an unknown test process. No current test pass is claimed. Source and Git state match before and after the refusal. The same selectors historically pass as 34 parameterized cases on the earlier source, covering complete replica union, topology and synchronization refusal, privileges, caller identity, row policies, extra replicas, streaming failures and budget behavior. Those passes retain their original source attribution.
- Current scoped Ruff checks and formatting checks pass for all 25 changed Python files, and the concern diff passes whitespace checks. The earlier focused attempt lacking its complete source snapshot remains invalid; its passing output is not used as evidence. Historical focused passes retain their original source attribution.
- The actual merged-parent adoption resolves one import conflict while retaining both shared refusal owners and the existing retirement implementation. Static comparison with the later target finds no concern-path overlap or new production coupling; unrelated target advancement alone does not require a source refresh.
- Final-head hosted CI reports 54 successful checks and two expected documentation publication skips. Both Postgres versions complete the native, standby, authenticated-isolation and ordered-reconciliation phases. Mixed-version checks pass. Cell acceptance passes the unchanged latency and throughput requirements, with provenance and cleanup receipts. Fresh independent entire pushed-source review reports no findings or nits. These results apply to this extraction; bounded census joins remain separate work.

### AI assistance in the code

- AI-generated: Codex wrote this change. Independent final-source review and required hosted gates remain mandatory.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [x] Scoped static checks pass; the refused local test attempt is recorded as unrun.
- [x] Required final-head GitHub CI and fresh whole-source review pass.
- [x] Documentation names the shared owner; generated translations are unchanged.
- [x] No migration or stored data change is introduced.
- [x] Commits are signed off.
zlareb1 added a commit that referenced this pull request Nov 20, 2026
* Close preparation spends the shared session-end deadline

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Drive placement deadline tests with a private clock

Use one idle-driven clock for the mock exchange timers and owned close. Keep the six cases and their bounds, with a private-loop task census at teardown.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit e1f47e6)

* Drive directory cleanup checks with one private clock

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Docs: serve documentation from the docs subpath (#1042)

* Serve documentation from the docs subpath

The documentation build keeps its links, assets, and discovery files below the configured path.

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Trigger documentation preview

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Redirect documentation preview root

Preview deployments open the documentation path when their root URL is requested.

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Placement deadline tests use one simulated clock

Work deadlines, cleanup, and the elapsed-time assertion use one private clock.
The selector advances it when the loop waits, so scheduler pauses cannot expire
a simulated reply or consume its cleanup allowance. Keep all six cases and
their duration bounds, and fail teardown if the private loop retains a task.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Co-authored-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* DSPy retrievers over Loom knowledge and memory search (#1049)

* memory.ask matches bridge slugs literally (#1037)

* memory.ask matches bridge slugs literally

Escape LIKE metacharacters so slug underscores cannot select another subject.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Expect literal bridge slugs in the governed-conflict test

The namespace regression must assert the escaped pattern used by the subject filter.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Record worker context after linear-cost assertions fail (#1058)

Selected durations alone omit worker allocation and collection state. Attach numeric context after the original assertion fails so the cost calls and decision remain unchanged.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Document memory selection for coding agents and software factories (#1021)

* Document memory selection for coding agents and software factories

Give the existing use-case pages distinct search titles and concrete recall, isolation and evidence-handoff checks without claiming comparative results.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Format coding-agent and factory memory selection tables

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Document memory write and worker progress regression tests (#1043)

* Document memory write and worker progress regression tests

Exercise derived effects, stored cursors and complete SQL rows so regression checks observe the boundaries they claim to protect.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Test warm worker passes beside cursor restarts

A restart can recover a missed boundary event through cold reconciliation, so the guide must retain warm continuation regressions.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Concurrency tests cover changed content at an equal version

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Document committed-write cleanup and private cache regression tests

Cancellation during fence release must not skip effects owed by an acknowledged
write. Caches of private strings need retention and erasure coverage as well as
result checks.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Framework examples select matching SDK artifacts (#1064)

* Framework examples select matching SDK artifacts

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Framework examples install matching published SDKs

Select available stable artifacts and their native peers, keeping DSPy on its matching checkout. Give the invalid-identity fixture its own product clock so semantic closure assertions do not spend real phase time. Preserve all real deadline and cancellation cases.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Pin retriever installation to available SDK packages (#1061)

* Pin retriever installation to available SDK packages

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* LangChain examples select published SDK artifacts

Install the matching stable SDKs and the declared native peer. Validation fixtures establish a post-exec witness and completed backend termination before checking successor ownership.

Reuses the complete published witness repair from 9ffae96 (#1046).

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Cell checks share topology and catalog helpers (#906)

Related: #776

### Changelog category

- Improvement

### Changelog entry

Cell checks share topology, catalog and synchronization helpers across command and server code.

### Why

Complete cell reads depend on topology validation, the caller's effective privileges, replica identity and synchronization. These implementations live in CLI modules, which server code cannot import. Shared owners let both layers use the same checks without copying them.

### Scope and risk

The complete topology module moves to `server/core/cell_ch.py`. Catalog, grant, default-role, row-policy and replica checks move to `server/insights/store/cell_catalog.py`; synchronization and budget helpers move to `server/core/cell_sync.py`. Callers and test patch points use those owners, and the old topology module is deleted.

SQL, query settings and tags, defaults, refusal conditions, exception identity, row interpretation and ordinary update behavior remain unchanged. A missed caller import or patch point is the main risk. Bounded participant and observation joins remain unimplemented; this extraction adds no later-cell activation or readiness result.

The configuration and placement foundation is merged. Its raw connection ownership, retirement admission, typed configuration refusal, deadlines and cleanup remain intact when the shared imports compose with it.

### Compatibility, rollout and rollback

- Wire and defaults: unchanged; no operations, arguments, settings, tables or stored encodings are added.
- Operator steps: none.
- Rollback: reverting restores the former internal owners and callers; there is no stored data transition.
- The English architecture page names the shared topology owner. Existing translated code references await the normal translation workflow; translated files are unchanged.

### Tests

- Static bindings retain all 37 moved definitions, all 574 audited test assertions and the single unique moved topology assertion. The merged foundation's 577 assertions in the audited test modules remain in source order, with only the sixteen existing shared-owner reference substitutions. Production assertion relocations retain their conditions and multiplicity. The older reported total counted the moved topology assertion in both source images; no assertion was removed.
- The local attempt to run sixteen directly related pure selectors is refused before a test child starts because admission detects an unknown test process. No current test pass is claimed. Source and Git state match before and after the refusal. The same selectors historically pass as 34 parameterized cases on the earlier source, covering complete replica union, topology and synchronization refusal, privileges, caller identity, row policies, extra replicas, streaming failures and budget behavior. Those passes retain their original source attribution.
- Current scoped Ruff checks and formatting checks pass for all 25 changed Python files, and the concern diff passes whitespace checks. The earlier focused attempt lacking its complete source snapshot remains invalid; its passing output is not used as evidence. Historical focused passes retain their original source attribution.
- The actual merged-parent adoption resolves one import conflict while retaining both shared refusal owners and the existing retirement implementation. Static comparison with the later target finds no concern-path overlap or new production coupling; unrelated target advancement alone does not require a source refresh.
- Final-head hosted CI reports 54 successful checks and two expected documentation publication skips. Both Postgres versions complete the native, standby, authenticated-isolation and ordered-reconciliation phases. Mixed-version checks pass. Cell acceptance passes the unchanged latency and throughput requirements, with provenance and cleanup receipts. Fresh independent entire pushed-source review reports no findings or nits. These results apply to this extraction; bounded census joins remain separate work.

### AI assistance in the code

- AI-generated: Codex wrote this change. Independent final-source review and required hosted gates remain mandatory.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [x] Scoped static checks pass; the refused local test attempt is recorded as unrun.
- [x] Required final-head GitHub CI and fresh whole-source review pass.
- [x] Documentation names the shared owner; generated translations are unchanged.
- [x] No migration or stored data change is introduced.
- [x] Commits are signed off.

* Live MCP results protect the SDK before automatic capture (#1067)

Compose rendering and transport observation so the privacy marker reaches protection before the next automatic write.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Drive directory cleanup checks with one private clock (#1054)

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Co-authored-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Co-authored-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant