Repository navigation
Conversation
…are takes the same lock The attach command holds the attach lock on a direct, marked session of the cell's Postgres for its run and resumes by the directory row and the cell's mode. It refuses before any write on the stores' identity, the session, the roles, settings and subscriptions that would let an unchecked writer through, the database encoding, durability, its own grants, the ClickHouse writers, the table list and the processes' heartbeats; posts attach/begin; creates the triggers under the migration lock; in recording scans and writes the tenancy rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable, waits out every transaction that could have read recording, rescans, places every org in batches and marks the cell serving. A refusal after the mode commit leaves the cell enforced and attaching, and a rerun resumes there. Prepare refuses a session behind a pooler or a database that is not UTF8, and takes the attach lock. Doctor runs attach's Postgres reads again and reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows inside the widened window, and PRUNE_AFTER_S names the sweep's hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end docs/operations/cell-attach.md is the operator's procedure: the preconditions attach checks and those it relies on, the flags, the point of no return at the mode commit, the orgs attach does not place and the manual removal of an org past the length limit, and doctor's reads after serving. cells.md describes the first attach as it runs (the lock, the refusals, the trigger pass, the mode commit and its durable read, step 4's wait, rescan and final checks), with its stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py measures memory.set and a delivery-sized insert through an attached cell with the org triggers on and off; results.md records a local run. Attach compares the final root-secret fingerprint with the one its begin wrote, and the nonce cleanup's message names no single command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module instances.read_recent takes the window it reads back; attach's heartbeat module computes it from the live window and a stopping process's drain and retirement budgets. PRUNE_AFTER_S is marked as the contract it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown The refusals name every table the trigger pass would alter whose owner role attach's role is not a member of, before any write; ensure_table and _run_ddl answer applied, busy or denied, so a denial during the pass refuses at once rather than after --wait as a busy lock. --wait is wall time: each bounded wait charges its elapsed time, lock tries included, and every lock try waits at most what is left; scan time stays outside it. From the mode commit on, a stop that comes before the durable read says the commit's outcome is unknown and to rerun attach, for a refusal and for any other error alike. The prune-window check runs with the refusals, before any write; an enforced resume runs one trigger pass; the lock session opens inside the block that closes it; the durable read retries a timeout or an outage only; refusals name no command that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure --stray-orgs labels only a spent budget as tables left unchecked; any other refusal names its table and the read continues. A failed catalog read reports both of attach's Postgres checks as not read. The runbook and doctor describe removing a stray or over-long org by hand, and name no command that does not exist yet. cells.md labels the first attach's list with the design's step names, so step 1 to step 4 mean the same everywhere, rewrites the stuck-process failure row and the planned tests for a mode that only rises, and says --wait is wall time. The trigger-cost harness uses cell_triggers.create_statement and deletes the rows its inserts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as The mode is taken from the first state read, set to unknown only once the router accepted attach/enforced, and set from what the durable read saw; an interrupt prints the mode line and goes on. The trigger DDL's lock bound is taken after the migration lock, from what --wait has left then. An ended mode transaction whose lock does not clear refuses, naming its backend. A busy table's holders come back with ensure_table's answer, and attach says them once per change rather than a boot's warning a try. Attach counts the role it runs as among its own; doctor does not. The dead SCIM recheck and the unused scim_ok parameter are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish cells.md and the runbook state that attach exempts the role it runs as and doctor does not. results.md gives the marked batch's per-round changes and their mean, says the delivery-pass gate is met on the mean but not established by three rounds alone, and that the memory.set p99 gate is unresolved locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS before --wait is judged spent, so --wait 0 is one try of each wait with no retry, and an idle cell attaches with it. The run's mode turns unknown just before a try sends its UPDATE and back to recording when that try rolls back. A trigger try that finds the migration lock busy names that lock's holder, in the retry line and in the refusal. The step-4 watch task is awaited after its cancel. The ATTACH_LOCK_KEY comment says who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them. Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup. Give gate batch scopes a unique prefix per run so cleanup preserves existing rows. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ancy Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise primary-visible commits, durable duplicate retries and cancellation cleanup on an isolated physical standby pair in both supported Postgres CI jobs. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…are takes the same lock The attach command holds the attach lock on a direct, marked session of the cell's Postgres for its run and resumes by the directory row and the cell's mode. It refuses before any write on the stores' identity, the session, the roles, settings and subscriptions that would let an unchecked writer through, the database encoding, durability, its own grants, the ClickHouse writers, the table list and the processes' heartbeats; posts attach/begin; creates the triggers under the migration lock; in recording scans and writes the tenancy rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable, waits out every transaction that could have read recording, rescans, places every org in batches and marks the cell serving. A refusal after the mode commit leaves the cell enforced and attaching, and a rerun resumes there. Prepare refuses a session behind a pooler or a database that is not UTF8, and takes the attach lock. Doctor runs attach's Postgres reads again and reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows inside the widened window, and PRUNE_AFTER_S names the sweep's hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end docs/operations/cell-attach.md is the operator's procedure: the preconditions attach checks and those it relies on, the flags, the point of no return at the mode commit, the orgs attach does not place and the manual removal of an org past the length limit, and doctor's reads after serving. cells.md describes the first attach as it runs (the lock, the refusals, the trigger pass, the mode commit and its durable read, step 4's wait, rescan and final checks), with its stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py measures memory.set and a delivery-sized insert through an attached cell with the org triggers on and off; results.md records a local run. Attach compares the final root-secret fingerprint with the one its begin wrote, and the nonce cleanup's message names no single command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module instances.read_recent takes the window it reads back; attach's heartbeat module computes it from the live window and a stopping process's drain and retirement budgets. PRUNE_AFTER_S is marked as the contract it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown The refusals name every table the trigger pass would alter whose owner role attach's role is not a member of, before any write; ensure_table and _run_ddl answer applied, busy or denied, so a denial during the pass refuses at once rather than after --wait as a busy lock. --wait is wall time: each bounded wait charges its elapsed time, lock tries included, and every lock try waits at most what is left; scan time stays outside it. From the mode commit on, a stop that comes before the durable read says the commit's outcome is unknown and to rerun attach, for a refusal and for any other error alike. The prune-window check runs with the refusals, before any write; an enforced resume runs one trigger pass; the lock session opens inside the block that closes it; the durable read retries a timeout or an outage only; refusals name no command that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure --stray-orgs labels only a spent budget as tables left unchecked; any other refusal names its table and the read continues. A failed catalog read reports both of attach's Postgres checks as not read. The runbook and doctor describe removing a stray or over-long org by hand, and name no command that does not exist yet. cells.md labels the first attach's list with the design's step names, so step 1 to step 4 mean the same everywhere, rewrites the stuck-process failure row and the planned tests for a mode that only rises, and says --wait is wall time. The trigger-cost harness uses cell_triggers.create_statement and deletes the rows its inserts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as The mode is taken from the first state read, set to unknown only once the router accepted attach/enforced, and set from what the durable read saw; an interrupt prints the mode line and goes on. The trigger DDL's lock bound is taken after the migration lock, from what --wait has left then. An ended mode transaction whose lock does not clear refuses, naming its backend. A busy table's holders come back with ensure_table's answer, and attach says them once per change rather than a boot's warning a try. Attach counts the role it runs as among its own; doctor does not. The dead SCIM recheck and the unused scim_ok parameter are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish cells.md and the runbook state that attach exempts the role it runs as and doctor does not. results.md gives the marked batch's per-round changes and their mean, says the delivery-pass gate is met on the mean but not established by three rounds alone, and that the memory.set p99 gate is unresolved locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS before --wait is judged spent, so --wait 0 is one try of each wait with no retry, and an idle cell attaches with it. The run's mode turns unknown just before a try sends its UPDATE and back to recording when that try rolls back. A trigger try that finds the migration lock busy names that lock's holder, in the retry line and in the refusal. The step-4 watch task is awaited after its cancel. The ATTACH_LOCK_KEY comment says who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them. Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the production query and statement budget while reproducing the shared-identity workload in an isolated temporary table. Exact-row, ordering and identity controls distinguish a timeout from an incorrect match. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Enroll before accepting org work, retain the permanent lifecycle through record cleanup, and preserve the first failure through both store closes. Classify the external feed table builder by its exact SQL sink. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A large absent batch can merge on a partial identity and exhaust the existing statement budget. Sort complete identities before correlated row locks so the batch cannot flatten into that join. Retain all six tenant predicates, the transaction and lock order, and the unchanged native regression and deadlines. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Register each owned database and session as it is acquired. Attempt every cleanup when setup or teardown fails and preserve the original setup or body error. Keep all native assertions and deadlines unchanged. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A scan can outlive its ATTACH backend and recreate tenancy through the pool after fenced cleanup. Use the existing owner transaction for each unchanged batch and stop after an unknown commit. Keep synthetic request state scoped in its test, account for the tracker owner, and attempt every native fixture cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A stalled backend can outlive bounded client cleanup. Require exact-holder verification when release cannot be proved. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Backend termination can remain unproved after cleanup, and later-cell removal verifies paired stores. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Personal callers await verification once; guarded identity rows retain signed domain claims and personal-token kind. The verified-email migration follows the existing journal. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A failed global-row or Keeper cleanup must still allow remaining exact-resource cleanup and disconnect. Propagate the first teardown failure after all attempts. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Compose the reviewed attachment foundation while retaining every update and placement integration selection and the physical-standby durability drill. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reserve the largest live or operator request with the same body limit. Keep parser diagnostics observable and attempt independent fixture cleanup after setup or teardown failures. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Carry the existing placement deadline, first-failure cleanup and regression coverage into shared control. Reserve materialized default fields without replacing the recorded fingerprint or dropping unknown fields. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Drain independently registered resources before propagating setup failures; keep teardown errors fatal after successful use. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
An unresolved shared contact could repair admin owner grants before refusing sign-in. Check active passwordless and linked accounts before repair or signup while retaining genuine first sign-ins. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse the cell provisioning census before claiming update ownership, after local acceptance and before final reservation release. Explicit cutoffs retain the same subject leases and immediate reporting as attachment. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Compare repeated identity and version pairs independently of their order so a newer replica row cannot hide conflicting older evidence. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Move complete topology, catalog and synchronization implementations below CLI owners so server readers can reuse the same checks. Preserve SQL, refusal behavior and assertion contracts. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain every captured process obligation within one count, metadata and deadline allowance. Share the ordinary participant selection and strict fingerprint codec without changing update policy. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
2 of 4 tasks
zlareb1
force-pushed
the
feat/cell-shared-census-infrastructure
branch
from
November 20, 2026 16:16
bf38c57 to
99e372a
Compare
Share complete process evidence across reads while retaining identity, metadata and deadline bounds. Native component tests use private tables and the actual bounded transport; both existing Postgres jobs select them. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1
changed the base branch from
feat/cell-shared-census-infrastructure
to
main
November 20, 2026 19:01
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Apply known defaults before checking accepted reservations. Keep the shared transaction allowance at its owning imports and preserve exact census validation while correcting inferred types. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The integration fixture provisions the lease table used by fenced memory clearing. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #906
Related: #775
Related: #900
Changelog category (leave one):
Why
A complete cell process census must retain obligations across polls. Separately bounded snapshots can otherwise omit processes during churn. This component shares the existing participant selection and topology checks, then applies explicit count, byte and deadline bounds to one attempt.
Scope and risk
The internal reader joins permanent participants, stored cell observations and web instances across every configured replica. It synchronizes replicated participant tables and uses sequential reads for shared tables. Each read shares an 8 MiB raw allowance. The attempt retains at most 1024 participant identities, 1024 observation identities and 8 MiB of protocol metadata, including retired history, without evicting obligations. Configured replicas are capped at 64.
Fingerprint, birth, role, protocol, acceptance, key and server-clock checks return unresolved obligations. Workers join without web instance rows. This component exposes no diagnostic route or activation operation. Native commands still need positive retirement; it grants no participant exemption.
Participant selection, the ordinary complete reader and the strict fingerprint codec move to shared owners. Their replaced definitions are removed. Ordinary update keeps its existing query and allowance. The bounded byte decoder is the exact existing implementation from the linked administration change. Observation production remains a separate dependency. Native component tests use private KeeperMap roots, the actual bounded raw reader and server clocks. They cover server and worker observations, field refusal, retirement and cumulative identity retention; they do not activate attach or prove physical isolation.
Compatibility, rollout and rollback
Wire changes: none. Settings, schema and operator steps: none. Existing ordinary update behavior is preserved. Reverting leaves no added durable state. This builds on the shared census infrastructure. Complete native and mixed-build validation remains required before integration.
Tests
On the original source, the fingerprint regression first produced two intended failures and one passing metadata-bound control. The same three cases and the directly related decoder, census, participant-conflict, ordinary-update and fingerprint-reservation tests then passed: 135 tests, no failures or skips. These historical results do not qualify the current parent adoption.
The original scoped Ruff, formatting and statement-limit results retain their source attribution. Independent original-source review found one fingerprint-limit defect; its correction has no remaining findings. The parent adoption preserves all 81 original ordered assertions and all 40 test function names. It retains typed configuration refusals, acknowledged directory initialization cleanup, control-session retirement and the current Postgres state shape alongside the shared owners. Current scoped Ruff lint and formatting checks pass for all 19 changed Python files. Seven selected fake-transport regressions pass, including both large-field cases, oversized metadata refusal, complete union, certified retirement, original deadline retention and exact byte boundaries. The first focused attempt is refused before tests start because capacity is occupied; an unused lint directive in the added native fixture is corrected before the final passing check. On prior
c861bf14, entire-source review finds an omitted-field comparison defect and the server mypy gate fails with ten errors across six files. Cell acceptance also fails on that prior head; its exact cause remains unverified, and these corrections claim no Cell or performance repair. The correction compares every known observed fingerprint field at its default when omitted, while retaining unknown supplied fields and the shared matching contract. It also uses direct transaction-allowance imports and preserves the same validation through typed locals. The fingerprint regression first fails six nondefault-reservation cases with one passing default/unknown-field control, then all seven pass. Fourteen selected cases pass on the corrected source in one second, and two exact boundary inventory cases pass after a causal one-failure/one-pass result; the inventory keeps its original owner, work and deadline assertion with the new constant owner. Scoped Ruff lint and formatting checks pass for all eight correction files. Independent finite review finds no remaining correction-scope issue. A fresh entire review and all required hosted gates on the actual pushed correction remain pending. The five added native cases are selected by both existing Postgres jobs and remain unrun locally. Native ClickHouse evaluation, process bootstrap and mixed builds remain required; new component contract tests are not a pre-feature failure baseline.The preceding published head failed the two replicated-erasure integration cases because their isolated database lacked
_migration_lock; fenced memory writes could not read the lease holder. This head includes the landed fixture repair. All twenty-three concern images outside the workflow remain byte-identical; the workflow combines both existing changes through the normal merge. The failed cohort remains recorded. Fresh independent entire pushed-source review and all required hosted gates are pending for this composition; earlier results do not qualify it.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated: Codex wrote the change. Independent agent review checked the complete component, shared-owner preservation, bounds and failure paths; focused tests exercised the actual corrected source.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist