Skip to content

Stopped cells restore metadata from the router directory - #796

Draft
zlareb1 wants to merge 256 commits into
mainfrom
feat/cell-restore
Draft

zlareb1 wants to merge 256 commits into
mainfrom
feat/cell-restore

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Related: #735
Related: #776

Changelog category

  • New Feature

Changelog entry

loom cell restore repairs a stopped cell's tenancy, mode and configuration metadata from the surviving router directory before reconciliation and startup.

Why

A restored Postgres backup can lose permanent tenancy rows or retain an older acceptance fingerprint. Reconciliation can then refuse an org whose placement still exists in the router directory. Restore copies the directory metadata under the cell's retained operator lock and a fresh directory generation claim. It creates no placement and deletes no content.

Bounded ordered pages and acknowledged commits recover recorded ownership without deriving it from content or collecting every placement in memory. Attach requires live probes and cannot replace stopped-cell recovery. A successful repair reports metadata completion separately from prepare, configuration recovery and readiness.

Scope and risk

The restore path checks paired-store identity, retains one raw Postgres session, applies pending cell schema, repairs tenancy and acceptance metadata, verifies triggers and rechecks the directory generation. The router adds an operator-authenticated, audited page read. Authentication precedes bounded body parsing and directory access. Page bytes bound memory; concurrent placement growth can exhaust the command deadline.

Public preflight verifies the ClickHouse authority, binds the Postgres backend and runs the shared first-cell write policy on that session before probes or repair. The policy retains its ten-second read-work cap. Restore has one total deadline including public close. Native fatal exceptions remain captured through cleanup; ordinary policy refusal remains an operator error. The session owner reports uncertain closure. Nonce cleanup preserves the first probe error when later cleanup raises a native fatal exception; a successful probe still reports its first fatal cleanup error. Ordinary best-effort cleanup remains unchanged.

The cell must remain stopped throughout store recovery, metadata repair and reconciliation. Activity checks refuse observed live processes but cannot prove every old process ended. Restore does not recover lost erasure evidence, restore revoked credentials, rebuild a lost directory, erase stray orgs or create a separate deployment from a copy. Missing source evidence must be recovered separately before serving.

The merged update and placement foundation supplies participant attribution, generation fencing, provisioning checks, synchronous durability and connection cleanup. Restore adds its retained owner and snapshot protocol. It keeps existing local-only tenancy rows and immutable placements; repair cannot compensate for an uncertain commit by removing them.

Restore-only native test records identify setup, call and teardown entry, return and failure before a later job cancellation. Records contain a node hash, fixed phase and outcome codes, monotonic sequence and time, and a bounded repository filename and line. They omit exception messages, SQL, credentials and row contents. A finite record cap reports overflow. Recording changes no original operation, assertion, timeout, randomization or cleanup.

Compatibility, rollout and rollback

  • Wire: adds loom cell restore --cell NAME [--router URL] [--wait SECONDS] and operator-authenticated POST /cells/v1/operator/cells/{cell}/restore-read. No public Operation, SDK field, setting or default changes.
  • Prerequisites: the paired stores and surviving router directory must retain the deployment identity. Restore refuses non-UTF8 databases before control claims or metadata writes. Recreate the database with UTF8 encoding and restore into it before retrying.
  • Operator steps: apply the prerequisite router and ClickHouse schema, stop every cell process and writer, restore the store, run metadata restore with the cell configuration and router operator credential, then reconcile before startup. The command applies pending cell Postgres schema only after identity and claim checks. An interrupted or uncertain commit requires repeating the same command before continuing. A successful metadata repair is not a readiness certificate.
  • Rolling upgrades: existing additive schema and participant upgrade requirements remain. Both Postgres versions, synchronous standby recovery and mixed-build CI must pass on the final source before merge.
  • Rollback: reverting removes the command and router endpoint. Completed metadata repairs and applied additive schema remain. Reverting code does not undo permanent placements or restore older acceptance values.

Tests

  • The nonce regression exercises ordinary errors, cancellation and both native fatal outcomes through the actual first-outcome capture path. The same-input baseline records twenty-four intended identity failures and eight passing neighbors; all thirty-two cases pass after the correction. The pool-adoption fixture adds two controlled cases. Scoped Ruff lint and formatting checks pass for the corrected files. Independent finite reviews report no remaining issue in these corrections; actual pushed-head entire review and hosted gates remain pending.

  • On prior 8b7124bc, both native Postgres jobs fail before Restore.run: the restored ledger count is one where zero is required. Each job reports 997 passes, one failure, one skip and one expected-failure case that passes. The first parameter fails and its sibling passes. All three private standby drills remain unentered; later authenticated isolation and ordered reconciliation pass but do not qualify the failed jobs. Source inspection and a controlled warm-pool case establish a fixture routing gap: pooled work can retain an earlier database while dedicated snapshot work follows the fixture DSN. The exact hosted cause remains unknown because backend identities are absent. The fixture closes its prior application pool before adopting the private database and checks both pooled and dedicated database identities before yielding. A controlled warm-pool case first fails while its cold neighbor passes; both cases then pass. An intermediate cold event-order assertion fails after the added identity probe changes connection order and is corrected to require both explicit connection and pool-creation events. These are controlled fake-driver results, not real SQL or proof of the historical hosted cause. All twenty original assertions remain, with two new native identity checks. Fresh final-source review and hosted validation remain required. This does not repair or qualify Cell acceptance.

  • The final source scopes the late-connection test's three hooks inside a nested monkeypatch context. They restore before the outer database finalizer reconnects to drop its owned database. On parent 94740aa9, both native jobs record a passing test call followed by teardown entry without a return before cancellation. Source inspection identifies the hook lifetime defect; no native stack captured the deepest blocked await. The fix preserves the complete original test after removing only the new scope and receiver rename, including its 0.4-second budget, 0.1-second tolerance and cleanup.

  • The stopped-cell reconciliation fixture creates its instances table with ENGINE = Memory AS SELECT ... LIMIT 0. Both native jobs on f42b392c fail the first live-process check: CREATE TABLE ... AS copied the fixed KeeperMap path, so the table alias still shared heartbeat keys. The new table has the real columns and starts empty, with the same owned create/query/drop lifecycle. Production live-process and worker-status guards are unchanged. Actual failed-run census rows and SHOW CREATE were not captured; worker-status, schema recovery and replay were not reached and remain unvalidated.

  • The placement identity unit test uses clocks scoped to its control and Postgres modules. Its three variants retain every original assertion and the 0.03-second owner allowance and 0.5-second test bound. Three controlled expired-close cases fail the original close oracle as expected; six unchanged-source acknowledgement/expiry controls pass. The candidate passes six selected cases, including the three identity variants and the original commit-stall, cancellation and stalled-entry controls. This is not a same-input expired RED-to-GREEN result or a reproduction of the hosted schedule. Hosted shard 2 on f42b392c has one identity-variant failure: closure is observed and termination also occurs; the actual triggering delay is unknown. The production cleanup contract and termination behavior are unchanged.

  • The real merge with the adopted main parent resolves one HTTP deadline fixture conflict by retaining main's deterministic-clock repair. Its separately audited mapping retains thirteen assertions exactly, rewrites one clock receiver and adds one task-leak assertion. The existing deadline remains unchanged. This source differs from f42b392c; prior hosted results do not qualify it.

  • All 535 assertions in the retained original audit remain exact in source order: 534 across eighteen test files and one production assertion. The placement unit test's 51 assertions and the HTTP deadline mapping have separate preservation checks. The seven witness controls are unchanged. The late-connection layout correction preserves its AST. Scoped Ruff check and format checks pass for the changed test files. The standard focused admission previously refused before a child started remains recorded; the later six-case run is a separate admitted check. Its supervisor was reaped and is absent, but its birth start-time identity was not saved. No heavy local suite is run.

  • The earlier f42b392c head finishes with 48 successful checks, five failed checks and three skips. Formatting, unit shard 2, both native Postgres versions and Cell acceptance fail; coverage is skipped after an upstream failure. Native jobs each report 996 passes, two failures, one skip and one expected-failure test that passed; later passing phases do not override those failed jobs. The Cell primary records two provenance refusals at the original monitor boundary and an INVALID verdict. Its retained artifact contains all ten completed arms, 20,500 HTTP 200 requests, 25,000 delivery rows and fifty worker passes. The provisional median arm-p99 values are 536.048925 ms on and 509.040754 ms off; their 27.008171 ms difference exceeds the unchanged 10 ms limit, while throughput loss remains within 10%. The child provenance receipt records completed checks of 45,500 rows, matching hashes and final censuses; the outer controller still refuses acceptance. The exact acceptance delay and latency causes remain unknown. Cleanup receipts report trigger restoration and owned resource removal. Thirteen referenced runtime inputs are absent from the sanitized artifact, so no complete offline replay is claimed. This follow-up source requires fresh entire pushed-head review and all hosted gates. These fixture changes claim no Cell or performance repair.

  • Parent 94740aa9 finishes with 51 successful checks, two expected publication skips, one failed Cell check and two cancelled native jobs. Both native jobs stop in the selected teardown; later standby, isolation and reconciliation stages are not reached. The failed Cell run completes all ten arms, 20,500 HTTP 200 requests, 25,000 delivery rows and fifty worker passes, then reaches the unchanged 7,020-second global deadline in provenance. Its raw timings reproduce a provisional median arm-p99 difference of 10.88105 ms, above the unchanged 10 ms limit; throughput loss of about 1.0761% stays within 10%. No provenance success summary is retained. The slow proof subphase and latency cause remain unknown; cleanup receipts report trigger restoration and owned resource removal. These fixture changes claim no Cell or performance repair. No failed, cancelled, pending or unrun check qualifies the final source.

  • Parent hosted unit shard 2 explicitly passes all seven unchanged witness controls, with 7,095 total passes and seven warnings. Its retained checkout and three witness-related images match that source. The newline framing repair preserves the strict record parser and ten-second control bound. Earlier 0b79e10f has four live-output failures and 7,091 passes; raw child buffers are absent, so exact historical timeout attribution remains unproved. Its three formatting failures are separately corrected with an exact AST-preserving layout change. Parent Ruff/mypy, all six unit shards, coverage and mixed-build checks report success; those results are historical evidence, not final-source passes.

  • Earlier ad51e083 has a timely HTTP refusal failure at its unchanged 0.1-second owner allowance and two cancelled native jobs. Its raw trace does not establish the stalled HTTP operation; the complete fixture matches its published foundation. The native failure markers and incomplete later stages remain recorded. No deadline, assertion, production guard or cancellation contract changes here.

  • Cell acceptance on ad51e083 fails the unchanged latency requirement despite all ten arms finishing: 20,500 HTTP 200 requests, 25,000 delivery rows and fifty worker passes. Raw statistics reproduce median arm-p99 values of 571.533264 ms on and 554.471917 ms off; the 17.061347 ms difference exceeds 10 ms. Throughput loss remains within 10%. The retained provenance receipt reports 45,500 verified rows, but private proof inputs are absent from the sanitized archive, so no independent database replay is claimed. Driver cleanup returns and triggers restore. The latency cause remains unknown; no threshold waiver, flake label or blind retry is claimed.

  • Historical focused regressions cover ownership before initialization, late acquisition, public close, native fatal outcomes, repeated cancellation and first-cell write refusal. A recorded causal run fails before correction and passes afterward; a later finite selection has 36 passes. These results belong to their recorded inputs.

  • Required final-source native coverage includes backend and lock release before command return, complete metadata readback, no unauthorized ClickHouse write, snapshot and erasure recovery, synchronous-standby interruption, retry and promotion. Both Postgres versions, the physical-standby drill, authenticated isolation and reconciliation remain required. Hosted format, lint, type, unit, coverage, generator, schema, integration, mixed-build and Cell checks must pass before merge. Prior failures and refused admissions remain preserved.

Final hosted qualification for the published correction failed: 53 checks succeeded, two documentation publication checks skipped, and Cell acceptance failed. Its fresh entire-source review has no findings, and both Postgres native jobs pass their main, three standby, authenticated and ordered-reconciliation stages. These results do not override the failed Cell gate. All ten measurement arms completed, but independent provenance verification refused at the global work-deadline check; no final provenance certificate was produced and the outer result is INVALID. The retained provisional latency delta is +43.176875 ms against the unchanged +10 ms limit; provisional throughput loss is 3.015776 percent against the unchanged 10 percent limit. These provisional measurements do not establish an accepted verdict. The underlying elapsed-work cause remains unknown. Cleanup receipts record completion; no independent remote absence audit is claimed. No failed job was rerun and no acceptance limit changed.

AI assistance in the code

  • AI-generated: Codex wrote most of this change. Independent final-source review and required hosted gates remain mandatory.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Required format, lint, type, unit, coverage and integration CI gates pass on the final head.
  • Applicable generated-file and CLI snapshot checks pass on the final head.
  • No migration is added or changed by this concern.
  • No plugin payload or frontend source changes.
  • Architecture and operations pages describe operator steps, wire changes and recovery.
  • Public-copy CI passes on the final head.
  • Commits are signed off.

zlareb1 and others added 30 commits November 16, 2026 20:44
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup.

Give gate batch scopes a unique prefix per run so cleanup preserves existing rows.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ancy

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise primary-visible commits, durable duplicate retries and cancellation cleanup on an isolated physical standby pair in both supported Postgres CI jobs.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as

The mode is taken from the first state read, set to unknown only once the
router accepted attach/enforced, and set from what the durable read saw; an
interrupt prints the mode line and goes on. The trigger DDL's lock bound is
taken after the migration lock, from what --wait has left then. An ended mode
transaction whose lock does not clear refuses, naming its backend. A busy
table's holders come back with ensure_table's answer, and attach says them once
per change rather than a boot's warning a try. Attach counts the role it runs as
among its own; doctor does not. The dead SCIM recheck and the unused scim_ok
parameter are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish

cells.md and the runbook state that attach exempts the role it runs as and
doctor does not. results.md gives the marked batch's per-round changes and
their mean, says the delivery-pass gate is met on the mean but not established
by three rounds alone, and that the memory.set p99 gate is unresolved locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named

Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS
before --wait is judged spent, so --wait 0 is one try of each wait with no
retry, and an idle cell attaches with it. The run's mode turns unknown just
before a try sends its UPDATE and back to recording when that try rolls back.
A trigger try that finds the migration lock busy names that lock's holder, in
the retry line and in the refusal. The step-4 watch task is awaited after its
cancel. The ATTACH_LOCK_KEY comment says who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them.

Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup.

Give gate batch scopes a unique prefix per run so cleanup preserves existing rows.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Count privileges a login can reach through SET ROLE or an ADMIN regrant, including noninherited group grants. Keep the owner exemptions and explicit role allow list.

Repeat the writer, trigger, Postgres catalog and standby checks after the rescan and placement before marking the cell serving. Retain the heartbeat watch and its final wait floor.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check USAGE of pg_read_all_stats and pg_signal_backend because attach uses their privileges without SET ROLE. Keep MEMBER for the separate untrusted-writer reachability checks.

Refuse noninherited capabilities before attach writes, and explain the inherited grant requirement in the refusal labels and runbook. Test both inherited neighbors and signaling a scratch backend.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove unused policy branches, describe dry-run participation and adopt the verified generated OAuth schema. Reuse the existing authorization import to retain the invalidation guard within the function limit.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blank-person refusal inside enforced provisioning and redact tenancy refusal details. Match test I/O fakes and public refusal expectations to the shared authentication path, and regenerate the schema statement corpus.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Read the diagnostic owner's four-value result and assert the prepared
first-cell identity. Preserve the marker and trigger checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning foundation while retaining update and placement control. Check the intended target before claims and reservation release, and remove an unused local tenancy observation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning and configuration control dependencies while preserving the stopped-cell recovery protocol and its regression cases.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind restore and control owners before deployment and codec initialization can wait. Preserve public close on late connections and initialization failures without changing ordinary dedicated-session cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Run the shared first-cell write policy on the bound backend under the original command deadline so cancellation closes that session before cancelling its read. Keep paired-store checks and native backend-release coverage.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind cleanup ownership before deployment and codec checks can wait, so cancellation closes the session within the control attempt deadline. Placement uses the same acquisition path once.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the connector's exact participant tag when waiting for synchronous replication, so the drill reaches cancellation and standby promotion checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep stopped-cell metadata repair and its bounded owner protocol on the shipped directory cleanup and configuration refusal paths.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 changed the base branch from feat/cell-update to main November 20, 2026 08:31
Keep the in-process transport on the integration target contract so the explicit-target gate accepts the private standby drill.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the published placement deadline and marked-rollup fixture corrections
with their assertions and workload intact. Print each native integration node
so a cancelled run retains failing and stalled test identities.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain bounded phase and failure locations outside pytest capture so unfinished native recovery checks expose their last phase without changing their operations or limits.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 8:01pm UTC

Request Review

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain the private event loop and owner clock fixes from the merged deadline fixture. Keep restore cleanup arguments in the required formatter layout without changing the test behavior.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The identity recheck expects acknowledged cleanup without testing elapsed time. Give that test its own clock while keeping the shared fixture and deadline tests unchanged.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A table alias cloned the fixed KeeperMap path and still exposed server heartbeats. Use an empty Memory table for the stopped-cell fixture while retaining the production liveness checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Attempt nonce cleanup without replacing an existing failure or cancellation. A cleanup fatal remains visible after a successful probe. Cover both stores with first-outcome identity checks and success-path controls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retire the prior test pool before switching the private database and verify both connection paths use that database. Preserve the restore and erasure assertions.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant