Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
211 commits
Select commit Hold shift + click to select a range
bc35583
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
6cf53b3
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
7382921
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
2575979
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
79c97d2
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
f2fad61
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
dcf7996
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
6229357
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
a64cbb2
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
84e6460
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
5615d41
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
b0f694b
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
fefe1e0
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
bd4cda3
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
3ba813c
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
98fc333
Operator placement durably records its directory winner and local ten…
zlareb1 Nov 16, 2026
042dd5e
Prove cell placement retries survive standby promotion
zlareb1 Nov 16, 2026
bb8e041
loom cell attach turns a prepared first cell into a serving one; prep…
zlareb1 Nov 16, 2026
1f05935
Attach's runbook, the cells.md first-attach steps and the trigger-cos…
zlareb1 Nov 16, 2026
55c0918
The widened heartbeat window lives with attach, so server.core import…
zlareb1 Nov 16, 2026
4a7bc88
ensure_table's integration test lands with attach, its caller
zlareb1 Nov 16, 2026
4193ffd
Fix a double space in the failure table's pooler row
zlareb1 Nov 16, 2026
c650119
Attach refuses a table its role cannot alter up front, charges --wait…
zlareb1 Nov 16, 2026
5884b04
Doctor's stray-org read names a table it cannot read and goes on, and…
zlareb1 Nov 16, 2026
691f4f0
Attach's closing line names the mode it last read, every DDL try ends…
zlareb1 Nov 16, 2026
4a25c41
The attach docs say which role attach exempts, and the trigger gate's…
zlareb1 Nov 16, 2026
0c37c68
Every attach try runs once whatever --wait has left, the mode reads u…
zlareb1 Nov 16, 2026
07f9ea1
The attach docs say what --wait 0 does, and cells.md keeps each code …
zlareb1 Nov 16, 2026
78c2808
Bound cell attach's mode commit and name booting processes
zlareb1 Nov 16, 2026
62c6b47
Require every cell probe grant before attach
zlareb1 Nov 16, 2026
95bb74c
Keep cell attach deadlines from waiting on rollback
zlareb1 Nov 16, 2026
d5ec3b9
Bound attach trigger attempts and scope gate cleanup to its run
zlareb1 Nov 16, 2026
20681aa
Check member-role privileges and repeat attach checks after placement
zlareb1 Nov 16, 2026
cb95fff
Require effective predefined role privileges before attach
zlareb1 Nov 16, 2026
fde3798
Describe the effective owner privileges attach requires
zlareb1 Nov 16, 2026
cad839b
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
cc4ed54
Fence cell control mutations across local lock loss
zlareb1 Nov 16, 2026
e0f756b
Index cell control recovery and its upgrade refusal
zlareb1 Nov 16, 2026
3f62dfd
Validate the cell name before recovery requests
zlareb1 Nov 16, 2026
db89cb2
Keep attaching cells closed through state cleanup
zlareb1 Nov 16, 2026
4380186
Build placement indexes outside migration transactions
zlareb1 Nov 16, 2026
417712d
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
c601646
Close owned Postgres sessions before cancelling their work
zlareb1 Nov 16, 2026
51a8ff6
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
1f69c41
Supervise native cell control cleanup before cancelling work
zlareb1 Nov 16, 2026
5f76257
Close attach sessions before cancelling transaction work
zlareb1 Nov 16, 2026
946ef2f
Validate prospective cell settings without changing process configura…
zlareb1 Nov 16, 2026
c9d1f0a
Check cell fingerprint reservations with shared org pattern rules
zlareb1 Nov 16, 2026
9ac5f0e
Keep target settings values out of update errors
zlareb1 Nov 16, 2026
f4702f6
Use one owner for native cell control sessions
zlareb1 Nov 16, 2026
789b25c
Refuse tagged YAML parser failures without quoting values
zlareb1 Nov 16, 2026
f9986b5
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5a62070
Propagate unexpected trigger errors before rollback cleanup
zlareb1 Nov 16, 2026
5809472
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
8a76924
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
deacbaa
Close directory control sessions before cancelling their work
zlareb1 Nov 17, 2026
0c10636
Measure cell triggers with the original API and delivery workload
zlareb1 Nov 17, 2026
2e1e1ae
Validate cell update targets against their registration
zlareb1 Nov 17, 2026
459dc34
Fix control batch bounds and schema coexistence checks
zlareb1 Nov 17, 2026
9caeb23
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
d71bfc9
Preserve native migration failures through owned cleanup
zlareb1 Nov 17, 2026
92d6d9b
Use common fatal transport for directory session close
zlareb1 Nov 17, 2026
f1e8231
Compose cell update targets with guarded participant retirement
zlareb1 Nov 17, 2026
a13eb96
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
0410333
Validate serving configuration and serialize fresh delivery rows
zlareb1 Nov 17, 2026
8c03200
Construct migration cleanup work inside its error capture
zlareb1 Nov 17, 2026
b429beb
Compose cell control fencing with configuration updates
zlareb1 Nov 17, 2026
da5ef76
Compose native session ownership with operator placement
zlareb1 Nov 17, 2026
d27ff2e
Close placement sessions before cancelling their work
zlareb1 Nov 17, 2026
8018e30
Keep attach cleanup inside its owning deadline
zlareb1 Nov 17, 2026
40be106
Reserve cell configuration and configured orgs atomically
zlareb1 Nov 17, 2026
8335af4
Describe configuration placement concurrency
zlareb1 Nov 17, 2026
46a0f57
Compose operator placement with cell configuration updates
zlareb1 Nov 17, 2026
c66786e
Preserve native owner errors when fallback abort fails
zlareb1 Nov 17, 2026
6a1d018
Compose shared directory cleanup and foreground pool initialization
zlareb1 Nov 17, 2026
cdce8ec
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
70c21f2
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
9d18614
Keep owned cancellation alive through repeated signals
zlareb1 Nov 17, 2026
0835f33
Migration fallback aborts preserve the first failure
zlareb1 Nov 17, 2026
723109d
Compose native control ownership with bounded abort cleanup
zlareb1 Nov 17, 2026
914d239
Compose placement with shared directory and migration cleanup
zlareb1 Nov 17, 2026
f378a28
Match persisted vector precision and preserve attach cleanup errors
zlareb1 Nov 17, 2026
8b2d1eb
WIP: coordinate cell configuration updates and address proofs
zlareb1 Nov 17, 2026
9dd0934
WIP: Preserve selected driver errors through artifact diagnostics
zlareb1 Nov 17, 2026
faf7a05
Preserve cell updates through shared cleanup failures
zlareb1 Nov 17, 2026
15f842a
Preserve artifact sync failures through file close
zlareb1 Nov 17, 2026
361bbbb
Bound placement exchanges and reserve both request bodies
zlareb1 Nov 17, 2026
a9580c1
The cell trigger driver retains its first failure through cleanup
zlareb1 Nov 17, 2026
2ff8931
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
ef7e606
Reserve native fingerprint defaults in placement bodies
zlareb1 Nov 17, 2026
d6e5eec
Scope SCIM credentials to the configured cell
zlareb1 Nov 17, 2026
c8ec1d6
Describe trigger creation before the attach mode commit
zlareb1 Nov 17, 2026
e8b9d6a
WIP: Compose attach checks with native cell control
zlareb1 Nov 17, 2026
69e9f2d
Exercise attach cleanup with a fenced directory snapshot
zlareb1 Nov 17, 2026
01621af
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
5d04387
Check trigger gate provenance in its repository checkout
zlareb1 Nov 17, 2026
17eee33
Keep native HTTP failures through client cleanup
zlareb1 Nov 17, 2026
dfa83dd
Preserve native control request failures through HTTP cleanup
zlareb1 Nov 17, 2026
6f61b36
Retire participants only after permanent quiescence
zlareb1 Nov 16, 2026
5db501c
Hold native principal work through command retirement
zlareb1 Nov 16, 2026
6badd7d
Check the participant identity in cell heartbeat tests
zlareb1 Nov 17, 2026
ce81164
Native nonce cleanup keeps the first failure
zlareb1 Nov 17, 2026
06355a3
Bind trigger timing receipts to distinct API and CLI configurations
zlareb1 Nov 17, 2026
f485b25
Retain the control deadline during cell configuration updates
zlareb1 Nov 17, 2026
c884dd1
Preserve native errors through nonce cleanup
zlareb1 Nov 17, 2026
46aa499
Compose permanent retirement with fenced cell controls
zlareb1 Nov 17, 2026
d62fdaa
Merge certified participant retirement into cell attachment
zlareb1 Nov 17, 2026
0a94de8
Native commands preserve their first failure through cleanup
zlareb1 Nov 17, 2026
c11f8ea
Native commands retain failures through participant cleanup
zlareb1 Nov 17, 2026
a7dda54
Compose current main with native cell attachment
zlareb1 Nov 17, 2026
b253487
WIP: preserve reviewed attach lifecycle corrections
zlareb1 Nov 17, 2026
d182c26
WIP: preserve attach registration and error regressions
zlareb1 Nov 17, 2026
858c90d
WIP: validate serving cell registration and retain wire error codes
zlareb1 Nov 17, 2026
9c02c28
WIP: compose cell attachment with the web listener
zlareb1 Nov 17, 2026
2a379d0
Native command cleanup has a configured deadline
zlareb1 Nov 17, 2026
300c665
Check repeated cancellation against the original clock deadline
zlareb1 Nov 17, 2026
fd3bc09
Compose cell updates with the attach foundation
zlareb1 Nov 17, 2026
98afee1
Retain the deterministic cell cleanup deadline check
zlareb1 Nov 17, 2026
c7efa71
Format the cell cleanup deadline test
zlareb1 Nov 17, 2026
8444e8d
Check that native participant censuses exclude certified history
zlareb1 Nov 17, 2026
30cce29
Compose the merged Postgres batching measurement repair
zlareb1 Nov 17, 2026
11f7b46
Document update control writers and nonce cleanup limits
zlareb1 Nov 17, 2026
77b0638
SCIM person authorization checks active identities across requests
zlareb1 Nov 17, 2026
2397af5
Describe the fixture owner's recorded ownership digest
zlareb1 Nov 17, 2026
8e9c013
Compose SCIM authorization with cell attachment
zlareb1 Nov 17, 2026
a49d33b
Document SCIM person checks and expose generated schema on CI mismatch
zlareb1 Nov 17, 2026
55db6b1
Name the cell actor in placement fence tests
zlareb1 Nov 17, 2026
9f84a2b
Exclude certified retirement history from participant reads
zlareb1 Nov 17, 2026
17f8d53
Compare declaration work after parser initialization
zlareb1 Nov 17, 2026
295ea3c
Personal bearer subjects cannot be impersonated without a bot
zlareb1 Nov 17, 2026
9254790
Prove worker crash recovery within the fixture process namespace
zlareb1 Nov 17, 2026
74f1d6b
Workers enroll before their cell acceptance read
zlareb1 Nov 17, 2026
a7aba18
Worker service loops finish before the final drain
zlareb1 Nov 17, 2026
724299b
Quickstart matches live workers to their registrations
zlareb1 Nov 17, 2026
824870f
Caller routes check orgless bearers in the configured signup org
zlareb1 Nov 17, 2026
727e601
Trigger acceptance runs on isolated CI stores
zlareb1 Nov 17, 2026
a881cc9
SCIM fixtures clean only the identities they create
zlareb1 Nov 17, 2026
1e71631
Trigger acceptance waits for TCP and reports owned startup state
zlareb1 Nov 17, 2026
5762c94
Placement fence tests name the cell actor
zlareb1 Nov 17, 2026
79cd546
Trigger stores use the CI server configuration
zlareb1 Nov 17, 2026
c975843
Trigger ClickHouse runs as its data-directory owner
zlareb1 Nov 17, 2026
bc6bcd6
Trigger setup verifies the owned Keeper configuration
zlareb1 Nov 17, 2026
30d57c3
Worker crash checks report the unproved predicate
zlareb1 Nov 17, 2026
f32c6f1
Attach fixtures isolate authority records and preserve failure causes
zlareb1 Nov 17, 2026
7ad45ce
Attach acceptance verifies two genuine API replicas
zlareb1 Nov 18, 2026
69b6efe
Worker crash checks read captured identities before signalling
zlareb1 Nov 18, 2026
eadf2bf
Standby drill checks ledger visibility after interrupted commit
zlareb1 Nov 18, 2026
bd11af3
Trigger acceptance keeps independent write heads stable
zlareb1 Nov 18, 2026
449769e
Retirement readers filter the native participant UUID
zlareb1 Nov 18, 2026
644f653
The worker census skips exited supervisor children
zlareb1 Nov 18, 2026
88165da
Trigger setup refusals retain bounded child diagnostics
zlareb1 Nov 18, 2026
1a1cee2
Diagnostic read failures preserve the original command refusal
zlareb1 Nov 18, 2026
1fcd50a
The replica fixture matches its migrated vector width
zlareb1 Nov 18, 2026
4e49edf
Keep attach and document source checks in the Postgres workflow
zlareb1 Nov 18, 2026
6521d1b
Wait for supervised replacement enrollment after retirement
zlareb1 Nov 18, 2026
3e82257
Attach deadline tests use an owned cleanup clock
zlareb1 Nov 18, 2026
d5e1411
Resource samples require a complete process forest
zlareb1 Nov 18, 2026
3f69214
Trigger protocol metadata describes its implemented runtime
zlareb1 Nov 18, 2026
c9685ba
Phone citation geometry waits for the selected Ask option
zlareb1 Nov 18, 2026
a400ba3
Native trigger failures retain bounded driver diagnostics
zlareb1 Nov 18, 2026
24a159f
Provider smoke retains bounded reset failure evidence
zlareb1 Nov 18, 2026
09c4f8b
Merge privacy, summary and probe changes into cell attachment
zlareb1 Nov 18, 2026
2460fa2
Trigger acceptance retains bounded Postgres failure evidence
zlareb1 Nov 18, 2026
e9bd5b1
Retirement census distinguishes method calls from local cleanup
zlareb1 Nov 18, 2026
79943b7
Worker parents retire verified stopped children
zlareb1 Nov 18, 2026
ba55790
Worker child retirement follows the application boundary
zlareb1 Nov 18, 2026
23bd673
Verify captured worker subtrees in native recovery checks
zlareb1 Nov 18, 2026
74d59a4
Test child retirement limits and clarify native token minting
zlareb1 Nov 18, 2026
9d6d586
Prove exact exit of single-thread zombie participants
zlareb1 Nov 18, 2026
fa65ac4
Compose cell attachment with the session-protection heartbeat
zlareb1 Nov 18, 2026
da95f2d
Match subtree diagnostic controls to kernel observations
zlareb1 Nov 18, 2026
a1ebaad
Distinguish store guard failures and reap interrupted test children
zlareb1 Nov 18, 2026
3d86ef4
Wait for the published ClickHouse HTTP listener
zlareb1 Nov 18, 2026
00ceb04
Wait for the complete single-thread exit witness
zlareb1 Nov 18, 2026
7112196
Reap the retirement fixture child after interrupted waits
zlareb1 Nov 18, 2026
1d752a7
Collect the attach lock fixture's delayed rollback task
zlareb1 Nov 18, 2026
efcfc40
Isolate retirement gates between integration scenarios
zlareb1 Nov 18, 2026
96bd15e
Trigger acceptance retains bounded staging failure evidence
zlareb1 Nov 18, 2026
1841255
Exercise absent head locks at the native staging boundary
zlareb1 Nov 18, 2026
d9977f7
Compose cell attach with current CLI and settings surfaces
zlareb1 Nov 18, 2026
310c407
Tracker feed holds native admission through owned cleanup
zlareb1 Nov 18, 2026
81bf248
Lock memory heads through ordered exact-key probes
zlareb1 Nov 18, 2026
5278b3d
Collect native fixture resources after setup failures
zlareb1 Nov 18, 2026
32d6790
Keep attach tenancy writes on their owning backend
zlareb1 Nov 18, 2026
00fafb2
Document unproved attach lock cleanup
zlareb1 Nov 18, 2026
b0c7dd6
Align attach cleanup and pairing descriptions
zlareb1 Nov 18, 2026
1140e9b
SCIM authorization retains token and SSO evidence
zlareb1 Nov 18, 2026
7589246
SCIM fixtures attempt every owned cleanup
zlareb1 Nov 18, 2026
cea20e1
Cell updates retain native attachment ownership
zlareb1 Nov 18, 2026
5c6913e
Cell configuration bounds include operator placements
zlareb1 Nov 18, 2026
d45c18a
Cell controls retain bounded placement exchanges
zlareb1 Nov 18, 2026
f14e39e
Placement fixtures preserve initialization failures
zlareb1 Nov 18, 2026
5c989f4
Cell attach checks fresh SCIM provisioning before serving
zlareb1 Nov 18, 2026
bdfe9ff
SCIM admits provisioned account aliases and ownerless resources
zlareb1 Nov 18, 2026
ca069f1
SSO refuses ambiguous accounts before JIT repair
zlareb1 Nov 18, 2026
f110384
Configuration updates verify target SCIM provisioning
zlareb1 Nov 18, 2026
6b1f82c
Merge SCIM callback ownership correction
zlareb1 Nov 18, 2026
8dcfd39
Cell control tests bound cancellation and isolate provisioning rows
zlareb1 Nov 18, 2026
6f704fd
Participant census rejects conflicts before selecting the newest row
zlareb1 Nov 18, 2026
bf38c57
Cell checks share topology and catalog helpers
zlareb1 Nov 18, 2026
609f5f7
Bound complete cell process census across polls
zlareb1 Nov 18, 2026
ec91117
Test RSS census exit and unreadability races
zlareb1 Nov 20, 2026
57ca597
Verify process disappearance after failed RSS reads
zlareb1 Nov 20, 2026
f0612d3
Keep placement test setup outside the HTTP deadline
zlareb1 Nov 20, 2026
63c5113
Merge placement test preparation into RSS census validation
zlareb1 Nov 20, 2026
c861bf1
Bound complete cell process census across polls
zlareb1 Nov 20, 2026
3f6d764
Compare complete known observation fingerprints
zlareb1 Nov 20, 2026
34c188a
Retain the landed placement fixture in RSS validation
zlareb1 Nov 20, 2026
e44a139
Use the landed replicated erasure fixture
zlareb1 Nov 20, 2026
1a9405c
Use the landed replicated erasure fixture
zlareb1 Nov 20, 2026
235b8fd
Verify disappearing processes during RSS census
zlareb1 Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1016,6 +1016,7 @@ jobs:
tests/integration/test_cell_update_addresses.py
tests/integration/test_cell_update_run_pg.py
tests/integration/test_cell_update_participants_ch.py
tests/integration/test_cell_observation_census.py
tests/integration/test_cell_place_cleanup.py
tests/integration/test_cell_attach_two_replicas.py
tests/integration/test_cell_attach_cleanup.py
Expand Down
24 changes: 24 additions & 0 deletions docs/architecture/cells.md
Original file line number Diff line number Diff line change
Expand Up @@ -1654,6 +1654,30 @@ before any cell's processes do, since a prepared first cell asks the state route
first cell-state read until it reads `serving`, and only a router of that release answers it
from its cache ([upgrades](../operations/upgrades.md#upgrade-procedure)).

## Bounded process census

The shared process reader has an explicit bounded mode for complete process evidence.
It reads permanent participants, every stored cell heartbeat and every stored web
instance across the configured replicas, then reads participants again. Replicated
participant tables synchronize before selection; shared tables use sequential reads.
Catalog, membership, privilege or conflicting-version uncertainty refuses the read.
The ordinary update reader keeps its existing query and allowance.

Each complete read shares an 8 MiB raw allowance across its streams. A caller keeps one
attempt across polls and physical rounds: at most 1024 participant identities, 1024
observation identities and 8 MiB retained metadata, including certified retirements.
Repeated evidence replaces bounded latest state without evicting captured obligations.
The reader binds the original monotonic deadline and accepts at most 64 configured
replicas. Oversized, truncated or malformed results refuse rather than yield a subset.

Supported process observations bind the actual participant, certified birth, cell,
accepted fingerprints and root-key fingerprint. Workers join without web instance rows;
traffic-serving first-cell servers also need their router marker. Missing, stale,
future or nonaccepting observations remain unresolved obligations. Native commands
without observations require positive retirement; this component grants no owner
exemption. Its internal result carries unresolved identities and selected evidence.
It performs no attach, exposes no diagnostic route and certifies no physical isolation.

## Invariants

- An org is served by exactly one cell, the one its placement names, and its placement
Expand Down
8 changes: 4 additions & 4 deletions scripts/gates/testsupport/cell_trigger/fixture.py
Original file line number Diff line number Diff line change
Expand Up @@ -826,11 +826,11 @@ def complete_forest() -> bool:
"resource process birth changed",
)
observed[pid] = before
need(
self.owned.rss(pid) * 1024 <= POLICY["stop_process_rss_bytes"],
"process RSS",
)
try:
need(
self.owned.rss(pid) * 1024 <= POLICY["stop_process_rss_bytes"],
"process RSS",
)
children = self.owned.children(pid)
except (FileNotFoundError, ProcessLookupError):
need(
Expand Down
6 changes: 2 additions & 4 deletions scripts/gates/testsupport/cell_trigger/owned_children.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,8 @@ def children(pid: int) -> list[int]:

@staticmethod
def rss(pid: int) -> int:
try:
status = Path("/proc", str(pid), "status").read_text()
except FileNotFoundError:
return 0 # Vanished sample, never proof that the process used no memory.
# The forest owner must verify disappearance after a failed status read.
status = Path("/proc", str(pid), "status").read_text()
return next(
(int(line.split()[1]) for line in status.splitlines() if line.startswith("VmRSS:")), 0
)
Expand Down
4 changes: 1 addition & 3 deletions server/cli/cell_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,11 @@

from server.cli import cell_attach_pg as apg
from server.core import pg, retirement
from server.core.cell_ch import Refusal
from server.core.cell_ch import TRANSACTION_S, Refusal
from server.core.cell_ch import Refused as Refused
from server.router.directory import CONTROL_FIELDS, CONTROL_MAX_S
from server.settings import settings

# A control transaction has few metadata statements; COMMIT's standby wait is bounded too.
TRANSACTION_S = 60.0 # not a setting: a single control transaction's deadline
OPEN_S = 75.0 # not a setting: connection initialization and direct-session checks
REQUEST_S = CONTROL_MAX_S + 30.0 # not a setting: directory bound plus probes/connect/close
Call = Callable[[str, dict[str, Any]], Awaitable[dict[str, Any]]]
Expand Down
3 changes: 2 additions & 1 deletion server/cli/cell_prepare.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
from server.cli import cell_control
from server.cli._ch import writes
from server.core import cell
from server.core.cell_ch import TRANSACTION_S
from server.settings import settings

# How long a run retries a busy or unreachable probe listener by default.
Expand Down Expand Up @@ -548,7 +549,7 @@ async def call(path: str, body: dict[str, Any]) -> dict[str, Any]:
current, nxt = _own_rows(rows, snapshot, every)
written = await owner.call(
lambda: _write_rows(conn, rows, answer, current, nxt),
total_s=cell_control.TRANSACTION_S,
total_s=TRANSACTION_S,
)
print(
json.dumps(
Expand Down
113 changes: 6 additions & 107 deletions server/cli/cell_update_participants.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,18 @@
from uuid import UUID

from server.cli import cell_control
from server.core import cell, cell_sync, ch_names, instances
from server.core.cell_ch import WHOLE_SQL, Refused, per_host, quote, read_topology
from server.core import cell, instances
from server.core.cell_ch import TRANSACTION_S, complete
from server.core.ch import query_tag
from server.core.provider_process import (
certified,
has_certificate,
participant_tag,
proves_stopped,
)
from server.insights.store import cell_catalog
from server.insights.domain.cell_participants import latest
from server.insights.store import provider_participants as participants
from server.insights.store.provider_participants import census


class CompleteReads:
Expand All @@ -36,44 +37,6 @@ async def query_all(self, sql: str, params: Any = None) -> list[dict[str, Any]]:
)


def complete(sql: str, *, sequential: int = 0) -> str:
return (
sql
+ " "
+ WHOLE_SQL
+ ", skip_unavailable_shards=0, prefer_localhost_replica=1, "
+ f"select_sequential_consistency={sequential}"
)


def latest(rows: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
out: dict[str, dict[str, Any]] = {}
# The latest winner cannot hide a conflicting older version from another replica.
seen: dict[tuple[str, int], tuple[str, str]] = {}
for row in rows:
try:
identity = str(UUID(str(row["participant_id"])))
version = row["version"]
state, proof = row["state"], row["host_proof"]
except (KeyError, TypeError, ValueError):
raise cell_control._refused("participant census contains an unreadable row") from None
if (
type(version) is not int
or version <= 0
or state not in ("running", "drained", "retired")
or not isinstance(proof, str)
):
raise cell_control._refused("participant census contains an unreadable row")
previous = out.get(identity)
observed = seen.get((identity, version))
if observed is not None and observed != (state, proof):
raise cell_control._refused(f"participant {identity} has conflicting equal versions")
seen[(identity, version)] = (state, proof)
if previous is None or int(previous["version"]) < version:
out[identity] = dict(row)
return out


def capture(rows: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
return {
identity: row
Expand All @@ -82,70 +45,6 @@ def capture(rows: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
}


async def census(ch: Any) -> dict[str, dict[str, Any]]:
"""Synchronize after the durable cutoff; every configured replica must answer."""
topology = await read_topology(ch)
if topology.cluster and not topology.entries:
raise cell_control._refused("participant census cannot resolve the configured cluster")
if problems := topology.refusals():
raise Refused(problems)
names = await cell_catalog.host_names(ch, topology) if topology.multi else {}
if problems := (await cell_catalog.own_refusals(ch, topology, names))[0]:
raise Refused(problems)
database = ch_names.database()
tables = await per_host(
ch,
topology,
names,
(
"SELECT hostName() AS h, engine FROM {source:tables} WHERE database={db:String} AND "
"name='_provider_participants'"
),
{"db": database},
)
if not tables or any(len(rows) != 1 for rows in tables.values()):
raise cell_control._refused("participant table is missing from a configured replica")
engines = {str(rows[0]["engine"]) for rows in tables.values()}
if len(engines) != 1 or not next(iter(engines)).endswith("MergeTree"):
raise cell_control._refused(
"participant replicas have unreadable or different table engines"
)
engine = next(iter(engines))
if engine.startswith("Replicated"):
rows = await per_host(
ch,
topology,
names,
(
"SELECT hostName() AS h, database, table, zookeeper_path, replica_name, "
"replica_is_active FROM {source:replicas} WHERE database={db:String} AND "
"table='_provider_participants'"
),
{"db": database},
)
if any(len(value) != 1 for value in rows.values()) or cell_catalog.extra_replicas(
row for value in rows.values() for row in value
):
raise cell_control._refused("participant replica set includes an unverified replica")
budget = cell_sync.Budget(cell_control.TRANSACTION_S)
await cell_sync._system(
ch,
f"SYSTEM SYNC REPLICA{cell_sync._on_cluster(topology)} "
f"{quote(database)}.`_provider_participants` LIGHTWEIGHT",
topology,
budget,
"participant census",
)
source = f"{quote(database)}.`_provider_participants`"
params = {}
if topology.multi:
source = f"clusterAllReplicas({{cluster:String}}, {source})"
params["cluster"] = topology.cluster
sql = f"SELECT participant_id, state, version, host_proof FROM {source} FINAL"
rows = await ch.query_all(complete(sql, sequential=int(engine.startswith("Shared"))), params)
return latest(rows)


async def stopped(ch: Any, initial: dict[str, Any], current: dict[str, Any] | None) -> bool:
if current is not None:
try:
Expand Down Expand Up @@ -222,7 +121,7 @@ async def replacement(
async def settle(run: Any, target: dict[str, Any]) -> None:
"""Fix one complete captured set; expiry never removes an obligation."""
with query_tag("cell.update.participants"):
initial = await run.owner.call(lambda: census(run.ch), total_s=cell_control.TRANSACTION_S)
initial = await run.owner.call(lambda: census(run.ch), total_s=TRANSACTION_S)
captured = capture(list(initial.values()))
done: set[str] = set()
deadline = time.monotonic() + run.args.wait
Expand All @@ -236,7 +135,7 @@ async def poll() -> bool:
done.add(identity)
return await replacement(run.ch, run, rows, set(captured), target)

live = await run.owner.call(poll, total_s=cell_control.TRANSACTION_S)
live = await run.owner.call(poll, total_s=TRANSACTION_S)
remaining = sorted(set(captured) - done)
if not remaining and live:
return
Expand Down
5 changes: 2 additions & 3 deletions server/cli/cell_update_pg.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@

from server.cli import cell_control
from server.core import cell
from server.router import fingerprints


@dataclass(frozen=True)
Expand Down Expand Up @@ -48,8 +47,8 @@ async def read(conn: Any, name: str) -> State:
"update needs complete prepared state naming this cell; prepare or restore it first"
)
current, pending = cell.json_object(row["current_fp"]), cell.json_object(row["next_fp"])
if not fingerprints.readable(current) or (
pending is not None and not fingerprints.readable(pending)
if not cell.readable_fingerprint(current) or (
pending is not None and not cell.readable_fingerprint(pending)
):
raise cell_control._refused("update needs readable local current/next fingerprints")
assert current is not None
Expand Down
7 changes: 4 additions & 3 deletions server/cli/cell_update_probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from typing import Any

from server.cli import cell_control, cell_prepare
from server.core.cell_ch import TRANSACTION_S
from server.router.app import REGISTER_ANSWER_MAX_S

# Read/probe/commit bounds already used by prepare, plus its reply margin.
Expand All @@ -21,7 +22,7 @@ async def clean() -> None:

try:
await run.owner.call(
lambda: cell_prepare._write_nonces(run.ch, nonces), total_s=cell_control.TRANSACTION_S
lambda: cell_prepare._write_nonces(run.ch, nonces), total_s=TRANSACTION_S
)
async with asyncio.timeout(run.args.wait + ANSWER_S):
answer = await cell_prepare._register(
Expand All @@ -36,9 +37,9 @@ async def clean() -> None:
cell_control.preconditions(answer)
except BaseException:
try:
await run.owner.call(clean, total_s=cell_control.TRANSACTION_S)
await run.owner.call(clean, total_s=TRANSACTION_S)
except BaseException: # noqa: BLE001 — preserve the original operation failure.
pass
raise
await run.owner.call(clean, total_s=cell_control.TRANSACTION_S)
await run.owner.call(clean, total_s=TRANSACTION_S)
return answer
46 changes: 46 additions & 0 deletions server/core/cell.py
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,52 @@ def matches(mine: Mapping[str, Any], value: Mapping[str, Any] | None) -> bool:
return True


def readable_fingerprint(value: Any) -> bool:
"""Known fields have exact types; unknown JSON fields remain reserved and intact."""
if not isinstance(value, dict) or set(value) != {"v", "fields"}:
return False
if type(value["v"]) is not int or value["v"] != FINGERPRINT_VERSION:
return False
fields = value["fields"]
if not isinstance(fields, dict):
return False
try:
# Reject cycles, non-JSON values and nonfinite floats before walking nested fields.
json.dumps(value, allow_nan=False)
except (TypeError, ValueError, RecursionError):
return False
pending = [value]
while pending:
item = pending.pop()
if isinstance(item, str):
try:
if "\0" in item:
return False
item.encode()
except UnicodeError:
return False
elif isinstance(item, dict):
if any(not isinstance(key, str) for key in item):
return False
pending.extend(item.keys())
pending.extend(item.values())
elif isinstance(item, list):
pending.extend(item)
elif item is not None and type(item) not in (bool, int, float):
return False
for name, default in DEFAULTS.items():
item = fields.get(name, default)
if type(item) is not type(default):
return False
if isinstance(item, list) and any(not isinstance(part, str) for part in item):
return False
for name in ("configured_orgs", "end_user_orgs"):
for org in fields.get(name, []):
if named_org(org) is None or org_name_refusal(org):
return False
return True


def digest(fields: Mapping[str, Any]) -> str:
"""A short digest of a fingerprint for the gate's reason; never compared."""
blob = json.dumps(dict(fields), sort_keys=True, separators=(",", ":"))
Expand Down
Loading