spec-7.3: LMS worker pool (D1-D9) + lane review fixes + main merge#33
Open
sqlrush wants to merge 23 commits into
Open
spec-7.3: LMS worker pool (D1-D9) + lane review fixes + main merge#33sqlrush wants to merge 23 commits into
sqlrush wants to merge 23 commits into
Conversation
added 23 commits
July 9, 2026 10:35
…7 P0 substrate
Bring the spec-7.2-ic-data-plane branch (25 commits, D0-D7) on top of
stage7-p0-integration so the D7 fault-injection legs (L4/L5/L6) can be
re-evaluated against the formation/reconfig-liveness/xid-authority P0
substrate (spec-6.15b PR#24 + spec-4.6a PR#26 + spec-2.29a PR#25).
Conflict resolutions (cross-surface grd/lmon/gcs + counter baselines):
cluster_lmon.c -- keep BOTH the stage7-p0 iter_started_at profiling
timestamp AND the spec-7.2 D1 lazy-duty >=1Hz floor;
reuse the single iteration-start timestamp for both
(TIER_1 and stub loops).
cluster_lms.c -- union of includes (cssd + epoch from P0, gcs_block +
write_fence from spec-7.2).
test_cluster_lmon.c -- keep BOTH new unit tests; UT_PLAN 6->7 (both sides
bumped 5->6 independently; the union is 7).
t/015,017,018,020-024,030 -- injection registry 158 base + spec-2.29a
(+1 cluster-qvotec-marker-service-hold) + spec-7.2 D6
(+2 cluster-lms-data-dispatch/conn-reset) = 161 (was
159 on P0 / 160 on spec-7.2 in isolation). Name list
regenerated from the merged cluster_inject.c registry
(ORDER BY name); t/030 M5 sub-keys 161x2 = 322.
Verified: clean build (all targets), cluster_unit 160/160 binaries pass,
zero conflict markers remain. D7 leg re-evaluation follows in later
commits.
…KIP t/360 L5
The cluster-lms-conn-reset injection modelled a single epoch-bump DATA-mesh
reset, but its only arm path is the process-local injection GUC, which stays
armed and re-set skip_pending on every LMS tick -- a reset storm that starved
the mesh and (on the passive side) tripped the accept/close WES churn into a
crash, then hit the F1-8 block_device recovery wall. That made t/360's L5
leg KNOWN-BLOCKED.
Fix: latch the injected reset so it fires exactly ONCE per LMS process, and
only latch after it has actually torn down a live connection (so an arm-
before-connect race retries instead of wasting the single shot). The real
epoch-bump reset path (cur_epoch != dp_last_epoch) is unaffected.
t/360 L5 is now a real e2e leg (installcheck, 3x stable):
L5.1 the injected reset fires once (reset-log count +1)
L5.2 the mesh reconnects -- both nodes emit a fresh "state CONNECTED"
after the reset (node0 re-dials, node1 re-accepts)
L5.3 one-shot: exactly one reset event, no per-tick storm
L5.2 proves reconnect via log evidence rather than a post-reset SQL probe:
by that point the value-gate ping-pong has burned fault_t's rows' xmax into
recycled TT slots, so ANY access (read or write) fail-closes with "cluster
TT status unknown" -- the pre-existing #119 recycled-slot wall, orthogonal to
the DATA plane and not caused by the reset. Block-shipping capability itself
is quantified by the L2 value gate (394 ships, p99 <= 500us).
L4 (F1-8) and L6 (F6-2) stay honest SKIP; the stage7 P0 substrate was
evaluated and does not unblock either (L4 needs the F1 recovery-vs-membership
fix; L6 needs the data-dispatch injection point repositioned -- observability
polish, block-over-DATA already proven by t/358 + the value gate).
Spec: spec-7.2-ic-data-plane-decoupling.md D7 / F6-1
cluster_lms_shard_for_tag() maps a block's BufferTag to a worker in
[0, n_workers) via a fixed-order 20B field hash (hash_bytes_extended).
It is the single source of the (tag -> worker) mapping: sender stream
select, receiver stream, outbound ring select and per-tag private tables
all route through it, so per-tag FIFO ordering survives the N-way DATA
plane split (8.A). N == 1 is the spec-7.2 topology identity (worker 0).
New files: cluster_lms_shard.{c,h} (+ CLUSTER_LMS_MAX_WORKERS cap = 8),
test_cluster_lms_shard.c (9 truth-table tests: in-range / N=1 identity /
determinism / tag-only dependence / all-msg-types same shard / field
sensitivity / distribution / boundary / golden). Wired into the backend
OBJS and the cluster_unit Makefile (links libpgcommon for the hash).
Spec: spec-7.3-lms-worker-pool.md (D1)
Add the LmsWorker1..7Process aux process types (worker 0 stays the plain
LmsProcess). Each is a distinct AuxProcType so it gets its own
BackendStatus / AuxiliaryProc slot (sidestepping the one-process-per-aux-type
assumption); its display BackendType is the reserved B_LMS_WORKER. N=1 forks
no worker, so LmsMain runs exactly as in spec-7.2 (topology identity).
- miscadmin.h: append LmsWorker1..7Process (preserves existing AuxProcType
numeric values) + AmLmsWorkerProcess() / ClusterLmsWorkerIdForType() helpers.
- proc.h: NUM_AUXILIARY_PROCS 18 -> 25 (StaticAssert >= NUM_AUXPROCTYPES holds).
- auxprocess.c: BackendType switch, ProcSignal-skip set, pre-main signal set,
and dispatch (-> LmsWorkerMain(worker_id)) all extended for the workers.
- postmaster.c: LmsWorkerPIDs[] mirrors LmsPID across all 9 lifecycle sites
(start/respawn/SIGHUP/reaper/crash/shutdown/pmState-gate/broadcast); pool
size driven by the new GUC.
- cluster_lms.{c,h}: LmsWorkerMain (spawn + identify + idle for D2; DATA-plane
wiring is D3-D6) + worker_pids[] in the shared state + accessor.
- GUC cluster.lms_workers (POSTMASTER, default 2, [1, CLUSTER_LMS_MAX_WORKERS]).
Tests: test_cluster_backend_types +1 (worker aux-slot contiguity / worker_id
derivation / aux-slot-count invariant); new TAP t/364_lms_worker_pool
(lms_workers=2/4/1 spawn + identity + N=1 no-worker + clean shutdown, 8/8).
NB: changing NUM_AUXPROCTYPES/NUM_AUXILIARY_PROCS needs a full make clean --
a stale backend_status.o PANICs "invalid auxiliary process type" at runtime.
Spec: spec-7.3-lms-worker-pool.md (D2)
Wire the DATA-plane worker identity into the HELLO handshake, filling the offsets reserved for spec-7.3 in the frozen 64B ABI (41 = worker_id, 42 = n_workers). cluster_ic_hello_set_worker_fields() stamps them on the send path (build_hello leaves them zero, so a CONTROL / pre-7.3 HELLO stays byte-identical to spec-7.2); cluster_ic_hello_worker_id() / cluster_ic_hello_n_workers() read them on the verify path. This is the wire substrate for the D3 shard-aligned topology + cluster-uniform n_workers negotiation; the tier1 send/verify integration (per-worker listener ports, fail-closed reject on worker_id/n_workers mismatch) lands next. test_cluster_ic +1: build-alone leaves worker fields zero (compat pin) then set + accessor roundtrip. Spec: spec-7.3-lms-worker-pool.md (D3)
…LLO gate Give each LMS DATA worker its own tier1 transport state so the per-worker mesh is correctness-clean, and enforce the shard-aligned topology + a cluster-uniform worker count on the HELLO handshake (8.A). tier1 (cluster_ic_tier1.c): - Carve one ClusterICTier1Shmem instance per slot: slot 0 = CONTROL, slot 1+c = DATA worker channel c. CONTROL and DATA worker 0 keep their spec-7.2 offsets, so lms_workers=1 is byte-identical. This de-collides peers[] (read by the send-gate) + listener metadata across worker procs. - set_my_data_channel(channel, n_workers) re-aims the Tier1Shmem alias at the worker's instance; listener/dial port become data_port + channel (the shard-aligned i<->i mesh: worker c only ever pairs with peer worker c). - HELLO send stamps (channel as worker_id, n_workers); both verify paths reject fail-closed on the DATA plane unless the peer claims MY channel and the SAME n_workers (a skew = shard misroute = order break = P0). - get_plane_misroute_reject(DATA) aggregates the worker channels. LMS wiring: - cluster_lms_data_plane_startup(worker_id, n_workers) selects the channel. - LmsWorkerMain runs its per-channel DATA mesh tick (was idle in D2); worker 0 keeps LmsMain's full loop on channel 0. Verification so far: builds + installs; single-node regression green (t/364 lms_workers=2/4/1, test_cluster_ic 22/22 incl HELLO worker fields, backend types 7/7). The 2-node e2e (worker i<->i mesh + n_workers-mismatch reject) is NOT yet run: it needs the ClusterPair harness to reserve a per-node data-port gap (spec-7.3 R3/F2) so per-worker listeners don't collide across the two same-host nodes; that harness change + the 2-node TAP land next. Spec: spec-7.3-lms-worker-pool.md (D3)
Close the D3 cross-node verification (was builds-only).
- Cluster.pm: get_free_port_range(n) -- reserve a block of n consecutive free
ports. The worker pool binds a listener per worker at data_port + worker_id,
so a 2-node test needs each node's [data_port, data_port+workers) range free
and non-overlapping (spec-7.3 R3/F2). Additive; existing callers unaffected.
- ClusterPair.pm: data_port_span option (default 1 = historic single-port
allocation, so every existing pair test is unchanged); span>1 uses the range.
- cluster_ic_tier1.c: tag the DATA-plane "HELLO verified, state CONNECTED" logs
with the worker channel so the test can prove the shard-aligned i<->i mesh
formed per worker (CONTROL messages kept verbatim).
- t/365_lms_worker_pool_2node.pl (13 tests, real 2-node install):
* matched pool (both lms_workers=2): each node spawns worker 1 + binds its
listener; the DATA mesh reaches CONNECTED on BOTH worker channels (verified
on the passive side, asserted on the merged log); zero plane misroutes.
* mismatch pool (node0=2, node1=3): CONTROL still forms; the DATA HELLO is
refused fail-closed with "HELLO DATA worker mismatch" -- the 8.A gate that
keeps the two ends' shard tables identical.
D3 complete and cross-node verified (rule 8.B).
Spec: spec-7.3-lms-worker-pool.md (D3)
…routing Route each backend block request to the worker owning its tag's shard. - cluster_lms_outbound.c: single DATA outbound ring -> rings[N] (one Q6-B single-consumer FIFO + its own lock per worker; sizing = compile-time cap, live count = cluster.lms_workers). enqueue/drain/depth take worker_id; rings[0] = worker 0 (lms_workers=1 byte-identical to 7.2). - cluster_lms.c: cluster_lms_wakeup(worker_id) kills worker_pids[worker_id]; LmsMain publishes worker_pids[0] + drains rings[0]; LmsWorkerMain drains rings[worker_id]. - cluster_gcs_block.c: cluster_gcs_block_payload_shard(msg_type, payload, len, n) extracts the BufferTag from a staging-path payload (REQUEST/FORWARD/ INVALIDATE, tag pinned at offset 16 by StaticAssert) and hashes it via cluster_lms_shard_for_tag; returns -1 (8.A fail-closed) for an unroutable DATA frame. - cluster_grd_outbound.c: the DATA-plane staging entry computes the shard and enqueues to rings[shard] (refuses to stage on -1). 8.A argument (survey-verified): only REQUEST/FORWARD/INVALIDATE are staged through the ring (each tag-affine); REPLY (no tag, request_id-correlated) and INVALIDATE-ACK are sent DIRECTLY from the receiving worker's dispatch handler, which runs in the worker[shard] process (D3 mesh) so already rides the correct channel -- the INVALIDATE-ACK vs re-REQUEST per-tag FIFO (D0-# WATCH) holds because both share worker[shard(tag)]'s channel. A misrouted frame would break message order, so payload_shard fails closed rather than default. Verification: builds + installs; regression green (t/364 8/8 single-node, t/365 13/13 2-node mesh, cluster_unit no new fails). The multi-tag routing e2e (distinct tags land on distinct workers; INVALIDATE-ACK channel) lands at D9. Spec: spec-7.3-lms-worker-pool.md (D4)
…cs multi-node TAP
The LMS worker pool binds a DATA listener per worker at data_port + worker_id
(D3), and cluster.lms_workers defaults to 2, so each node now needs a 2-port
DATA range. The pre-worker-pool 2-node gcs TAP tests requested only a single
DATA port (data_port_span default 1), so node 0's worker-1 port collided with
node 1's base ("bind 127.0.0.1:<p> failed: Address already in use"). Reserve
data_port_span => 2 on each, matching t/365.
- t/111, t/112, t/113, t/114, t/116, t/252: data_port_span => 2
t/115 (3-node ClusterTriple) has the same latent collision but ClusterTriple
lacks a data_port_span option; deferred to a follow-up harness addition.
Spec: spec-7.3-lms-worker-pool.md (D3/D5)
Shard the master-side GCS block dedup table into per-worker private instances so the LMS worker pool never contends on one lock and never shares dedup state across workers. - cluster_gcs_block_dedup.c/.h: the single ClusterGcsBlockDedupShared (HTAB + lock + 5 counters) becomes dedup_shards[CLUSTER_LMS_MAX_WORKERS], each with its own HTAB + lock + counters, plus an always-present ctl header holding the live shard count and a misroute fail-closed counter. Hot-path APIs (lookup_or_register / install_reply[_ex] / remove) take an explicit worker_id = the serving worker's DATA channel; a worker_id outside [0, live shards) is a mis-route -> FULL + misroute_failclosed_count++ (never a wrong-shard serve). GC (TTL sweep / node-dead / backend-exit) runs in non-worker processes and iterates every shard; counter accessors sum across shards. Shard 0 keeps the spec-2.34 shmem/HTAB names so lms_workers=1 is a byte-identical topology. Sizing scales with cluster.lms_workers; per-shard capacity is unchanged so dedup_full_count does not regress. - cluster_gcs_block.c: the master REQUEST handler computes shard(req->tag) and asserts it equals its own DATA channel before touching the shard; on a mismatch it fails closed (counter + LOG-once + drop; sender retransmits via 53R90) rather than serve from a shard it does not own. All 5 dedup calls in the handler pass the worker id. Correctness: request_id is stably bound to one tag, so all of a request's messages route to worker[shard(tag)] and its dedup entry lives in exactly one shard; sharding by shard(tag) preserves dedup semantics and a mis-route fails closed in-spec (8.A). Scope: only the DATA-plane master-side dedup HTAB is sharded. The GES retransmit dedup (cluster_ges_dedup) is CONTROL plane and stays on worker 0; the requester-side outstanding-reply slot table is per-backend, not an LMS table. Test: test_cluster_gcs_block_dedup.c -- 9 behavioral tests over an N-shard fake HTAB (per-worker isolation, dedup lifecycle, cross-shard counter sum, cross- shard GC, out-of-range fail-closed, N=1), verified to fail when sharding is disabled. Regression: t/112 (dedup 2-node), t/364/t/365 (worker pool), t/020 (shmem registry) green at default cluster.lms_workers=2. Multi-tag dispatch + injection-forced misroute e2e land at D9. Spec: spec-7.3-lms-worker-pool.md (D5)
…(park kept for control) When the GCS block family is on the DATA plane, the worker[shard] that receives a GCS_BLOCK_FORWARD CR / undo-fetch / undo-verdict request now serves it INLINE and ships on its own channel -- retiring the spec-6.12b park -> LMS-poll -> LMON-ship indirection there (no worker-0 handoff, no 100 ms latency; CR construction parallelizes across workers by shard). The park-serve path is RETAINED for the CONTROL-plane fallback: a node with the data plane off (no data_addr) still dispatches the FORWARD in LMON, whose tight IC dispatch loop must not walk undo I/O (the light-work rule), so it parks for LMS worker 0. (The requester gates the CR fetch on the crossnode_cr_data_plane GUC, not on the data plane being up, so the control-plane path is reachable and must stay.) - cluster_cr_server.c: extract cr_serve_slot() (the PG_TRY -> DENIED serve envelope, verbatim from the drain) and cr_build_and_send_reply() (the reply build) as shared helpers; the park drain + ship_ready and the new inline serve both call them. Add cluster_gcs_block_forward_serve_inline(fwd, kind): populate a stack carrier from the payload, serve through cr_serve_slot() inside a per-call scratch MemoryContext (reset on return -- the DATA dispatch loop is long-lived), then ship exactly one reply (DENIED on any refusal / wave-GUC-off / malformed request, Rule 8.A). - cluster_gcs_block.c: the forward handler routes CR/undo/verdict to the inline serve when cluster_gcs_block_family_on_data_plane(), else to the light-work park submit. 8.A: the construction (cluster_cr.c) re-throws on any uncertainty; cr_serve_slot's PG_TRY -> FlushErrorState -> DENIED envelope (shared, unchanged) converts it to a fail-closed DENIED, never a worker exit or a wrong-order construction. Test: t/354 adds L6 -- the cluster-lms-cr-construct fail-closed injection keeps the requester at 53R9G, leaves the origin's serving worker alive, and recovers once cleared. t/354 + t/346 reserve data_port_span=2 (the D3 per-worker port contract). Regression green: t/354 (21), t/346 (undo/verdict), t/113, t/364, t/020 (161 injections + region list unchanged), cluster_unit (cr_server_policy 7/7). The control-plane park path is unchanged (shared helpers validated by the inline path); its data-plane-off e2e is a follow-up. Spec: spec-7.3-lms-worker-pool.md (D6)
…S worker pool) - fence ×N: cluster_gcs_block_forward_serve_inline gains a write-fence gate ahead of the kind switch / any construct/read, so a write-fenced node refuses (immediate DENIED) to ship a CR image / undo bytes / commit verdict on the DATA plane -- restoring the parity the worker-0 park ship path keeps that D6's inline move dropped (8.A: split-brain read / false-committed verdict). New cr_server_fence_refused_count + dump row + cluster-lms-cr-fence-refuse injection for deterministic TAP. - epoch-reset ×N: per-worker DATA-mesh reset LOG; each worker observes the shared epoch bump in its own process (the tier1 sender gate is the backstop). - graceful per-worker recycle (Path A): SIGTERM one DATA worker respawns just that slot with no node crash cascade; a hard SIGKILL cascades the whole node (PG aux crash model), recovery blocked on the orthogonal F1 wall (t/360 L4). - tests: t/354 L7 (CR fence), t/346 L4c (undo/verdict fence), t/365 L5 (epoch ×N reset) + L6 (graceful recycle isolation). - port fix (pre-existing lms_workers=2 default): data_port_span=2 for the ClusterPair 2-node CR tests D5 missed (t/215/216/360); t/360 L5.3 -> one-shot per worker. Baselines: injection registry 161->162, cr category 57->58. Spec: spec-7.3-lms-worker-pool.md (D7)
… arm unconditionally P2: the || short-circuit let a genuine fence skip cluster_injection_should_skip, leaking a pending injection arm to a later call (test-determinism only). Hoist the consume into a local ahead of the predicate (F6-1 local-var idiom). Spec: spec-7.3-lms-worker-pool.md (D7)
- 4 per-worker counters in ClusterLmsSharedState (dispatch / direct_reply / conn_resets / inline_serve), bumped from the worker's own slot: dispatch in cluster_ic_dispatch_envelope (real dispatch path, after the chunk short-circuit; no-op outside an LMS process), direct_reply + inline_serve in the D6 inline serve, conn_resets in the D7 epoch-reset closure. - per-worker inline-serve duration histogram (16 buckets, LMS-owned -- the requester-side ship_latency_hist is a different measurement, not resized; the R4 slow-shard backstop). - dump surface: bare keys = pool aggregates (7.2 single-value semantics' natural generalization), _w<N> suffixed keys = per-worker detail bounded to the live pool (w < cluster.lms_workers). - cluster.lms_nice (Q8): optional SIGHUP int [-20,0] default 0 = leave the inherited priority alone; setpriority best-effort at worker entry + on config reload (POSIX, #ifndef WIN32). - injection points: NONE added -- cluster-lms-worker-crash is untestable under the D7 Path A crash model (hard-crash cascades; graceful recycle needs plain SIGTERM), and the per-worker dispatch counter supersedes cluster-lms-worker-dispatch as the assertable surface (the mispositioned cluster-lms-data-dispatch point stays a spec-7.2 follow-up). - tests: t/364 L2b/L4b (lms_nice GUC + live-pool key bound + 16-bucket agg hist), t/354 L5b (inline serve + hist moved; direct_reply >= inline_serve), t/365 L5 (per-worker conn-reset counters, aggregate = sum), t/360 L2b (dispatch aggregate = per-worker sum on both nodes). - baseline sweep: 7 injection-count sites D7 missed (t/017 x2, t/018, t/021, t/022, t/023, t/024, t/030 M1 + M5 322->324) bumped to 162. Spec: spec-7.3-lms-worker-pool.md (D8)
…e2e legs
Renumber (hub arbitration): main (PR#27) owns t/365 for the spec-7.1a
write-write TAP and nightly shards select by numeric prefix, so this
lane's unpushed 2-node pool TAP moves to t/367 (364 = 7.3 single-node,
365 = 7.1a on main, 366 = 7.2a, 367 = this file). Banner and the
t/360 / t/112 cross-references move with it.
New D9 legs (wire-traced grounding; the 'single tag' SQL premise is
false under pgrac MVCC -- a heap read also fetches its writers' undo /
verdict blocks whose tags shard independently):
L7 multi-tag fan-out: X ping-pong over a ~40-block shared table moves
BOTH per-worker dispatch counters on BOTH nodes + aggregate inline
serve liveness + zero plane misroutes + zero dedup-shard drops.
L8 per-tag affinity: single-block invalidate/re-read keeps the
serving-side per-frame shard check clean and moves the SAME
worker set on both ends (double-end shard agreement, R1).
L9 N=1 whole-stack identity sentinel: lms_workers=1 pool on the
pre-7.3 single data port -- no worker sibling, no worker-1
channel, live traffic rides worker 0 alone (aggregate == _w0),
zero misroutes (spec-7.3 §3 contract 5 under real traffic).
Spec: spec-7.3-lms-worker-pool.md (D9)
… SQL surface Extract cluster_gcs_block_payload_shard into cluster_gcs_block_shard.c (a pure file, same D1/D5 pattern) so the cluster_unit suite links the REAL staging-path router: new test_cluster_gcs_block_shard pins the D9 ring-group routing truth table (route == shard_for_tag agreement for REQUEST / FORWARD / INVALIDATE, ACK -> same-tag re-REQUEST affinity, DATA registry partition REPLY / INVALIDATE-ACK direct-send whitelist, payload-length ABI pin refuses mismatched frames, N=1 degenerate, non-tag fields never move the route). Surface the D5 per-worker dedup-shard fail-closed drop counter in SQL (gcs key dedup_misroute_failclosed_count, 85 -> 86; rule 17: the D5 counter had an accessor but no observable surface) with the test_cluster_debug stub; bump the gcs key-count baselines across the eight consumers (t/110-116, t/200 unaffected) and fix the stale wait-event / key-count comments in t/112 / t/116 while there. Spec: spec-7.3-lms-worker-pool.md (D4/D9)
run_73_pool_scaling.pl boots the 2-node block_device pair at cluster.lms_workers = 1 and 2 and measures (a) aggregate serve throughput (lms_direct_reply_count delta/s) under the spec's motivating CR-construction workload -- node0 UPDATE churn + node1 lagged REPEATABLE READ scans -- with per-worker inline-serve fan-out evidence, and (b) single-block requester ship-latency p99 from the ship_hist delta. Medians over PS_ROUNDS; PS_EXTRA_GUC injects lever GUCs for control experiments; the reader probe is DO-wrapped so fail-closed retryable visibility errors roll back the probe, not the client. Measured verdict (2026-07-10, 本机 loopback): p99 leg PASS (1.00x); the >=1.6x aggregate throughput leg is structurally unreachable on loopback -- the dominant cross-node serve stream is the runtime visibility undo/TT probe whose synthetic tag is constant per origin-segment (block-0 slice), so it rides ONE worker by per-tag FIFO design, and the residual block-family load sits far below single-worker saturation. Numbers + root-cause analysis in the pgrac measure report; gate disposition escalated per rule 6.A. Spec: spec-7.3-lms-worker-pool.md (D9/Q9)
…, 367) t/364 (single-node pool smoke) and t/367 (2-node topology + routing + fault legs) were nightly orphans -- the shard ranges stop at 363. Give the family its own stage7-lms-pool shard; 365 / 366 stay with the spec-7.1a (main) / spec-7.2a lanes per the hub number ledger. Spec: spec-7.3-lms-worker-pool.md (D9)
… default
With cluster.lms_workers defaulting to 2 (spec-7.3 D2) every node binds
DATA listeners at data_port + {0,1}, but ClusterPair only reserved a
single allocator port unless the test passed data_port_span, and
ClusterTriple had no span support at all -- any 2-/3-node test with an
active data plane could FATAL on 'Address already in use' (t/348 did,
in the D9 regression gate). The D5/D7 point fixes covered 8 files of
~85 affected.
Root fix: both fixtures now default data_port_span to 2 (the shipped
cluster.lms_workers default) and allocate the range via
get_free_port_range; ClusterTriple gains the data_port_span option.
Tests that override the pool size keep passing an explicit span.
Verified: t/348/349/350/355 (84 tests, incl. the 3-node triple) + the
15-file lane family batch (426 tests, incl. the previously unspanned
t/117 canary) all green.
Spec: spec-7.3-lms-worker-pool.md (D9)
… shard misroute guards P1-1: the inline-serve fence gate ran only BEFORE construction; a qvotec lease expiring while the serve walks undo I/O / TT scans (ms-scale) let the just-constructed CR image / undo bytes / commit verdict leave the now-fenced node (TOCTOU; the park ship path gated at ship time and never had this window). Re-check the same predicate after cr_serve_slot and before the reply build; on a hit discard the constructed result and ship the fail-closed DENIED (fence_refused bumps; requester keeps its retransmit budget / 53R90). New decision-style injection point cluster-lms-cr-fence-recheck models the mid-serve expiry for the TAP TOCTOU leg (registry 162 -> 163; baselines t/015/017/018/020-024/030, M5 324 -> 326). P2-1: per-frame shard(tag)==my_channel cross-checks covered only the REQUEST dedup path (1/5 message classes). Add the same Assert + note_misroute fail-closed guard to the FORWARD inline-serve entry, the INVALIDATE receive side, and the INVALIDATE-ACK receive side, so a future routing regression surfaces as a counted drop instead of a silent per-tag order break. Spec: spec-7.3-lms-worker-pool.md
…AP legs for real
The t/354 L6/L7 and t/346 L4c legs were guarded by
$pair->can('inject_skip_set'), a helper that never existed anywhere in
the tree -- the guard was always false, so all three legs skipped
forever and their closure counts were skip-green: the fence gate (the
D7 headline deliverable) had zero executing test coverage.
Rewrite them on the t/360 L5 conf-arm pattern
(cluster.injection_points='<point>:skip' + reload; the LMS workers
re-read the GUC on SIGHUP), with counter snapshots taken only after the
arm is observably effective so pre-arm serves cannot contaminate the
static-counter assertions. Disarm uses an explicit '<point>:none'
entry: a bare empty list does not recall a colon-typed arm (the
assign_hook's not-in-list revert only recalls WARNING arms).
New t/354 L7b pins the P1-1 ship-time re-check via the TOCTOU discard
signature: construction DID run (full/partial advanced) yet the reply
stays the fail-closed DENIED (fence_refused advanced, requester keeps
53R9G), proving the discard sits between construct and ship.
RED/GREEN proven: with both fence branches locally neutered the legs
fail exactly on the gate assertions (t/354: 25-27, 31-32; t/346 L4c:
fence_refused static + undo served while fenced); with the shipped
gate they pass 35/35 + 45/45.
t/358 gains the standard Author/IDENTIFICATION banner block (it now
enters the CI matrix; comment-header gate was red on it).
Spec: spec-7.3-lms-worker-pool.md
… + fault family) t/358 and t/360 were in no CI matrix: the nightly shard ranges skipped 358-360 and fast.yml's whitelist never included them, so neither file ever ran in automation. Add a stage7-data-plane shard covering 358-360 (t/359 is an unused reserved number; the range matcher iterates existing files) and drop the stale 'reserved by the parallel spec-7.2 lane' comment -- that lane landed and the reservation is resolved. Spec: spec-7.3-lms-worker-pool.md
…emantic merge Merge origin/main (774f088) into the spec-7.3 lane. Merge, not rebase: 22 lane commits vs 62 main-side commits overlap 100+ files, and the lane hashes are referenced by the review/closure record. Semantic resolutions (union of both lanes' features): - ClusterCrServerStat: main's MULTI_VERDICT_SERVED/DENIED keep 7/8; the lane's FENCE_REFUSED renumbers to 9 (later lander moves). - cluster_cr_server.c: lane D6 inline-serve skeleton + main's spec-7.1 multixact member-verdict machinery (submit/serve/shared own-xid verdict core) + spec-7.1a authority_scn co-sample; the reply build adopts GcsBlockReplyStatusCarriesUndoAuthTrailer + SetAuthorityScn; cr_serve_slot dispatches all three undo kinds. Multi-verdict stays on the park path (worker[shard] parks, worker 0 drains, fence-gated ship) -- inline serve for it is a follow-up, not merge invention. - dedup: spec-7.2a capacity/eager-GC (auto-size floor, probe-bounded eager reclaim, evict_count, max_entries stamp, saturation LOG-once) ported onto the lane's per-worker shard structure; the floor applies per shard (a hot tag's shard must absorb the worst case alone). The 7.2a unit suites adapt to the sharded API (shard 0). - conn-reset observability: main's scalar lms_conn_resets is superseded by the lane's per-worker counters (aggregate = sum); main-only fix F6-1 r2 Finding 3 (epoch bump takes precedence over the injected reset in the same tick) is ported into the lane's data-plane tick. - t/360: lane's per-worker one-shot L5.3 semantics + main's new L5.4/L5.5 post-reset convergence legs (fresh table avoids the #119 wall; counter key follows the merged per-worker name). - t/360 double-occupancy resolved per the hub number ledger: the 7.1 multi-member refuse test renames to t/368 (the lms data-plane faults file keeps 360); nightly gains stage7-multixact (359 368) and stage7-data-plane narrows to '358 360'. - t/366 pins cluster.lms_workers=1: its hand-rolled rig reserves one data port per node, and the shipped default pool of 2 cross-wires consecutive free ports (HELLO DATA worker mismatch). N=1 is the spec-7.2 topology identity the test was written against. - Baselines recomputed from the merged authoritative sources: injection registry 164, t/030 M5 328, cr dump 71, gcs dump 89; t/015 name list carries all three new points; stale 161/162 description texts swept to 164. Verified on the merged tree: cluster_unit 168 binaries, cluster_regress 13/13, PG 219/219, TAP batches (015-030 baselines, 110-117 gcs family, 215/216/300, 346/354/358/359/360/364/365/366/367/368) all green; clang-format/comment-headers/cppcheck/scn-cmp/clog-overlay/ges-mode gates clean. Spec: spec-7.3-lms-worker-pool.md
sqlrush
force-pushed
the
spec-7.3-lms-worker-pool
branch
2 times, most recently
from
c3d870a to
774b823
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spec-7.3 LMS worker-pool lane (D1-D9 complete), plus the lane-wide code-review fix round, merged with current main (7.1/7.1a/7.2/7.2a).
Lane deliverables (18 commits, D1-D9)
Review fix round (3 commits)
Merge with main (1 merge commit)
Test plan
Spec: spec-7.3-lms-worker-pool.md