feat(cluster): spec-7.1 cross-instance positive interread — integrate read-side onto main#31
Merged
Merged
Conversation
added 24 commits
July 10, 2026 07:36
Six vis53r97_leg_* counters attribute every cross-instance fail-closed refusal to its source leg (3 origin-serve legs: invalid_scn / zero_match / srv_other; 3 requester legs: covers_refuse / multi_unresolvable / xmax_unprovable), exposed as cluster_dump_state() cr-section rows. The census before/after ledger reads these instead of a lump failclosed total, so each leg's zeroing is provable by the mechanism that closed it. No behavior change: counting only, every refusal direction untouched. Spec: spec-7.1-cross-instance-positive-interread.md (D0 census / D5)
t/990 (not in any schedule; run by hand) drives the D0 before-census and the D4 after-census on the same yardstick: S3-shaped / S5-shaped dual-node pgbench loads with client-side ERROR histograms plus per-phase cluster_dump_state() counter deltas, and six deterministic multixact cross-read probes (A-F). Crash-aware: a node death under load is reported with its TRAP signature as census data instead of failing the run. Spec: spec-7.1-cross-instance-positive-interread.md (D0-①)
t/991 (scratch; promoted + numbered at ship) applies one deterministic seeded DML workload (updates / delete+reinsert / FOR SHARE / KEY SHARE + update) to a plain single-node PG and to a 2-node pair with statements alternating nodes, then requires the final table fingerprint read from BOTH cluster nodes to equal the native gold standard. Fail-closed refusals retry with fresh transactions (correctness-neutral); a statement or fingerprint that never succeeds fails honestly -- pre-D4 the cluster leg is expected red on availability, and that red IS the before-state this spec's D1-D4 must close before L6 can gate the ship. Validated: native leg green (fingerprint captured); cluster leg applies under cross-node row-lock convoys (slow pre-D4 -- run with PARITY_NSTMTS small for smoke). Spec: spec-7.1-cross-instance-positive-interread.md (D6 L6)
Same gap fast-gate caught on t/357: the local header check skips untracked files, so the census / parity scratch drivers were committed without the Author banner.
…eleton t/993 (scratch; promoted + numbered at ship) is the缺口 C.1 positive acceptance: a heap block written from BOTH nodes, then the final content read cross-node from each, with a pageinspect infomask铁证 that no XMAX_COMMITTED+XMAX_INVALID poison hint survives (the C.1 silent- corruption signature). Every increment is write_retry'd so a transient pre-D4 fail-closed does not silently drop it; the exact COUNT is asserted unconditionally (dup/lost rows = read corruption), while the exact SUM is a pure positive assertion only when all writes committed and otherwise defers to D4 (a write-path completeness gap, distinct from the read silent-wrong this leg targets). Skeleton posture: read fail-closed and write-incompleteness both take the documented-escape path (t/347 L2b口径), so the file is green now; D4 (census收敛) removes the escapes and asserts the exact values purely once the D1/D3 interread legs land. Smoke (ROUNDS=1): 8/8, all_committed=0 (pre-D4 double-write fail-closes, as expected). Spec: spec-7.1-cross-instance-positive-interread.md (D6 L4)
… origin for a verdict The census S5 dominant 53R97 leg is a live-slot overlay miss (HC181): a foreign xmin whose V4 status hint has not ARRIVED, which is NOT the same as the xid having no terminal state -- the origin's durable TT is authoritative (Fast Commit stamps it pre-commit). Before failing closed, the xmin resolve now asks the origin for a complete by-xid verdict. 落点: heapam_visibility.c HC181, reusing cluster_runtime_visibility_try_ resolve_remote (the SAME wire + D2 covers SCN-domain gate as the shipped recycled-slot ask, 6.12i) -- res has already returned from cluster_visibility_resolve_tuple_scn, so no new lock is held across the wire (same buffer-content-lock context, lighter than the recycled leg which asks from inside resolve; 执行条件 1 confirmed). positive proof only: COMMITTED -> decide_by_scn -> visible (then the remote deleter/xmax side is still checked, 4.5a G6); ABORTED -> invisible; any refuse / UNKNOWN / covers-miss falls through to the unchanged 53R97 (Rule 8.A, direction unchanged). This is reader-side only (does not touch the cr_server serve path), so it does not wait on 7.2-D3. D5: vis53r97_leg_xmin_overlay_verdict_ask / _hit counters. ask far exceeding the distinct-missed-xid count over a census window is the same-xid thundering-herd signal (N backends miss one hot xid -> N asks; singleflight is a 7.2/7.3 concern, D5 only makes it observable -- 执行 条件 2). Verified (t/994 scratch probe): with the V4 hint ENABLED the ask never fires and node1's cross-node reads are all exact (fallback safe); with the hint DISABLED (deterministic overlay-miss) 212 asks all hit and every read is a SELF-CONSISTENT PREFIX -- cnt <= expected (no over-read / false-visible) and sum == cnt*(cnt+1)/2 (no corruption / lost row). A lagging-but-consistent prefix is正确 MVCC (node0's newest commit not yet propagated), the exact opposite of the C.1 both-wrong-and-self-consistent corruption signature. Local gates: build + unit 14/14 + 11/11 + format + scn-cmp-gate + comment headers + t/346 39/39 no-regress + t/994 6/6. Spec: spec-7.1-cross-instance-positive-interread.md (D1 / IN-9)
…cycle watermark cluster_itl_stamp_multixact_marker could evict a completed DATA slot (wrap++ path) without folding the evicted write_scn into the page's itl_recycle_watermark_scn, unlike the data-writer recycle in cluster_itl_stamp_active. The dropped undo anchor left the per-page CR candidate set silently incomplete, so own-instance CR could reconstruct a post-snapshot version instead of failing closed (latent since the spec-3.6 marker landed; triggered when the page ITL is full and the marker allocator falls back to a reusable data slot). Fold via the shared cluster_itl_recycle_watermark_contribution helper before overwriting the slot, passing new_xid = InvalidTransactionId: the new occupant is a MultiXactId, not a data writer, so the helper's same-xid reuse exemption cannot mis-fire on a numeric xid/multixact-id collision. No redo change: the marker write never travels as an ITL delta (heap_lock_tuple keeps cluster_did_lock_stamp false for the marker branch), so the eviction and the fold move atomically with the page image and the redo-side ACTIVE-only fold stays exact. Tests (cluster_unit test_cluster_itl_reader_real_triple, TDD red->green): T33 marker reuse of a completed DATA slot folds the watermark T34 marker reuse of a completed lock-only slot contributes nothing T35 marker into a FREE slot contributes nothing T36 numeric xid/multixact-id collision must not suppress the fold Spec: spec-7.1-cross-instance-positive-interread.md
… gap-walk
Give a cross-node reader a way to attribute a foreign updater-bearing
multixact xmax to its origin node, replacing the silent native-SLRU
alias (spec-7.1 census probe F) with a fail-closed floor plus positive
resolution where it is provable.
Substrate:
- New pure derivation layer cluster_mxid_stripe.{c,h}: node slot k
issues multixact ids congruent to k (mod 16) at or above a durable
activation floor, so a raw id above the floor is self-describing.
PostgreSQL has no FullMultiXactId, so derivation is valid only in
the half-space [floor, floor + 2^31) where the modular distance is
unambiguous; the allocator refuses (SQLSTATE 53RB4) past that
boundary. Unit test_cluster_mxid_stripe (17 truth-table cases).
- Durable "PGXM" extension record rides the region-6 xid activation
slot at a fixed offset, written atomically in the same 512-byte
slot as the "PGXA" record; a pre-extension slot reads back
all-zeros and the mxid face stays inactive (fail-closed) while the
xid face is unchanged.
- GetNewMultiXactId issues the striped candidate when latched and
walks every skipped offsets-SLRU page before use (the varsup.c
D5e gap-walk verified independently for the ZeroMultiXactOffsetPage
path; the walk starts past nextMXact so it never re-zeroes a live
page). Guardrail + underivable-read counters exposed under the
xid_stripe dump category.
Reader (HeapTupleSatisfiesMVCC only):
- Derive the multixact origin from its striped value instead of the
D7b page marker (which stamps the reader's own id). Derived-own
multis decode natively (alias-free above the floor); derived-
foreign multis resolve through the member overlay when it hits,
else fail closed 53R9C; underivable fails closed. Gated by
cluster.multi_xmax_remote_resolve (default on; off is the D3-0
floor verbatim).
- The proactive V4 overlay does not cover updater-bearing multis
(the updater has no TT binding at heap_update compose time), so
cross-node positive resolution of those is served later by the
member-serve path; until then the reader fails closed on the miss.
Tests: t/359 (activation floor, class-0 and non-0 page-boundary
crossings, restart, 53RB4 injection, overlay-miss floor); t/357
turned to the honest D3-a state (foreign fail-closed, derived-own
native positive); t/015 + t/017 injection-count baselines.
Spec: spec-7.1-cross-instance-positive-interread.md
…-7.1 D3-a) D3-a extended cluster_debug.c's cluster_dump_state() to emit the striped multixact origin-derivation observability counters (cluster_multixact_get_mxid_halfspace_refuse_count and cluster_multixact_get_mxid_underivable_read_count), which live in cluster_multixact.o. The test_cluster_debug unit target links only cluster_debug.o and stubs every backend counter accessor cluster_dump_state references; these two new accessors were left un-stubbed, so a clean `make -C src/test/cluster_unit` failed to link test_cluster_debug with undefined symbols. Add the two missing return-0 stubs following the existing dump-state stub pattern (targeted unit gates build individual targets and so did not surface this in a full-suite clean build). Spec: spec-7.1-cross-instance-positive-interread.md
Spec: spec-7.1-cross-instance-positive-interread.md The MULTI-phase census driver snapshotted only the 'cr' dump category; D3-a's foreign-multi fail-closed legs are counted under the 'xid_stripe' category (mxid_stripe_halfspace_refusals / _underivable_reads / activated_floor). Snapshot them too so the multixact-leg attribution is complete (foreign updater-multi overlay-miss vs allocator half-space vs underivable are now separable in one run).
Rebase-resolution fix. When folding spec-7.1 onto the stage7-72 (spec-7.2 + P0) base, the cluster_unit Makefile conflict was resolved by taking the base 'ours' side (which carries test_cluster_xid_authority from the xid P0 lane) plus the filter-out entry, but that dropped the D3-a additions to TESTS and the standalone test_cluster_mxid_stripe link rule -- so the binary silently never built or ran (160 vs 161). Restore both; cluster_unit is 161/161 with test_cluster_mxid_stripe exercising the PGXM activation-slot record + half-space guardrail. Spec: spec-7.1
…ositive ABORTED via CLOG The origin LMS verdict serve refused every durable XID_MATCH_INVALID_SCN hit (the delayed-cleanout window) with 53R97. Per IN-5 the real population of that leg is aborted-unstamped -- an abort writes no durable commit_scn -- so cross-checking CLOG (authoritative for our own xid) lets a provably-ABORTED xid answer positively (invisible at the requester) instead of failing closed. Mirrors the shipped RECYCLED_ZERO_MATCH CLOG-abort leg in the same function. 8.A (positive proof only): ONLY an explicit CLOG abort upgrades; a committed-but-unstamped, in-flight, 2PC-prepared or crashed-without-abort xid stays fail-closed (we never fabricate a commit_scn). The decision is the pure, unit-tested cluster_cr_server_invalid_scn_verdict. D5: new vis53r97_leg_live_upgrade_hit_count (invalid_scn_refuse is the miss counterpart) + dump row + test stub. Tests: test_cluster_cr_server_policy 7->9 (aborted->ABORTED / not-aborted->REFUSE, the 8.A tooth); t/995 8.A-safety e2e (aborted INSERT poison never visible cross-node). The INVALID_SCN leg is census-cold (three-phase zero, IN-5) so its positive trigger is verified at the serve-function unit level rather than a flaky cold-leg TAP. cluster_unit 161/161, clang-format-18 clean. Spec: spec-7.1-cross-instance-positive-interread.md (D1 serve)
…as inverted cluster_multixact_resolve_visibility's COMMITTED/CLEANED_OUT updater branch had an inline decide_by_scn compare with INVERTED polarity vs its own header truth table, the single-xmax path (cluster_vis_cr_xmax_verdict), and MVCC first principles: it returned INVISIBLE when the updater committed AFTER the snapshot (commit_scn > read_scn) and VISIBLE when it committed BEFORE (commit_scn <= read_scn). Correct is the reverse -- a committed delete that is visible at read_scn hides the tuple. Latent, not yet a live bug: pre-spec-7.1-D3-b a foreign updater-multi always missed the member overlay and fell to the D3-a fail-closed floor (53R9C), so this branch never saw a real committed updater terminal. t/212 only covers lock-only / aborted / in-progress (all VISIBLE), never the committed-updater scn polarity, so nothing caught it. D3-b's member serve is the first path that feeds this branch a live committed updater -> the inversion would become a false-visible / false-invisible P0 (Rule 8.A). Fix: route the committed-updater decision through cluster_vis_cr_xmax_verdict -- the polarity SSOT already shared by the single-xmax path and unit-tested correctly (test_cluster_visibility_variants test_obs_cr_xmax_full_table: COMMITTED+decide-VISIBLE -> CVV_INVISIBLE, COMMITTED+decide-INVISIBLE -> CVV_VISIBLE). The ABORTED / IN_PROGRESS / SUBCOMMITTED / other branches are unchanged. Also lands the spec-7.1 D3-b served-verdict resolver (cluster_multixact_resolve_visibility_served) on the SAME SSOT from the start (instead of mirroring the buggy inline body): it maps each origin-served member terminal to (ClusterTTStatus, scn_decision) and defers polarity to cluster_vis_cr_xmax_verdict. Committed-updater two legs added in test_cluster_multixact_served (before-snapshot -> INVISIBLE / after-snapshot -> VISIBLE). cluster_unit 162/162, test_cluster_multixact 16/16, test_cluster_multixact_ served 14/14, test_cluster_visibility_variants 7/7, t/212 PASS, clang-format-18 clean. Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, A1 hotfix)
…dict page
Add the wire contract for the origin-served multixact member verdict a
requester asks for when a foreign multixact xmax structurally misses its
member overlay (the updater has no compose-time TT binding, IN-12), so the
serve (D3b-2) and requester (D3b-3) that produce/consume it land on a frozen
ABI.
- New FORWARD request kind: reserved_0[6] value 3 (multiplexed alongside
1=undo-fetch, 2=xid-verdict) + Set/Is inline accessors. The asked-for
MXID rides the widened watermark carrier (MultiXactId is 32-bit like
TransactionId, upper 32 bits zero).
- New reply status GCS_BLOCK_REPLY_UNDO_MULTI_VERDICT_RESULT = 20 (tail
value, StaticAssert-pinned) that ships a page + auth trailer exactly like
the single verdict.
- ClusterGcsUndoMultiVerdictPage: 24B header (magic "PGMV" / version /
mxid_echo / nmembers / status) + members[] of 24B ClusterGcsUndoMulti-
VerdictMember (commit_scn / horizon_scn / xid / wrap / verdict /
member_status); cap 256 => 6168B, StaticAssert fits BLCKSZ.
- cluster_vis_undo_multi_verdict_page_usable: pure structural validator
mirroring the single-verdict one -- SERVED status, magic/version/mxid
echo, reserved-zero, per-member verdict/scn consistency (lock-only carry
none; updaters carry a known kind), any inconsistency refuses the whole
page (Rule 8.A: never a partial-proof multi).
Only the origin ships a page and only when every updater member is proven;
an unprovable multi is a DENIED reply, keeping the requester's 53R97.
TDD: RED->GREEN unit truth table in test_cluster_runtime_visibility
(test_undo_multi_verdict_page_usable, batch boundaries 1/2/MAX/MAX+1).
cluster_unit 162/162; full cluster backend + all test binaries build clean;
clang-format-18 clean.
Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, A1/A3, Q-D3b1)
The origin -- the sole owner of a multixact's pg_multixact members -- serves a
batched per-member verdict for a foreign multi a requester cannot resolve
locally (its member overlay structurally missed; IN-12). Wired end to end on
the origin side; the requester ask that triggers it is D3b-3.
- lms_undo_multi_verdict_serve: gate cluster_mxid_is_mine, co-sample the live
authority triple, GetMultiXactIdMembers over our own pg_multixact, and for
each UPDATER member (status 4-5) resolve its terminal; lock-only members
(A2) record {xid,status} only. A member set < 2 / over wire capacity, a
foreign-or-underivable updater xid, or any unprovable updater terminal
refuses the WHOLE multi (8.A -> DENIED, requester keeps 53R97; the residue
is the feature #119 forward). Runs under the drain's PG_TRY.
- Extract the shared lms_resolve_own_xid_verdict core from the single-xid
verdict serve so the RESOLVED_SCN / RECYCLED_ZERO_MATCH / INVALID_SCN
terminal decision lives in ONE place (mini-plan no-fork); it bumps no
counter so each serve owns its census. lms_undo_verdict_serve is
refactored onto it with byte-identical behavior + the same per-reason
census legs.
- Submit (cluster_lms_undo_multi_verdict_submit; MXID rides the carrier,
StaticAssert sizeof(MultiXactId)==sizeof(TransactionId)), LMON dispatch
branch, drain 3rd kind, ship path, and reply-receive parse/auth extended
for GCS_BLOCK_REPLY_UNDO_MULTI_VERDICT_RESULT via the DRY
GcsBlockReplyStatusCarriesUndoAuthTrailer predicate.
- Origin census: cr_server_multi_verdict_served/denied_count (field + init +
bump + accessor + dump row + test_cluster_debug stubs, per the D3-a
clean-build lesson).
cluster_unit 162/162; full cluster backend links + all test binaries build
clean; clang-format-18 clean.
Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, A1/A2, Q-D3b1/2/3)
…ring
Close the cross-node path: a foreign multixact xmax whose member overlay
structurally misses now asks the origin for a batched member verdict and feeds
the served terminals to the pure combination resolver, instead of failing
closed on the miss.
- cluster_gcs_block_undo_multi_verdict_fetch_and_wait: requester fetch
mirroring the single-verdict fetch (MXID rides the carrier, placeholder
synthetic tag since the member scan is complete), waits for
UNDO_MULTI_VERDICT_RESULT, structurally validates via
cluster_vis_undo_multi_verdict_page_usable and Lamport-observes every
member SCN that crossed the wire (AD-008). Any timeout / DENIED /
checksum / non-SERVED / malformed page fails closed (53R97).
- cluster_multixact_remote_xmax_resolve overlay-miss branch: was UNKNOWN,
now calls the new cluster_multixact_remote_xmax_ask_origin helper -> fetch
-> map the served page to ClusterMultiXactServedMember[] ->
cluster_multixact_resolve_visibility_served(read_scn). A resolved
VISIBLE/INVISIBLE is returned to the caller; UNKNOWN keeps the unchanged
53R97 (fail-closed direction untouched). GUC off / origin==self never
asks (D3-a floor).
- Requester census: vis53r97_leg_multi_member_serve_ask/hit (ask = every
request; hit = a definite decision; unprovable = ask - hit = the 53R97
residue / feature #119 forward). Field + init + note + accessor + decl +
dump row + test_cluster_debug stubs.
cluster_unit 162/162; full backend links + all test binaries build clean;
clang-format-18 clean. End-to-end TAP (t/357 L2/L3 转正) is D3b-5.
Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, A1, §2.3)
D3-b makes foreign updater-multi cross-node reads resolve positively via the
origin member serve, so the TAP legs that asserted the pre-D3-b fail-closed
floor now assert the positive read (else they would fail against the new
behavior).
- t/357 L2/L3: was "remote read fails closed 53R9C"; now read_converge to
the committed update (11|100) -- node1 asks the origin for the member
verdict, resolves the updater terminal against its own snapshot, and
converges once its clock observes the origin commit_scn. A lagging
snapshot may briefly still see the pre-update row (correct MVCC, not a
false-visible); convergence proves the positive path AND the absence of a
PERSISTENT superseded read (the probe-F P0 guard).
- t/359 G5: the overlay cap no longer fails closed -- the origin member serve
reads its own pg_multixact directly, independent of the requester overlay
(which never covers updater multis anyway, IN-12); now asserts the
positive converge (17|500).
- L5 (t/357) keeps the requester-ask-off fail-closed floor.
Verified on the 2-node harness: t/357 4x PASS (18/18), t/359 PASS (19/19),
t/346 PASS (39/39, refactored single-verdict serve is behavior-preserving),
t/212 PASS (14/14).
Follow-ups (D3b-5 remainder): dedicated origin-refuse e2e negative leg
(cluster-lms-undo-fetch is PGC_POSTMASTER, must be boot-armed -- t/346 idiom),
crash matrix, L6 native readback, full S4-S7 census after (D4).
Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, D3b-5)
Prove the FAIL-CLOSED direction of the D3-b member-serve path: node1 asks the
origin for a foreign updater-multi verdict and the origin DECLINES to serve, so
the read keeps 53R9C and never returns the superseded row.
Distinct from t/357 L5 (requester ask GUC off -> node1 never asks): here node1
DOES ask and the ORIGIN refuses. The refusal is forced by turning
cluster.crossnode_runtime_visibility off on node0 only (asymmetric, the t/346
L4 idiom -- a PGC_SIGHUP reload, unlike the PGC_POSTMASTER injection point),
which makes node0's LMS submit gate reply an immediate DENIED. This exercises
the requester fetch's DENIED handling and the reader's UNKNOWN -> 53R9C.
- node1 read fails closed with a clean cluster message (the origin refuses
both the member-xmax ask and the new version's xmin verdict ask), never a
raw native multixact id-space error, never the superseded 3|0 row.
- Control: node0 (the creator, derived-own) still reads 3|900 natively,
proving the multi is well-formed and the fail-closed is the peer-serve path.
Verified: 3x PASS (10/10) on the 2-node harness.
Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, negative leg)
…ounters Add the D3-b requester/origin counters to the scratch census driver's key set so CENSUS_PHASE=MULTI attributes the multixact fail-closed legs before/after: vis53r97_leg_multi_member_serve_ask/hit + cr_server_multi_verdict_served/denied. CENSUS_PHASE=MULTI result (this build): probes B/E/F all resolve positively to the correct v=100 (probe F -- node1 owning aliasing multi ids -- is the pre-D0 silent-wrong direction, now correct AND positive). Requester ask=hit=5 (unprovable residue 0), origin served=5 denied=0; the pre-D3-b 5x vis53r97_leg_multi_unresolvable are gone (transferred to member_serve_hit). The section-3.7 before->after attribution closes exactly. Spec: spec-7.1-cross-instance-positive-interread.md (D3-b, D4)
…opy (review P2) The D3-b multi-verdict consume in cluster_gcs_block_undo_multi_verdict_ fetch_and_wait ran the checksum, cluster_vis_undo_multi_verdict_page_usable validation, and the per-member Lamport-observe loop directly on the volatile reply slot (slot->reply_block_data), then memcpy'd it to the caller's page. The consume holds no blk->lock and the reply-ingest handler has no !reply_received guard, so a spec-2.34 retransmit landing in that window could overwrite the slot after it validated -- handing a torn, unvalidated nmembers (up to 65535) to the downstream variable-length member loop and reading past the caller's BLCKSZ page (OOB read; the torn member verdicts could also feed the resolver). This is the first variable-length wire-count consumer, so it is the first place the pre-existing unlocked-consume race becomes an OOB. Validate the STABLE local copy instead: memcpy first, then checksum/validate/ observe on page_out. A torn copy fails the checksum (block-then-header read order), and nmembers is bounded to [2, MAX] before any member is dereferenced. Add a belt re-check of the [2, MAX] bound in cluster_multixact_remote_xmax_ ask_origin, and tighten the validator floor from == 0 to < 2 to match the wire NO_MEMBERS contract (a real multi has >= 2 members) so the loop is provably bounded. Rule 15/16. Spec: spec-7.1-cross-instance-positive-interread.md (D3-b)
…D upgrade (review) lms_undo_multi_verdict_serve treated LMS_OWN_XID_PROVEN_UPGRADE (the D1-serve CLOG-abort upgrade) identically to LMS_OWN_XID_PROVEN and did not bump cluster_vis53r97_note_live_upgrade_hit(), whereas the single-verdict serve does. The verdict is correct (ABORTED); this only restores census parity so a multixact-member upgrade is reflected in the counter. Spec: spec-7.1-cross-instance-positive-interread.md (D3-b)
…ols to main's authority_scn/read_scn model
The spec-7.1 -> main rebase drops D2 (covers->SCN) as fully superseded by main's
7.1a covers migration: authority_scn / demand_scn=read_scn is strictly sounder for
read_scn consumption than 7.1's page_block_scn gate (when read_scn > page_block_scn
the page-anchored gate could leave a commit window unscanned; demand=read_scn closes
exactly that). Later D1/D3-b commits, auto-merged without conflict, still referenced
D2's parallel symbols; retarget them to main's names so the integrated tree compiles
and keeps main's proven covers semantics:
- cluster_cr_server.c multi-verdict serve co-sample: live_hwm_scn -> authority_scn
- cluster_gcs_block.c multi-verdict reply consume: reply_undo_live_hwm_scn ->
reply_undo_authority_scn (the unified undo-trailer reception parse already
populates reply_undo_authority_scn for any undo-trailer reply, incl the
multi-verdict, so the value is present and correct)
- heapam_visibility.c D1-req xmin overlay-miss: drop the extra page_block_scn arg
to cluster_runtime_visibility_try_resolve_remote (main's 7-arg fn demands the
snapshot read_scn), and refresh the stale (D2) covers comment
Locally verified: build (cassert), cluster_unit 162/162, PG core regress 219/219,
clang-format 0, comment-headers 0, TAP t/212 t/346 t/357 t/359 t/360 t/365.
Spec: spec-7.1-cross-instance-positive-interread.md (main integration)
…pec-1.15 Q8/L4) The multixact polarity-hotfix banner described the read_scn ordering as "(commit_scn <= read_scn)"; the fast-gate scn-cmp-gate regex flags any _scn identifier followed by a comparison operator, including inside comments (its comment-exclusion pattern expects a filename:line prefix that grep -n does not emit). Reword to "(commit_scn at/before read_scn)" -- no operator, identical meaning, zero behavior change. Surfaced now because this is the branch's first fast-gate run. Spec: spec-7.1-cross-instance-positive-interread.md (main integration)
…62 (D3-a mxid-halfspace) t/030 M1/M5 still asserted 161 injection points (161x2=322 sub-keys). The spec-7.1 D3-a mxid-stripe deliverable registered cluster-mxid-halfspace-hard-limit, taking the registry to 162; t/015 and t/017 were already bumped to 162 by that work, but t/030 was the missed straggler (surfaced on the branch's first fast-gate run). Bump M1 161->162 and M5 322->324 and add the spec-7.1 D3-a entry, matching the live registry and the t/015 / t/017 descriptive lists. Spec: spec-7.1-cross-instance-positive-interread.md (main integration)
sqlrush
force-pushed
the
spec-7.1-positive-interread
branch
from
9676855 to
7845b6b
Compare
added 4 commits
July 10, 2026 09:13
… test wiring The rebase's cluster_unit Makefile conflict resolution (checkout --ours + re-add to the filter-out list) silently dropped test_cluster_multixact_served from TESTS and lost its explicit link rule -- the same failure mode the earlier test_cluster_mxid_stripe restore fixed. The 14 polarity / 8.A truth-table cases for the pure served-verdict combination resolver were therefore not built or run by `make check` (the reported 162/162 did not include them). Restore the TESTS entry and the standalone link rule (cluster_multixact_served.o + cluster_visibility_verdict.o) verbatim from the pre-rebase branch. Verified: the binary builds and passes 14/14; full cluster_unit check now runs 163 binaries. Spec: spec-7.1-cross-instance-positive-interread.md (integration review F4, P1)
… before publishing PENDING Every LMS work-slot submitter (CR construct, undo-TT fetch, undo verdict, and the new undo multi-verdict) CAS'd the slot directly FREE -> PENDING and only then wrote the request fields, with a redundant trailing PENDING store after the write barrier. The LMS drain CAS-acquires PENDING slots, so a drain racing the submitter could (a) serve a half-written request and (b) have its BUSY/READY overwritten by the submitter's trailing store -- mixed requests, double serves, lost replies, or spurious timeouts. Introduce the producer-only CLUSTER_LMS_CR_FILLING state: the submit CAS lands on FILLING (invisible to the drain), the request fields are written, and PENDING is published once, after the existing write barrier -- pairing with the drain's read barrier. A producer killed between reserve and publish leaks the slot in FILLING, which is fail-closed (submits degrade to false; the requester keeps 53R97), never a garbage serve. All four submitters unified. Spec: spec-7.1-cross-instance-positive-interread.md (integration review F3, P1)
…t consume on the covers window The D3-b requester consumed the origin's served member terminals without running the co-sampled live authority through the covers gate, unlike the single-xid verdict leg (rtvis_try_origin_verdict). An epoch-stale reply (origin remastered or fenced after sampling, D-i3) or an origin authority clock behind the snapshot read_scn (an updater could still commit after the member scan with a commit_scn at/below read_scn) was consumed as conclusive -- on the polarity-sensitive xmax path that can hide or expose the wrong tuple version. Run cluster_vis_live_authority_covers(snap->read_scn, auth) before mapping the members, exactly like the single-verdict consumer: a refusal returns UNKNOWN (the caller keeps the 53R97 fail-closed boundary) and bumps both covers-refuse census counters (the spec-7.1a D6 SCN-refuse counter + the spec-7.1 D0 per-leg counter). The fetch already Lamport-observed every shipped SCN, so a refusal self-heals on the next snapshot. Spec: spec-7.1-cross-instance-positive-interread.md (integration review F1, P0)
…on conflicting "PGXM" evidence
The boot scan treated an absent OR corrupt "PGXM" mxid activation extension as
mxid_floor = 0 while the PGXA xid face stood, and adopted whichever floor rode
the newest-generation PGXA copy with no cross-copy consistency check and no
regression guard. A node (or a freshly latched backend) could therefore fall
back to vanilla DENSE multixact allocation while other nodes stayed striped: a
dense mxid landing at/above another node's latched floor is misattributed by the
value-mod-16 origin derivation and served from the wrong node's pg_multixact --
the probe-F silent-wrong class reopened.
Fail-closed rework:
- stripe_read_activation_one classifies the extension tri-state: ABSENT
(all-zero area, pre-extension binary), VALID (CRC'd record + floor), or
CORRUPT (non-zero bytes failing validation).
- cluster_mxid_stripe_resolve_floor (new, pure, unit-tested) folds the
per-copy evidence: two valid floors disagreeing -> POISONED; a valid
extension anywhere with none at the newest generation -> POISONED
(activation can never regress to absent); corrupt bytes with no valid copy
-> POISONED ("never activated" unprovable); all-absent -> ABSENT; otherwise
the single coherent floor.
- StripeBootShmem gains mxid_disk_state {UNKNOWN,ABSENT,PUBLISHED,POISONED}
with sticky/monotonic publication: POISONED never clears, a PUBLISHED floor
never moves and never regresses to absent (an unreadable-evidence scan keeps
the published floor and logs).
- The lazy runtime latch latches POISONED dominantly: derivation stays
underivable (foreign reads keep their 53R9C fail-closed boundary) and
GetNewMultiXactId REFUSES allocation (ERRCODE_DATA_CORRUPTED with detail +
hint) instead of degrading to the dense allocator. A later active latch
cannot override poison.
- Observability: xid_stripe dump gains mxid_stripe_disk_state; the poison
transition logs a WARNING with the conflicting evidence.
Unit coverage: test_cluster_mxid_stripe grows the resolve truth table (absent /
single-valid / torn-sibling repair / same-gen + cross-gen floor conflict /
newest-gen regression / corrupt-only) and the poisoned-runtime dominance leg
(17 -> 19 cases).
Spec: spec-7.1-cross-instance-positive-interread.md (integration review F2, P0)
Owner
Author
|
Review round 1 (2 P0 + 2 P1) — all four findings fixed, one atomic commit each:
Local gates @ Note (out of this PR's scope): the scn-cmp-gate comment-exclusion regex bug is confirmed and |
…njection baselines
The branch's first nightly run surfaced the stragglers the fast-gate TAP subset
does not cover: three shards red on exactly two stale baselines, no behavioral
failures.
- cr category 57 -> 70: the spec-7.1 census + multi-verdict serve counters
(11 vis53r97_leg_* + 2 cr_server_multi_verdict_*) were not reflected in the
five nightly-only cr-count assertions (t/215 t/216 t/300 t/306 t/311).
- injection registry 161 -> 162: the D3-a cluster-mxid-halfspace-hard-limit
bump reached t/015 t/017 t/030 but not the six foundation assertions
(t/018 t/020 t/021 t/022 t/023 t/024).
All eleven descriptive breakdowns gain the spec-7.1 entry, mirroring the
t/015 / t/017 convention. Verified: all eleven tests pass locally; a full-tree
grep finds no remaining 57/161 count assertions.
Spec: spec-7.1-cross-instance-positive-interread.md (integration; nightly sweep)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates spec-7.1 (cross-instance positive interread, read-side) onto main,
which already carries the spec-7.1a write-side.
spec-7.1-positive-interreadwasrebased onto current main (spec-7.1a + spec-7.2 + prior Stage-7 work); the two
subsystems share the origin verdict-serve, the live-authority covers gate, and the
multixact visibility paths, so the overlapping code is reconciled to a single form.
Scope (21 replayed commits — see commit log)
verdict SSOT (
cluster_vis_cr_xmax_verdict).(
lms_resolve_own_xid_verdict); the covers gate uses the origin authority SCNagainst the snapshot read_scn.
Local verification (@ a128fe9)
--enable-cluster, cassert): cleant/212 t/346 t/357 t/359 t/360 t/365all passDRAFT — review before merge
This branch reconciles two correctness-sensitive visibility/serve paths. Held as a
draft for code review; do not merge until review passes and it is marked ready.
Spec: spec-7.1-cross-instance-positive-interread.md