Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -4132,10 +4132,10 @@ vacuum_lock_privilege vacuum_lock_privilege truncate refuses a non-owner before
vacuum_lock_privilege vacuum_lock_privilege truncate tells a non-owner nothing about the relation type (#1384) 15;16;17;18;19 2026-10-14 truncate checking ownership after its lock and its type check (main before #1384's fix)
vacuum_lock_privilege vacuum_lock_privilege vacuum refuses a non-owner before taking the exclusive lock 15;16;17;18;19 2026-10-14 every maintenance entry point checks ownership after taking its lock;pgcolumnar.vacuum checks ownership after taking its AccessExclusiveLock
vacuum_lock_privilege vacuum_lock_privilege vacuum_sorted refuses a non-owner before taking the exclusive lock 15;16;17;18;19 2026-10-14 every maintenance entry point checks ownership after taking its lock
validity_elision validity_elision a chunk claiming no bitmap while it holds fewer values than rows is refused 15;16;17;18;19 never -
validity_elision validity_elision a chunk claiming no bitmap while it holds fewer values than rows is refused 15;16;17;18;19 2026-10-16 the no-nulls guard (claims no nulls but does not account for every row) disabled, so a later guard raised the XX001 (mutation, #1339)
validity_elision validity_elision a column with no nulls stores no validity bitmap 15;16;17;18;19 2026-09-18 writer: flush_one_column's presentCount == rowCount forced false, so the bitmap is written even when the chunk holds no null
validity_elision validity_elision a null-free column elides its bitmap beside a null-bearing one in the same row group 15;16;17;18;19 never -
validity_elision validity_elision an implausible row count on an elided chunk is refused before anything is allocated 15;16;17;18;19 never -
validity_elision validity_elision an implausible row count on an elided chunk is refused before anything is allocated 15;16;17;18;19 2026-10-16 the no-nulls guard (claims no nulls but does not account for every row) disabled, so a later guard raised the XX001 (mutation, #1339)
validity_elision validity_elision and the backend survives that refusal 15;16;17;18;19 never -
validity_elision validity_elision and the backend survives that refusal too 15;16;17;18;19 never -
validity_elision validity_elision premise: each fixture is a single row group, which is what the expected size assumes 15;16;17;18;19 never -
Expand Down
12 changes: 11 additions & 1 deletion test/check_ledger_budget.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3320,4 +3320,14 @@ suites_not_covered 199
# never 2980 + 10 = 2990
# dated 1191 + 8 = 1199
# 2990 + 1199 == 4189
checks_never_observed_red 2990
# #1339, validity_elision's two guard arms ask which guard refused (jdatcmd): 2990 ->
# 2988. "a chunk claiming no bitmap while it holds fewer values than rows is refused"
# and "an implausible row count on an elided chunk is refused before anything is
# allocated" are dated by disabling the no-nulls guard, after which a later guard
# raised the same XX001. No row is added, removed or renamed.
#
# ledger rows 4189 + 0 = 4189
# never 2990 - 2 = 2988
# dated 1199 + 2 = 1201
# 2988 + 1201 == 4189
checks_never_observed_red 2988
55 changes: 40 additions & 15 deletions test/pytest/test_validity_elision.py
Original file line number Diff line number Diff line change
Expand Up @@ -250,16 +250,27 @@ def test_the_layout_change_returns_the_same_rows(pgc_conn, expect):
)


def _guard(exc, phrase, label):
"""`SQLSTATE|label` when the error is the guard whose message holds `phrase`;
otherwise `SQLSTATE|another guard: <message>`, so a red names what fired."""
if exc is None:
return "NOERR"
msg = getattr(getattr(exc, "diag", None), "message_primary", None) or str(exc)
state = getattr(exc, "sqlstate", None) or "none"
return f"{state}|{label}" if phrase in msg else f"{state}|another guard: {msg}"


def test_the_guards_the_elision_added_refuse_a_lying_catalog(pgc_conn, expect):
"""The two guards, each corrupted into firing.

Neither is reachable from a table written correctly, so the catalog is
poisoned the way corruption.sh poisons the fields that predate this change.

THE SQLSTATE IS THE ASSERTION, not the message. `XX001` is
ERRCODE_DATA_CORRUPTED and comes only from a guard that ran; a message grep
is equally satisfied by a missing function, a bad argument or a login
failure.
THE SQLSTATE AND THE GUARD'S OWN MESSAGE. `XX001` is ERRCODE_DATA_CORRUPTED
and comes only from a guard that ran; a message grep alone is equally
satisfied by a missing function, a bad argument or a login failure. But
sixteen guards in the reader raise XX001, so the SQLSTATE alone cannot say
WHICH ran (#1339): `_guard` asks for both.
"""
import psycopg

Expand All @@ -285,16 +296,26 @@ def test_the_guards_the_elision_added_refuse_a_lying_catalog(pgc_conn, expect):
raised = None
except psycopg.Error as exc:
raised = exc
expect.sqlstate(
raised, "XX001",
"a chunk claiming no bitmap while it holds fewer values than rows is refused",
)
# WHICH GUARD, not only that one ran (#1339). XX001 comes from sixteen guards in
# the reader: with this one disabled, the scan still raised XX001 and the arm
# stayed green. So the message must be this guard's own.
expect.text(_guard(raised, "claims no nulls but does not account for every row",
"the no-nulls guard"),
"XX001|the no-nulls guard",
"a chunk claiming no bitmap while it holds fewer values than rows is refused")
expect.num(_count(c, "SELECT 1"), 1, "and the backend survives that refusal")

# GUARD 2: a row count the chunk cannot hold. The descriptor still
# accounts for its own rows, so the claim is false in the other
# direction and the reader refuses BEFORE synthesizing half a gigabyte
# of bits.
# direction and the reader refuses BEFORE synthesizing any bits.
#
# NINE BILLION, NOT FOUR, SO THE ORDER IS VISIBLE (#1339 review): 1.125 GB
# of bits is over MaxAllocSize, so the "implausible row count" backstop sits
# between the no-nulls guard and the allocation, and a no-nulls guard moved
# below the allocation is met by the backstop first. At four billion (500 MB)
# that move stayed green. A guard placed between the backstop and the
# allocation is still unseen. A reworded guard message reddens both arms in
# both twins, which must then be updated together.
#
# BOUNDING THE SYNTHESIZED BITMAP BY THE ROW GROUP'S BYTE LENGTH DOES
# NOT WORK: an elided group of two well-encoded bigint columns measured
Expand All @@ -303,7 +324,7 @@ def test_the_guards_the_elision_added_refuse_a_lying_catalog(pgc_conn, expect):
c.execute(
"""
UPDATE pgcolumnar.row_group r
SET row_count = 4000000000
SET row_count = 9000000000
FROM pgcolumnar.storage s
WHERE s.storage_id = r.storage_id
AND s.relation_oid = 'vb_full'::regclass
Expand All @@ -314,10 +335,14 @@ def test_the_guards_the_elision_added_refuse_a_lying_catalog(pgc_conn, expect):
raised = None
except psycopg.Error as exc:
raised = exc
expect.sqlstate(
raised, "XX001",
"an implausible row count on an elided chunk is refused before anything is allocated",
)
# Refused BEFORE anything is allocated means by the value-count guard, which
# runs first; the MaxAllocSize check after it is only a backstop. With the
# value-count guard disabled at four billion rows, the bitmap was allocated,
# a later guard raised XX001, and this arm stayed green (#1339).
expect.text(_guard(raised, "claims no nulls but does not account for every row",
"the no-nulls guard"),
"XX001|the no-nulls guard",
"an implausible row count on an elided chunk is refused before anything is allocated")
expect.num(_count(c, "SELECT 1"), 1, "and the backend survives that refusal too")


Expand Down
48 changes: 35 additions & 13 deletions test/validity_elision.sh
Original file line number Diff line number Diff line change
Expand Up @@ -319,18 +319,31 @@ check "single-row fetches of the null-bearing column return its nulls as nulls"
# catalog -- which is what a bit flip or a crafted file does, and what
# corruption.sh does for the fields that predate this change.
#
# ASSERT THE SQLSTATE, NOT THE TEXT. `42501`, `22023`, a login FATAL and a
# ASSERT THE SQLSTATE, NOT ONLY THE TEXT. `42501`, `22023`, a login FATAL and a
# missing function all satisfy a grep for "ERROR"; XX001 is ERRCODE_DATA_CORRUPTED
# and comes only from a guard that ran.
sqlstate() { # SQL -> the 5-char SQLSTATE it raises, or NOERR
local out
#
# AND WHICH GUARD (#1339). XX001 comes from sixteen guards in the reader: with the
# no-nulls guard disabled, both arms below still saw XX001 from a later one and
# stayed green. So each reads the message too, and a red names what fired.
# SQL -> `SQLSTATE|the no-nulls guard`, `SQLSTATE|another guard: <message>` or NOERR.
guard_of() {
local out state msg
out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" \
-U postgres -d "$PGC_DB" -qtA 2>&1 <<SQLEOF
\\set VERBOSITY sqlstate
\\set VERBOSITY verbose
$1;
SQLEOF
)"
printf '%s\n' "$out" | sed -n 's/^ERROR: \([0-9A-Z]\{5\}\).*/\1/p' | head -1 | grep . || echo NOERR
state="$(printf '%s\n' "$out" | sed -n 's/^ERROR: \([0-9A-Z]\{5\}\): .*/\1/p' | head -1)"
msg="$(printf '%s\n' "$out" | sed -n 's/^ERROR: [0-9A-Z]\{5\}: //p' | head -1)"
if [ -z "$state" ]; then
echo NOERR
elif grep -qF "claims no nulls but does not account for every row" <<<"$msg"; then
echo "$state|the no-nulls guard"
else
echo "$state|another guard: $msg"
fi
}

# GUARD 1: a chunk that CLAIMS it stored no bitmap while its own descriptor
Expand All @@ -347,29 +360,38 @@ psql_run "UPDATE pgcolumnar.column_chunk c
AND c.column_index = 1;"

check "a chunk claiming no bitmap while it holds fewer values than rows is refused" \
"$(sqlstate "SELECT sum(v) FROM ve_nulls")" \
"XX001"
"$(guard_of "SELECT sum(v) FROM ve_nulls")" \
"XX001|the no-nulls guard"

check "and the backend survives that refusal" "$(q "SELECT 1;" | tail -1)" "1"

# GUARD 2: a row count the chunk cannot possibly hold. The elided chunk's
# descriptor still accounts for its own 100,000 values, so a row_count of four
# descriptor still accounts for its own 100,000 values, so a row_count of nine
# billion makes the same claim false in the other direction, and the reader
# refuses BEFORE it synthesizes anything -- which is the ordering that matters,
# since the bitmap it would otherwise invent is half a gigabyte.
# refuses BEFORE it synthesizes anything -- which is the ordering that matters.
#
# NINE BILLION, NOT FOUR, SO THE ORDER IS VISIBLE (#1339 review). The bitmap it
# would need is 1.125 GB, over MaxAllocSize, so the "implausible row count"
# backstop sits between the no-nulls guard and the allocation: a no-nulls guard
# moved below the allocation is met by the backstop first and names it. At four
# billion the bitmap was 500 MB, under the backstop, and that move stayed green.
# It still cannot see a guard placed between the backstop and the allocation.
#
# A reworded guard message reddens both arms here and in the pytest twin, which
# must then be updated together: the red prints the real message.
#
# BOUNDING THE SYNTHESIZED BITMAP BY THE ROW GROUP'S BYTE LENGTH DOES NOT WORK,
# and this suite is where that was measured: ve_full's group is 360 bytes on
# disk and would need 12,500 bytes of bits, which is the saving, not a defect.
psql_run "UPDATE pgcolumnar.row_group r
SET row_count = 4000000000
SET row_count = 9000000000
FROM pgcolumnar.storage s
WHERE s.storage_id = r.storage_id
AND s.relation_oid = 've_full'::regclass;"

check "an implausible row count on an elided chunk is refused before anything is allocated" \
"$(sqlstate "SELECT sum(v) FROM ve_full")" \
"XX001"
"$(guard_of "SELECT sum(v) FROM ve_full")" \
"XX001|the no-nulls guard"

check "and the backend survives that refusal too" "$(q "SELECT 1;" | tail -1)" "1"

Expand Down
Loading