Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -657,7 +657,8 @@ other than where the feature is used:
`ON CONFLICT DO NOTHING` takes no row lock and works, with or without a conflict
target.

`CREATE TABLE` refuses an unlogged columnar table for the same reason. Both cases
`CREATE TABLE` refuses an unlogged columnar table for the same reason, and so does
`ALTER TABLE ... SET UNLOGGED`. Both cases
follow one rule. This access method refuses a configuration that it cannot
support at the point where you choose it. It does not refuse it at each later
use.
Expand Down Expand Up @@ -783,7 +784,11 @@ therefore still be cheaper as a scan.
of a deleted row only while a transaction that began before the delete is still
running, as it does on heap.
- `CREATE INDEX CONCURRENTLY` (the concurrent validate path) and partial
block-range index builds are not supported.
block-range index builds are not supported. A refused `CREATE INDEX
CONCURRENTLY` leaves its index behind, marked invalid, as PostgreSQL does for any
concurrent build that fails. Drop that index before you build it again.
- `ALTER TABLE ... SET TABLESPACE` is not supported on a columnar table. It is
refused, and the table stays where it was.

## Constraints on the import path

Expand Down
122 changes: 122 additions & 0 deletions test/advisory_lock_class.sh
Original file line number Diff line number Diff line change
Expand Up @@ -245,4 +245,126 @@ check "row_lock_buckets is fixed at server start, so a session cannot change it
"$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -v VERBOSITY=sqlstate -c "SET pgcolumnar.row_lock_buckets = 7;" 2>&1 | sed -n 's/^.*ERROR:[[:space:]]*\([0-9A-Z]\{5\}\).*$/\1/p')|$(q "SELECT context || '|' || setting FROM pg_settings WHERE name = 'pgcolumnar.row_lock_buckets'")" \
"55P02|postmaster|1024"

# The documented bound (docs/configuration.md): row_lock_buckets "bounds the row locks
# a transaction holds, so a bulk update cannot exhaust the lock table". One transaction
# updates 200000 rows and then counts its own class-104 locks (one -c string is one
# transaction): one per bucket, not one per row. A row lock per row would not fit the
# default lock table at all.
psql_run "CREATE TABLE rlb (id int, v int) USING pgcolumnar;
INSERT INTO rlb SELECT g, g FROM generate_series(1, 200000) g;" >/dev/null
check "a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them" \
"$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "UPDATE rlb SET v = v + 1;
SELECT count(*) || '|' || current_setting('pgcolumnar.row_lock_buckets') FROM pg_locks
WHERE pid = pg_backend_pid() AND locktype = 'advisory' AND objsubid = 104;" 2>&1 |
grep -E '^[0-9]+[|]|ERROR' | tail -1)" "1024|1024"
check "and the UPDATE committed every row" "$(q "SELECT count(*) || '|' || sum(v - id) FROM rlb")" "200000|200000"

# ---- the row-identity lock cannot be reached from SQL (#1341 item 56) ------------
# The INSERT arms above ask this of the unique-key and storage-row locks; nothing
# asked it of class 104. An UPDATE is held open and every advisory lock it took is
# read. None may be in field 1 or 2, the only classes SQL can produce. Then a second
# session takes each of those exact tags with pg_advisory_xact_lock(int4, int4), and
# a columnar UPDATE of the same row must not wait on it. An oid above the int4 range
# is addressed by its negative int4, the same 32 bits lockfuncs.c puts in the tag.
rl_wait_for() { # rl_wait_for APPLICATION_NAME -> its pid, once it has named itself
local i pid=""
for i in $(seq 1 100); do
pid="$(q "SELECT pid FROM pg_stat_activity WHERE application_name = '$1'")"
[ -n "$pid" ] && break
sleep 0.1
done
echo "$pid"
}
rl_end() { # rl_end PID BGPID: end a holder's backend and its psql
q "SELECT pg_terminate_backend($1)" >/dev/null 2>&1
kill "$2" 2>/dev/null; wait "$2" 2>/dev/null
local i
for i in $(seq 1 60); do
[ "$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND pid <> pg_backend_pid()")" = 0 ] && break
sleep 0.2
done
}
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "BEGIN; UPDATE rl SET v = v + 1 WHERE id = 7; SET application_name = 'rl_tagger'; SELECT pg_sleep(30); COMMIT;" \
>/dev/null 2>&1 &
rl_bg=$!
rl_pid="$(rl_wait_for rl_tagger)"
RL_TAGS="$(q "SELECT CASE WHEN classid::bigint > 2147483647 THEN classid::bigint - 4294967296 ELSE classid::bigint END
|| ' ' || CASE WHEN objid::bigint > 2147483647 THEN objid::bigint - 4294967296 ELSE objid::bigint END
|| ' ' || objsubid
FROM pg_locks WHERE locktype = 'advisory' AND granted AND pid = ${rl_pid:-0}")"
rl_end "${rl_pid:-0}" "$rl_bg"
RL_N="$(printf '%s\n' "$RL_TAGS" | grep -c '[0-9]')"
rl_reach=""
while read -r _c _o _f; do
case "${_f:-}" in 1|2) rl_reach="$rl_reach ($_c,$_o,field4=$_f)" ;; esac
done <<<"$RL_TAGS"
check "the locks an UPDATE takes are not in a SQL-reachable class" \
"$(if [ "$RL_N" -eq 0 ]; then echo "unknown (no advisory lock seen; holder pid ${rl_pid:-none})"
elif [ -n "$rl_reach" ]; then echo "reachable:$rl_reach"
else echo unreachable; fi)" "unreachable"
rl_grab="BEGIN;"
while read -r _c _o _f; do
[ -z "${_f:-}" ] && continue
rl_grab="$rl_grab SELECT pg_advisory_xact_lock($_c::int, $_o::int);"
done <<<"$RL_TAGS"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "$rl_grab SET application_name = 'rl_user'; SELECT pg_sleep(30); COMMIT;" >/dev/null 2>&1 &
rl_bg=$!
rl_pid="$(rl_wait_for rl_user)"
check "premise: the other session holds every one of those tags through the SQL functions" \
"$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND granted AND objsubid = 2
AND pid = ${rl_pid:-0}")" "$RL_N"
rl_upd="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "SET statement_timeout = '10s';" -c "UPDATE rl SET v = v + 1 WHERE id = 7;" 2>&1)"
case "$rl_upd" in
*timeout*|*canceling*) rl_verdict="BLOCKED by the user lock" ;;
*ERROR*) rl_verdict="ERROR: $(head -1 <<<"$rl_upd")" ;;
*) rl_verdict=ok ;;
esac
check "a user advisory lock on those tags does not block a columnar UPDATE" "$rl_verdict" "ok"
rl_end "${rl_pid:-0}" "$rl_bg"

# ---- bulk_parallel_writer set by hand still serialises a first write (#1341 item 55) --
# The setting lets parallel_copy's loaders skip the storage-row creation lock (class
# 102). The reference says setting it by hand is safe, because the skip fires only
# once the storage row is committed. Two sessions with it on each make the first
# write to a committed, empty table: the second must wait for the first and then
# land its row. A skip that did not wait for the row to be committed lets both
# create it, and the second fails on storage_pkey.
#
# The first writer commits only once the suite has looked for the second one waiting
# and then inserted a committed flag row, not after a fixed sleep: a slow second
# writer would otherwise start after the first had committed, and the premise below
# would fail on timing (OffgridwithJD, review of #1415). The pytest twin does the same.
psql_run "CREATE TABLE bw (id int) USING pgcolumnar; CREATE TABLE bw_go (go int) USING heap;"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "SET statement_timeout = '60s'; BEGIN; SET LOCAL pgcolumnar.bulk_parallel_writer = on; INSERT INTO bw VALUES (1); SET application_name = 'bw_first';
DO \$\$ BEGIN WHILE NOT EXISTS (SELECT 1 FROM bw_go) LOOP PERFORM pg_sleep(0.1); END LOOP; END \$\$; COMMIT;" \
>/dev/null 2>&1 &
bw_bg=$!
bw_pid="$(rl_wait_for bw_first)"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "SET pgcolumnar.bulk_parallel_writer = on; SET statement_timeout = '30s';" -c "INSERT INTO bw VALUES (2);" \
>"$PGC_WORKDIR/bw_second.out" 2>&1 &
bw_bg2=$!
# The second write must be waiting on the first before the first commits, or the arm
# measures two writes in sequence.
bw_waited=no
for _i in $(seq 1 100); do
[ "$(q "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' AND query LIKE 'INSERT INTO bw%'")" -ge 1 ] && { bw_waited=yes; break; }
sleep 0.1
done
psql_run "INSERT INTO bw_go VALUES (1);" # release the first writer, whatever the poll saw
check "premise: the second first write waited on the first" "$bw_waited" "yes"
wait "$bw_bg2"; wait "$bw_bg"
bw_out="$(cat "$PGC_WORKDIR/bw_second.out")"
case "$bw_out" in
*ERROR*) bw_verdict="ERROR: $(grep -m1 ERROR <<<"$bw_out")" ;;
*) bw_verdict=ok ;;
esac
check "with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land" \
"$([ -n "$bw_pid" ] && echo held || echo "no holder")|$bw_verdict|$(q "SELECT count(*) FROM bw")" "held|ok|2"

pgc_summary
19 changes: 19 additions & 0 deletions test/am_contracts.sh
Original file line number Diff line number Diff line change
Expand Up @@ -351,4 +351,23 @@ check "REINDEX after a committed delete of most rows reclaims their entries, as
"$(ix_reclaim pgcolumnar)|$(ix_reclaim heap)|$(q "SET enable_seqscan = off; SELECT count(*) FROM ix_pgcolumnar WHERE a > 0;" | tail -1)" \
"reclaimed|reclaimed|10000"

# ---- documented refusals (#1341 items 44, 51, 77) ----------------------------------
#
# Each is refused where it is chosen, as limitations.md says, and refused cleanly:
# nothing is left changed by the attempt.
rf_count() { q "SELECT count(*) || '|' || sum(a) FROM rf"; }
psql_run "CREATE TABLE rf (a int) USING pgcolumnar; INSERT INTO rf SELECT g FROM generate_series(1, 5000) g;"
check "CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table" \
"$(state "CREATE UNLOGGED TABLE rf_u (a int) USING pgcolumnar")|$(q "SELECT count(*) FROM pg_class WHERE relname = 'rf_u'")" "0A000|0"
check "ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows" \
"$(state "ALTER TABLE rf SET UNLOGGED")|$(q "SELECT relpersistence FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|p|5000|12502500"
pgc_pg "mkdir -p -m 700 '$PGC_WORKDIR/rf_ts'"
psql_run "CREATE TABLESPACE rf_ts LOCATION '$PGC_WORKDIR/rf_ts';"
check "ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows" \
"$(state "ALTER TABLE rf SET TABLESPACE rf_ts")|$(q "SELECT reltablespace FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|0|5000|12502500"
# A failed concurrent build leaves its index behind, marked invalid, as core does for
# any CREATE INDEX CONCURRENTLY that fails after its first phase.
check "CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid" \
"$(state "CREATE INDEX CONCURRENTLY rf_cic ON rf (a)")|$(q "SELECT indisvalid FROM pg_index WHERE indexrelid = 'rf_cic'::regclass")" "0A000|f"

pgc_summary
21 changes: 21 additions & 0 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
advisory_lock_class advisory_lock_class a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row)
advisory_lock_class advisory_lock_class a duplicate key is still rejected 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class a user advisory lock on that tag does not block a columnar insert 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h)
advisory_lock_class advisory_lock_class a user advisory lock on those tags does not block a columnar UPDATE 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class an UPDATE holds a row-identity lock (class 104) while enable_row_update_lock is on 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class and holds none with enable_row_update_lock off 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class and the UPDATE committed every row 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row)
advisory_lock_class advisory_lock_class premise: the discovering session is gone and holds nothing 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the insert took an advisory lock we can see 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the lock is enabled, or nothing below proves anything 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the other session holds every one of those tags through the SQL functions 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the other session really holds that exact tag in class 2 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the second first write waited on the first 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class row_lock_buckets is fixed at server start, so a session cannot change it (55P02) 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class the lock an insert takes is not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h)
advisory_lock_class advisory_lock_class the locks an UPDATE takes are not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land 15;16;17;18;19 2026-10-15 the storage-row creation lock skipped whenever bulk_parallel_writer is on, not only once the row is committed (src/columnar_metadata.c)
alter_table_scans alter_table_scans ADD COLUMN with a volatile default rewrites the table, as on heap 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c)
alter_table_scans alter_table_scans ADD PRIMARY KEY is refused over a duplicate (23505), as on heap 15;16;17;18;19 never -
alter_table_scans alter_table_scans ADD PRIMARY KEY succeeds once the duplicate is deleted, as on heap 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c)
Expand All @@ -13,6 +30,10 @@ alter_table_scans alter_table_scans and the rewrite kept the live rows and not t
alter_table_scans alter_table_scans the deleted rows are gone from both tables, which hold the same rows 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c)
alter_table_scans alter_table_scans the lookup of id 1 is an index scan on the primary key 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c)
alter_table_scans alter_table_scans the primary key still finds id 1 once, through the index, after both rewrites 15;16;17;18;19 never -
am_contracts am_contracts ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows 15;16;17;18;19 2026-10-15 relation copy returning without copying (SET TABLESPACE)
am_contracts am_contracts ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table
am_contracts am_contracts CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid 15;16;17;18;19 2026-10-15 the concurrent index validate scan doing nothing
am_contracts am_contracts CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table
am_contracts am_contracts ON CONFLICT DO NOTHING skips the duplicate and inserts the new row 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
am_contracts am_contracts ON CONFLICT DO UPDATE is refused as unsupported (0A000), and changes nothing 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
am_contracts am_contracts ON CONFLICT DO UPDATE with no conflicting row inserts it 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
Expand Down
35 changes: 33 additions & 2 deletions test/check_ledger_budget.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#
# Adding a check to a suite that is already covered does not move it, which is
# what makes it safe to bound.
suites_not_covered 231
suites_not_covered 230
#
# checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become
# one.
Expand Down Expand Up @@ -2414,4 +2414,35 @@ suites_not_covered 231
# never 2282 + 0 = 2282
# dated 726 + 9 = 735
# 2282 + 735 == 3017
checks_never_observed_red 2282
#
# #1341 items 44, 51 and 77, the documented refusals (jdatcmd): 2282 -> 2282.
# am_contracts gains four checks from five-major runs (31/31 on each of 15-19), all
# dated: CREATE UNLOGGED and SET UNLOGGED by the storage hook accepting an unlogged
# table, CREATE INDEX CONCURRENTLY by its validate scan doing nothing, and SET
# TABLESPACE by the relation copy returning without copying.
#
# ledger rows 3017 + 4 = 3021
# never 2282 + 0 = 2282
# dated 735 + 4 = 739
# 2282 + 739 == 3021
#
# #1341 items 55, 56 and 16, the row-identity lock out of SQL's reach,
# bulk_parallel_writer set by hand, and the row_lock_buckets bound (jdatcmd): 2282 ->
# 2291, and suites_not_covered 231 -> 230. advisory_lock_class had no rows; it enters
# with all 17 checks from five-major runs (17/17 on each of 15-19). Eight are dated on
# PG17:
# - the row-identity class put at 2: the class-104 arm, the UPDATE's reachable-class
# arm and its contention arm;
# - the unique-key class put back at 2 (#430): the INSERT's reachable-class arm and
# its contention arm;
# - the storage-row lock skipped whenever bulk_parallel_writer is on: the second
# first write fails on storage_pkey;
# - the bucket taken as the raw key, one row lock per row: the bulk UPDATE fails
# with "out of shared memory", which reddens its lock-count arm and its row arm.
# The other nine arrive never: seeding a suite moves the census up by its undated rows.
#
# ledger rows 3021 + 17 = 3038
# never 2282 + 9 = 2291
# dated 739 + 8 = 747
# 2291 + 747 == 3038
checks_never_observed_red 2291
Loading
Loading