Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -657,7 +657,8 @@ other than where the feature is used:
`ON CONFLICT DO NOTHING` takes no row lock and works, with or without a conflict
target.

`CREATE TABLE` refuses an unlogged columnar table for the same reason. Both cases
`CREATE TABLE` refuses an unlogged columnar table for the same reason, and so does
`ALTER TABLE ... SET UNLOGGED`. Both cases
follow one rule. This access method refuses a configuration that it cannot
support at the point where you choose it. It does not refuse it at each later
use.
Expand Down Expand Up @@ -783,7 +784,11 @@ therefore still be cheaper as a scan.
of a deleted row only while a transaction that began before the delete is still
running, as it does on heap.
- `CREATE INDEX CONCURRENTLY` (the concurrent validate path) and partial
block-range index builds are not supported.
block-range index builds are not supported. A refused `CREATE INDEX
CONCURRENTLY` leaves its index behind, marked invalid, as PostgreSQL does for any
concurrent build that fails. Drop that index before you build it again.
- `ALTER TABLE ... SET TABLESPACE` is not supported on a columnar table. It is
refused, and the table stays where it was.

## Constraints on the import path

Expand Down
122 changes: 122 additions & 0 deletions test/advisory_lock_class.sh
Original file line number Diff line number Diff line change
Expand Up @@ -245,4 +245,126 @@ check "row_lock_buckets is fixed at server start, so a session cannot change it
"$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At -v VERBOSITY=sqlstate -c "SET pgcolumnar.row_lock_buckets = 7;" 2>&1 | sed -n 's/^.*ERROR:[[:space:]]*\([0-9A-Z]\{5\}\).*$/\1/p')|$(q "SELECT context || '|' || setting FROM pg_settings WHERE name = 'pgcolumnar.row_lock_buckets'")" \
"55P02|postmaster|1024"

# The documented bound (docs/configuration.md): row_lock_buckets "bounds the row locks
# a transaction holds, so a bulk update cannot exhaust the lock table". One transaction
# updates 200000 rows and then counts its own class-104 locks (one -c string is one
# transaction): one per bucket, not one per row. A row lock per row would not fit the
# default lock table at all.
psql_run "CREATE TABLE rlb (id int, v int) USING pgcolumnar;
INSERT INTO rlb SELECT g, g FROM generate_series(1, 200000) g;" >/dev/null
check "a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them" \
"$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "UPDATE rlb SET v = v + 1;
SELECT count(*) || '|' || current_setting('pgcolumnar.row_lock_buckets') FROM pg_locks
WHERE pid = pg_backend_pid() AND locktype = 'advisory' AND objsubid = 104;" 2>&1 |
grep -E '^[0-9]+[|]|ERROR' | tail -1)" "1024|1024"
check "and the UPDATE committed every row" "$(q "SELECT count(*) || '|' || sum(v - id) FROM rlb")" "200000|200000"

# ---- the row-identity lock cannot be reached from SQL (#1341 item 56) ------------
# The INSERT arms above ask this of the unique-key and storage-row locks; nothing
# asked it of class 104. An UPDATE is held open and every advisory lock it took is
# read. None may be in field 1 or 2, the only classes SQL can produce. Then a second
# session takes each of those exact tags with pg_advisory_xact_lock(int4, int4), and
# a columnar UPDATE of the same row must not wait on it. An oid above the int4 range
# is addressed by its negative int4, the same 32 bits lockfuncs.c puts in the tag.
rl_wait_for() { # rl_wait_for APPLICATION_NAME -> its pid, once it has named itself
local i pid=""
for i in $(seq 1 100); do
pid="$(q "SELECT pid FROM pg_stat_activity WHERE application_name = '$1'")"
[ -n "$pid" ] && break
sleep 0.1
done
echo "$pid"
}
rl_end() { # rl_end PID BGPID: end a holder's backend and its psql
q "SELECT pg_terminate_backend($1)" >/dev/null 2>&1
kill "$2" 2>/dev/null; wait "$2" 2>/dev/null
local i
for i in $(seq 1 60); do
[ "$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND pid <> pg_backend_pid()")" = 0 ] && break
sleep 0.2
done
}
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "BEGIN; UPDATE rl SET v = v + 1 WHERE id = 7; SET application_name = 'rl_tagger'; SELECT pg_sleep(30); COMMIT;" \
>/dev/null 2>&1 &
rl_bg=$!
rl_pid="$(rl_wait_for rl_tagger)"
RL_TAGS="$(q "SELECT CASE WHEN classid::bigint > 2147483647 THEN classid::bigint - 4294967296 ELSE classid::bigint END
|| ' ' || CASE WHEN objid::bigint > 2147483647 THEN objid::bigint - 4294967296 ELSE objid::bigint END
|| ' ' || objsubid
FROM pg_locks WHERE locktype = 'advisory' AND granted AND pid = ${rl_pid:-0}")"
rl_end "${rl_pid:-0}" "$rl_bg"
RL_N="$(printf '%s\n' "$RL_TAGS" | grep -c '[0-9]')"
rl_reach=""
while read -r _c _o _f; do
case "${_f:-}" in 1|2) rl_reach="$rl_reach ($_c,$_o,field4=$_f)" ;; esac
done <<<"$RL_TAGS"
check "the locks an UPDATE takes are not in a SQL-reachable class" \
"$(if [ "$RL_N" -eq 0 ]; then echo "unknown (no advisory lock seen; holder pid ${rl_pid:-none})"
elif [ -n "$rl_reach" ]; then echo "reachable:$rl_reach"
else echo unreachable; fi)" "unreachable"
rl_grab="BEGIN;"
while read -r _c _o _f; do
[ -z "${_f:-}" ] && continue
rl_grab="$rl_grab SELECT pg_advisory_xact_lock($_c::int, $_o::int);"
done <<<"$RL_TAGS"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "$rl_grab SET application_name = 'rl_user'; SELECT pg_sleep(30); COMMIT;" >/dev/null 2>&1 &
rl_bg=$!
rl_pid="$(rl_wait_for rl_user)"
check "premise: the other session holds every one of those tags through the SQL functions" \
"$(q "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND granted AND objsubid = 2
AND pid = ${rl_pid:-0}")" "$RL_N"
rl_upd="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "SET statement_timeout = '10s';" -c "UPDATE rl SET v = v + 1 WHERE id = 7;" 2>&1)"
case "$rl_upd" in
*timeout*|*canceling*) rl_verdict="BLOCKED by the user lock" ;;
*ERROR*) rl_verdict="ERROR: $(head -1 <<<"$rl_upd")" ;;
*) rl_verdict=ok ;;
esac
check "a user advisory lock on those tags does not block a columnar UPDATE" "$rl_verdict" "ok"
rl_end "${rl_pid:-0}" "$rl_bg"

# ---- bulk_parallel_writer set by hand still serialises a first write (#1341 item 55) --
# The setting lets parallel_copy's loaders skip the storage-row creation lock (class
# 102). The reference says setting it by hand is safe, because the skip fires only
# once the storage row is committed. Two sessions with it on each make the first
# write to a committed, empty table: the second must wait for the first and then
# land its row. A skip that did not wait for the row to be committed lets both
# create it, and the second fails on storage_pkey.
#
# The first writer commits only once the suite has looked for the second one waiting
# and then inserted a committed flag row, not after a fixed sleep: a slow second
# writer would otherwise start after the first had committed, and the premise below
# would fail on timing (OffgridwithJD, review of #1415). The pytest twin does the same.
psql_run "CREATE TABLE bw (id int) USING pgcolumnar; CREATE TABLE bw_go (go int) USING heap;"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -Atq \
-c "SET statement_timeout = '60s'; BEGIN; SET LOCAL pgcolumnar.bulk_parallel_writer = on; INSERT INTO bw VALUES (1); SET application_name = 'bw_first';
DO \$\$ BEGIN WHILE NOT EXISTS (SELECT 1 FROM bw_go) LOOP PERFORM pg_sleep(0.1); END LOOP; END \$\$; COMMIT;" \
>/dev/null 2>&1 &
bw_bg=$!
bw_pid="$(rl_wait_for bw_first)"
env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -At \
-c "SET pgcolumnar.bulk_parallel_writer = on; SET statement_timeout = '30s';" -c "INSERT INTO bw VALUES (2);" \
>"$PGC_WORKDIR/bw_second.out" 2>&1 &
bw_bg2=$!
# The second write must be waiting on the first before the first commits, or the arm
# measures two writes in sequence.
bw_waited=no
for _i in $(seq 1 100); do
[ "$(q "SELECT count(*) FROM pg_stat_activity WHERE wait_event_type = 'Lock' AND query LIKE 'INSERT INTO bw%'")" -ge 1 ] && { bw_waited=yes; break; }
sleep 0.1
done
psql_run "INSERT INTO bw_go VALUES (1);" # release the first writer, whatever the poll saw
check "premise: the second first write waited on the first" "$bw_waited" "yes"
wait "$bw_bg2"; wait "$bw_bg"
bw_out="$(cat "$PGC_WORKDIR/bw_second.out")"
case "$bw_out" in
*ERROR*) bw_verdict="ERROR: $(grep -m1 ERROR <<<"$bw_out")" ;;
*) bw_verdict=ok ;;
esac
check "with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land" \
"$([ -n "$bw_pid" ] && echo held || echo "no holder")|$bw_verdict|$(q "SELECT count(*) FROM bw")" "held|ok|2"

pgc_summary
19 changes: 19 additions & 0 deletions test/am_contracts.sh
Original file line number Diff line number Diff line change
Expand Up @@ -351,4 +351,23 @@ check "REINDEX after a committed delete of most rows reclaims their entries, as
"$(ix_reclaim pgcolumnar)|$(ix_reclaim heap)|$(q "SET enable_seqscan = off; SELECT count(*) FROM ix_pgcolumnar WHERE a > 0;" | tail -1)" \
"reclaimed|reclaimed|10000"

# ---- documented refusals (#1341 items 44, 51, 77) ----------------------------------
#
# Each is refused where it is chosen, as limitations.md says, and refused cleanly:
# nothing is left changed by the attempt.
rf_count() { q "SELECT count(*) || '|' || sum(a) FROM rf"; }
psql_run "CREATE TABLE rf (a int) USING pgcolumnar; INSERT INTO rf SELECT g FROM generate_series(1, 5000) g;"
check "CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table" \
"$(state "CREATE UNLOGGED TABLE rf_u (a int) USING pgcolumnar")|$(q "SELECT count(*) FROM pg_class WHERE relname = 'rf_u'")" "0A000|0"
check "ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows" \
"$(state "ALTER TABLE rf SET UNLOGGED")|$(q "SELECT relpersistence FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|p|5000|12502500"
pgc_pg "mkdir -p -m 700 '$PGC_WORKDIR/rf_ts'"
psql_run "CREATE TABLESPACE rf_ts LOCATION '$PGC_WORKDIR/rf_ts';"
check "ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows" \
"$(state "ALTER TABLE rf SET TABLESPACE rf_ts")|$(q "SELECT reltablespace FROM pg_class WHERE relname = 'rf'")|$(rf_count)" "0A000|0|5000|12502500"
# A failed concurrent build leaves its index behind, marked invalid, as core does for
# any CREATE INDEX CONCURRENTLY that fails after its first phase.
check "CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid" \
"$(state "CREATE INDEX CONCURRENTLY rf_cic ON rf (a)")|$(q "SELECT indisvalid FROM pg_index WHERE indexrelid = 'rf_cic'::regclass")" "0A000|f"

pgc_summary
34 changes: 34 additions & 0 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
advisory_lock_class advisory_lock_class a bulk UPDATE of 200000 rows holds one row-identity lock per bucket, row_lock_buckets of them 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row)
advisory_lock_class advisory_lock_class a duplicate key is still rejected 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class a user advisory lock on that tag does not block a columnar insert 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h)
advisory_lock_class advisory_lock_class a user advisory lock on those tags does not block a columnar UPDATE 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class an UPDATE holds a row-identity lock (class 104) while enable_row_update_lock is on 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class and holds none with enable_row_update_lock off 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class and the UPDATE committed every row 15;16;17;18;19 2026-10-15 columnar_row_lock.c: the row-identity bucket taken as the key mod 4294967291, not mod row_lock_buckets (one lock per row)
advisory_lock_class advisory_lock_class premise: the discovering session is gone and holds nothing 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the insert took an advisory lock we can see 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the lock is enabled, or nothing below proves anything 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the other session holds every one of those tags through the SQL functions 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the other session really holds that exact tag in class 2 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class premise: the second first write waited on the first 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class row_lock_buckets is fixed at server start, so a session cannot change it (55P02) 15;16;17;18;19 never -
advisory_lock_class advisory_lock_class the lock an insert takes is not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_UNIQUE_KEY 103 changed to 2, the #430 defect (src/columnar.h)
advisory_lock_class advisory_lock_class the locks an UPDATE takes are not in a SQL-reachable class 15;16;17;18;19 2026-10-15 PGCOLUMNAR_LOCKCLASS_ROW_IDENTITY 104 changed to 2, the class SQL's two-int4 form produces (src/columnar.h)
advisory_lock_class advisory_lock_class with bulk_parallel_writer set by hand, a second first write waits for the first and both rows land 15;16;17;18;19 2026-10-15 the storage-row creation lock skipped whenever bulk_parallel_writer is on, not only once the row is committed (src/columnar_metadata.c)
alter_table_scans alter_table_scans ADD COLUMN with a volatile default rewrites the table, as on heap 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c)
alter_table_scans alter_table_scans ADD PRIMARY KEY is refused over a duplicate (23505), as on heap 15;16;17;18;19 never -
alter_table_scans alter_table_scans ADD PRIMARY KEY succeeds once the duplicate is deleted, as on heap 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c)
Expand All @@ -13,6 +30,10 @@ alter_table_scans alter_table_scans and the rewrite kept the live rows and not t
alter_table_scans alter_table_scans the deleted rows are gone from both tables, which hold the same rows 15;16;17;18;19 2026-10-15 every scan returns deleted rows, the reader's delete filter disabled (src/columnar_reader.c)
alter_table_scans alter_table_scans the lookup of id 1 is an index scan on the primary key 15;16;17;18;19 2026-10-15 the index build marks every row alive, deleted ones included (src/columnar_tableam.c)
alter_table_scans alter_table_scans the primary key still finds id 1 once, through the index, after both rewrites 15;16;17;18;19 never -
am_contracts am_contracts ALTER TABLE ... SET TABLESPACE is refused (0A000), and the table stays where it was with its rows 15;16;17;18;19 2026-10-15 relation copy returning without copying (SET TABLESPACE)
am_contracts am_contracts ALTER TABLE ... SET UNLOGGED is refused (0A000), and the table stays logged with its rows 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table
am_contracts am_contracts CREATE INDEX CONCURRENTLY is refused (0A000), and the index it leaves is marked invalid 15;16;17;18;19 2026-10-15 the concurrent index validate scan doing nothing
am_contracts am_contracts CREATE UNLOGGED TABLE ... USING pgcolumnar is refused (0A000) and leaves no table 15;16;17;18;19 2026-10-15 the storage hook accepting an unlogged columnar table
am_contracts am_contracts ON CONFLICT DO NOTHING skips the duplicate and inserts the new row 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
am_contracts am_contracts ON CONFLICT DO UPDATE is refused as unsupported (0A000), and changes nothing 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
am_contracts am_contracts ON CONFLICT DO UPDATE with no conflicting row inserts it 15;16;17;18;19 2026-10-14 speculative insert refused as unsupported, as before #1341
Expand Down Expand Up @@ -2026,6 +2047,19 @@ latest_reading latest_reading the GROUP BY form returns what DISTINCT ON returns
latest_reading latest_reading the LATERAL form returns what DISTINCT ON returns 15;16;17;18;19 2026-10-12 the document's LATERAL recipe orders by ts ascending, so it returns each device's FIRST reading
latest_reading latest_reading with the btree the LATERAL form reads under a tenth of the table 15;16;17;18;19 never -
latest_reading latest_reading without the btree the LATERAL form reads the table once per device 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback after a savepoint rolls back two rewrites, the rows written later in the transaction commit beside the old ones 15;16;17;18;19 2026-10-15 PgColumnarAllocateFreeSpace reuses space whatever its freeing xid, so a rolled-back maintenance function's freed bytes are overwritten
maintenance_rollback maintenance_rollback after compact rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback after compact_rewrite rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 2026-10-15 PgColumnarAllocateFreeSpace reuses space whatever its freeing xid, so a rolled-back maintenance function's freed bytes are overwritten
maintenance_rollback maintenance_rollback after recluster rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback after vacuum rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback after vacuum_sorted rolls back, the table holds the same rows and row groups, by scan and by index 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback and every function then runs for real and keeps the same rows 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback compact changed the row groups inside its transaction 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback compact_rewrite changed the row groups inside its transaction 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback premise: the table holds the rows it was given 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback recluster changed the row groups inside its transaction 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback vacuum changed the row groups inside its transaction 15;16;17;18;19 never -
maintenance_rollback maintenance_rollback vacuum_sorted changed the row groups inside its transaction 15;16;17;18;19 never -
native_agg native_agg an aggregate that needs zone maps reads them 15;16;17;18;19 never -
native_agg native_agg avg(int4,nulls) 15;16;17;18;19 never -
native_agg native_agg control: decoding ts reads the table's blocks 15;16;17;18;19 never -
Expand Down
Loading
Loading