From 996413c03e35c91d61203a77e448e577fe840ca5 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 22:57:27 +0000 Subject: [PATCH 1/4] test: types the differential matrix never stored answer as heap (#1341) The encoder has fast paths for fixed widths 1, 2, 4 and 8, and every other width and every varlena takes a generic path. A new suite, type_matrix, and its pytest twin store what no test had: - by-reference fixed widths 6, 8, 12, 16 and 64: macaddr, macaddr8, timetz, point, name; - the 1-byte "char"; - time, money, inet, cidr, bit, varbit, tsvector, xml, json; - an enum, a composite, a domain; - numeric NaN and +-Infinity, timestamp and date +-infinity. 12000 rows in row groups of 1000, each column NULL on its own pattern, are generated once into heap and copied. Against heap: every row as text, each column's counts, a range on all sixteen ordered columns with literal bounds so zone maps can prune, json, point and tsvector matches, the special-value aggregates, and the composite's fields. 24/24 on each of PG15-19, and the twin on each. Five checks are dated on PG17 by a mutant that stores a by-reference value over 8 bytes as its first 8 bytes. cluster_tests 565 -> 566. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N Squashed for the rebase onto main; it replaces: - test: type_matrix's pinned cidr arm names #1433, the cause OffgridwithJD found (#1341) - test: type_matrix's rows re-dated, restated on main (#1341) - test: type_matrix varies timetz's zone, asserts each range arm's pruning, and names a missing libxml (#1341) - test: types the differential matrix never stored answer as heap (#1341) Squashed for the rebase; it replaces: - test: type_matrix's rows re-dated, restated on main 6f0bd12e (#1341) - test: type_matrix's bit and varbit columns increase with the row, its bounds keep their typmod, and vb must skip (#1341) - test: types the differential matrix never stored answer as heap (#1341) --- test/check_ledger.tsv | 41 ++++++++ test/check_ledger_budget.txt | 19 +++- test/pytest/TESTS.md | 11 +++ test/pytest/expected_tests.txt | 3 +- test/pytest/test_compare_to_bash.py | 1 + test/pytest/test_type_matrix.py | 140 ++++++++++++++++++++++++++++ test/run_all_versions.sh | 1 + test/type_matrix.sh | 128 +++++++++++++++++++++++++ 8 files changed, 342 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_type_matrix.py create mode 100755 test/type_matrix.sh diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 560957880..25931eddd 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -3325,6 +3325,47 @@ ttl_expire ttl_expire the retention is recorded where the other options live 15; ttl_expire ttl_expire with the extent, expire retires the same groups 15;16;17;18;19 2026-10-13 the early exit compares the extent's maximum to the cutoff instead of its minimum ttl_expire ttl_expire without the extent the same expire probes every group's zone map 15;16;17;18;19 never - ttl_expire ttl_expire without the extent, expire retires the aged groups 15;16;17;18;19 2026-10-14 expire returning 0 whatever it retired +type_matrix type_matrix a range on bt finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on bt lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on cd finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on cd skips no chunk groups: a cidr key is treated as cross-type against inet and dropped (#1433) 15;16;17;18;19 never - +type_matrix type_matrix a range on ch finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on ch skips no chunk groups: every row group holds every value, since the column cycles through 26 letters 15;16;17;18;19 never - +type_matrix type_matrix a range on d finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on d lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on dom finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on dom lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on e finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on e skips no chunk groups: every row group holds every value, since the column cycles through the 4 labels 15;16;17;18;19 never - +type_matrix type_matrix a range on ip finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on ip lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on mac finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mac lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on mac8 finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mac8 lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on mo finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mo lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on n finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on n skips no chunk groups: every row group holds NaN and both infinities, so each zone map spans every value 15;16;17;18;19 never - +type_matrix type_matrix a range on nm finds heap's rows 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix a range on nm lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen);the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on tm finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on tm lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on tsi finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on tsi lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on tz finds heap's rows 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix a range on tz lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a range on vb finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on vb lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) +type_matrix type_matrix a tsvector match finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix each column's count and NULL count is heap's 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix equality on json, as text, finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix every row, as text, is heap's 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix point equality (~=) finds heap's rows 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix premise: both tables hold 12000 rows, the columnar one in at least twelve row groups 15;16;17;18;19 never - +type_matrix type_matrix premise: the server supports xml (built with libxml) 15;16;17;18;19 never - +type_matrix type_matrix the aggregates over numeric NaN and Infinity and timestamp and date infinity are heap's 15;16;17;18;19 never - +type_matrix type_matrix the composite's fields and the enum's order read back as heap's 15;16;17;18;19 never - vacuum_lock_privilege vacuum_lock_privilege add_projection refuses a non-owner before taking its lock (#749) 15;16;17;18;19 2026-10-14 every maintenance entry point checks ownership after taking its lock vacuum_lock_privilege vacuum_lock_privilege cluster refuses a non-owner before taking the exclusive lock 15;16;17;18;19 2026-10-14 every maintenance entry point checks ownership after taking its lock vacuum_lock_privilege vacuum_lock_privilege compact does the same, which is the behaviour drop_projection now matches 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index a3960011a..ad80824b3 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2760,4 +2760,21 @@ suites_not_covered 224 # never 2544 + 13 = 2557 # dated 836 + 11 = 847 # 2557 + 847 == 3404 -checks_never_observed_red 2557 +# +# #1341, types the differential matrix never stored answer as heap (jdatcmd): +# 2557 -> 2581. type_matrix enters with all 41 checks from five-major runs (41/41 on +# each of 15-19). 17 are dated on PG17 by two mutants (OffgridwithJD, reviews of +# #1426): +# - a by-reference fixed-width value over 8 bytes stored as its first 8 bytes and +# zero padding: every row, the counts, the tz range (its zone offset varies, so +# bytes 8-11 matter), the nm range and its pruning, and point equality; +# - the scan never skipping a row group by zone map: the 12 range arms that must +# skip chunk groups, varbit's included now that its values increase with the row. +# The rest arrive never, among them the four range arms pinned at 0 skipped groups, +# with the reason in each name (cidr's names #1433), and the libxml premise. +# +# ledger rows 3404 + 41 = 3445 +# never 2557 + 24 = 2581 +# dated 847 + 17 = 864 +# 2581 + 864 == 3445 +checks_never_observed_red 2581 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 2e32bcdcb..e3fb22797 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -176,6 +176,7 @@ behaviour, the source of that number is named. - [128. test_objstore_userinfo.py: userinfo in a URL is refused on every entry point](#128-test_objstore_userinfopy-userinfo-in-a-url-is-refused-on-every-entry-point) - [129. test_savepoint_flush.py: a savepoint rolled back after a flush](#129-test_savepoint_flushpy-a-savepoint-rolled-back-after-a-flush) - [130. test_rls_query_paths.py: row-level security holds on every read path](#130-test_rls_query_pathspy-row-level-security-holds-on-every-read-path) +- [131. test_type_matrix.py: types the differential matrix never stored answer as heap](#131-test_type_matrixpy-types-the-differential-matrix-never-stored-answer-as-heap) ## 1. How to read a test in here @@ -7175,3 +7176,13 @@ The pytest twin of `test/rls_query_paths.sh` (#1341), every check by the shell's | test | what it holds | | --- | --- | | `test_row_level_security_holds_on_every_read_path` | the read paths against heap (the custom scan, both vectorized aggregates, parallel, index-only, the covering projection), their plan premises, and the leak arms, including one through each vectorized node | + +## 131. test_type_matrix.py: types the differential matrix never stored answer as heap + +The pytest twin of `test/type_matrix.sh` (#1341), every check by the shell's name. The encoder has fast paths for fixed widths 1, 2, 4 and 8, and every other width and every varlena takes a generic path. This matrix stores what no test had: by-reference fixed widths 6, 8, 12, 16 and 64 (macaddr, macaddr8, timetz, point, name), the 1-byte `"char"`, time, money, inet, cidr, bit, varbit, tsvector, xml, json, an enum, a composite, a domain, and numeric NaN and ±Infinity with timestamp and date ±infinity. 12000 rows in row groups of 1000, each column NULL on its own pattern, are generated once into heap and copied. Every row as text, each column's counts, a range on every ordered column with literal bounds read from heap, json, point and tsvector matches, the special-value aggregates, and the composite's fields must all equal heap's. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_types_the_matrix_never_stored_answer_as_heap` | the 21 columns against heap: rows, counts, sixteen ranges, the three non-btree matches, and the special values | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index e230ed616..968ada983 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -724,4 +724,5 @@ guard_tests 438 # cluster_tests 599 -> 600 for #1341's test_savepoint_flush.py, one test, by collection # cluster_tests 600 -> 601 for #1341's test_rls_query_paths.py, one test, by collection # over the cluster files on this tree. -cluster_tests 601 +# cluster_tests 601 -> 602 for #1341's test_type_matrix.py, one test, by collection +cluster_tests 602 diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 267066828..43943a462 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -164,6 +164,7 @@ "tier_partition", "timeseries_recipes", "ttl_expire", + "type_matrix", "vacuum_lock_privilege", "validity_elision", "vm_privilege", diff --git a/test/pytest/test_type_matrix.py b/test/pytest/test_type_matrix.py new file mode 100644 index 000000000..9b72e510e --- /dev/null +++ b/test/pytest/test_type_matrix.py @@ -0,0 +1,140 @@ +"""Types the differential matrix never stored, against heap (#1341). The pytest twin +of `test/type_matrix.sh`, every check by the shell's name. + +The encoder has fast paths for fixed widths 1, 2, 4 and 8; every other width and every +varlena takes a generic path, and a zone map is built for each column whose type has a +btree ordering. These columns cover widths 6, 12, 16 and 64 by reference, the 1-byte +"char", time, money, inet, cidr, bit, varbit, tsvector, xml, json, an enum, a composite, +a domain, and numeric, timestamp and date special values. 12000 rows in row groups of +1000, each column NULL on its own pattern, generated once into heap and copied. + +Each comparison is of the rows themselves (`expect.row_set`), so a failure names the rows +that differ. The range bounds are the values heap holds at ids 3005 and 9005, written as +literals so the columnar scan can prune row groups by zone map (not 3001, which holds n's +NaN). Each range arm also says whether the zone maps skipped chunk groups, from EXPLAIN +ANALYZE; four that measure 0 are pinned at 0 with the reason. timetz's zone offset varies +with the row, so its 12-byte width is used past byte 8. +""" +import re + +import psycopg + +COLS = """id int, tz timetz, mac macaddr, mac8 macaddr8, nm name, pt point, ch "char", + tm time, mo money, ip inet, cd cidr, bt bit(16), vb varbit, tsv tsvector, x xml, + j json, e tm_mood, cmp tm_pair, dom tm_pos, n numeric, tsi timestamp, d date""" + +LOAD = """SELECT g, + CASE WHEN g % 7 = 0 THEN NULL ELSE ((('00:00'::time + (g * 7) * interval '1 s')::text || '+' || lpad((g % 13)::text, 2, '0'))::timetz) END, + CASE WHEN g % 11 = 0 THEN NULL ELSE ('08:00:2b:' || lpad(to_hex(g / 256 % 256), 2, '0') || ':' || lpad(to_hex(g % 256), 2, '0') || ':01')::macaddr END, + CASE WHEN g % 13 = 0 THEN NULL ELSE ('08:00:2b:01:02:03:' || lpad(to_hex(g / 256 % 256), 2, '0') || ':' || lpad(to_hex(g % 256), 2, '0'))::macaddr8 END, + CASE WHEN g % 17 = 0 THEN NULL ELSE ('name_' || lpad(g::text, 6, '0'))::name END, + CASE WHEN g % 19 = 0 THEN NULL ELSE point(g, -g) END, + CASE WHEN g % 23 = 0 THEN NULL ELSE chr(65 + g % 26)::"char" END, + CASE WHEN g % 29 = 0 THEN NULL ELSE ('00:00'::time + g * interval '7 s') END, + CASE WHEN g % 31 = 0 THEN NULL ELSE (g * 1.25)::numeric::money END, + CASE WHEN g % 37 = 0 THEN NULL ELSE ('10.' || g / 256 % 256 || '.' || g % 256 || '.' || g % 7 || '/32')::inet END, + CASE WHEN g % 41 = 0 THEN NULL ELSE ('10.' || g / 256 % 256 || '.' || g % 256 || '.0/24')::cidr END, + CASE WHEN g % 43 = 0 THEN NULL ELSE g::bit(16) END, + CASE WHEN g % 47 = 0 THEN NULL ELSE (g::bit(16) || (g % 7)::bit(3))::varbit END, + CASE WHEN g % 53 = 0 THEN NULL ELSE to_tsvector('simple', 'word' || g || ' common term' || g % 9) END, + CASE WHEN g % 59 = 0 THEN NULL ELSE ('' || md5(g::text) || '')::xml END, + CASE WHEN g % 61 = 0 THEN NULL ELSE json_build_object('g', g, 's', md5(g::text)) END, + CASE WHEN g % 67 = 0 THEN NULL ELSE (enum_range(NULL::tm_mood))[1 + g % 4] END, + CASE WHEN g % 71 = 0 THEN NULL ELSE ROW(g, 'p' || g)::tm_pair END, + CASE WHEN g % 73 = 0 THEN NULL ELSE (g)::tm_pos END, + CASE WHEN g % 79 = 0 THEN NULL WHEN g % 1000 = 1 THEN 'NaN'::numeric WHEN g % 1000 = 2 THEN 'Infinity'::numeric + WHEN g % 1000 = 3 THEN '-Infinity'::numeric ELSE g / 7.0 END, + CASE WHEN g % 83 = 0 THEN NULL WHEN g = 500 THEN 'infinity'::timestamp WHEN g = 501 THEN '-infinity'::timestamp + ELSE '2026-01-01'::timestamp + g * interval '1 min' END, + CASE WHEN g % 89 = 0 THEN NULL WHEN g = 600 THEN 'infinity'::date WHEN g = 601 THEN '-infinity'::date + ELSE '2026-01-01'::date + g END + FROM generate_series(1, 12000) g""" + +ALL = ("id, tz::text, mac::text, mac8::text, nm::text, pt::text, ch::text, tm::text, mo::text, ip::text, " + "cd::text, bt::text, vb::text, tsv::text, x::text, j::text, e::text, cmp::text, dom::text, n::text, " + "tsi::text, d::text") + + +def _rows(conn, sql): + with conn.cursor() as cur: + cur.execute(sql) + return cur.fetchall() + + +def _both(conn, sql): + return _rows(conn, sql.replace("%T", "t_col")), _rows(conn, sql.replace("%T", "t_heap")) + + +NO_PRUNE = { + "ch": "every row group holds every value, since the column cycles through 26 letters", + "e": "every row group holds every value, since the column cycles through the 4 labels", + "n": "every row group holds NaN and both infinities, so each zone map spans every value", + "cd": "a cidr key is treated as cross-type against inet and dropped (#1433)", +} + + +def _removed(conn, sql): + plan = "\n".join(r[0] for r in _rows(conn, "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) " + + sql.replace("%T", "t_col"))) + m = re.search(r"Columnar Chunk Groups Removed by Filter: (\d+)", plan) + return m.group(1) if m else "none" + + +def test_types_the_matrix_never_stored_answer_as_heap(pgc_conn, expect): + # xml needs a server built with libxml; without it every arm after the load would + # fail for that reason, so this says so first. + try: + xml_ok = "yes" if _rows(pgc_conn, "SELECT ''::xml IS NOT NULL")[0][0] else "no" + except psycopg.Error as exc: + xml_ok = f"no ({str(exc).splitlines()[0]})" + expect.text(xml_ok, "yes", "premise: the server supports xml (built with libxml)") + with pgc_conn.cursor() as cur: + cur.execute("CREATE TYPE tm_mood AS ENUM ('sad', 'ok', 'happy', 'elated')") + cur.execute("CREATE TYPE tm_pair AS (a int, b text)") + cur.execute("CREATE DOMAIN tm_pos AS int CHECK (VALUE > 0)") + cur.execute(f"CREATE TABLE t_heap ({COLS})") + cur.execute(f"CREATE TABLE t_col ({COLS}) USING pgcolumnar") + cur.execute("SELECT pgcolumnar.set_options('t_col', stripe_row_limit => 1000)") + cur.execute(f"INSERT INTO t_heap {LOAD}") + cur.execute("INSERT INTO t_col SELECT * FROM t_heap") + groups = _rows(pgc_conn, "SELECT count(*) FROM pgcolumnar.row_group " + "WHERE storage_id = pgcolumnar.get_storage_id('t_col')")[0][0] + n_heap = _rows(pgc_conn, "SELECT count(*) FROM t_heap")[0][0] + n_col = _rows(pgc_conn, "SELECT count(*) FROM t_col")[0][0] + expect.text(f"{n_heap}|{n_col}|{'many' if groups >= 12 else groups}", "12000|12000|many", + "premise: both tables hold 12000 rows, the columnar one in at least twelve row groups") + + got, want = _both(pgc_conn, f"SELECT {ALL} FROM %T") + expect.row_set(got, want, "every row, as text, is heap's") + counts = ", ".join(f"count({c}), count(*) FILTER (WHERE {c} IS NULL)" + for c in "tz mac mac8 nm pt ch tm mo ip cd bt vb tsv x j e cmp dom n tsi d".split()) + got, want = _both(pgc_conn, f"SELECT {counts} FROM %T") + expect.row_set(got, want, "each column's count and NULL count is heap's") + + for c in ("tz", "mac", "mac8", "nm", "ch", "tm", "mo", "ip", "cd", "bt", "vb", "e", "dom", "n", "tsi", "d"): + # format_type keeps the typmod: pg_typeof would make bt's bounds bit(1). + ty = _rows(pgc_conn, f"SELECT format_type(atttypid, atttypmod) FROM pg_attribute " + f"WHERE attrelid = 't_heap'::regclass AND attname = '{c}'")[0][0] + lo, hi = (_rows(pgc_conn, f"SELECT quote_literal({c}::text) || '::' || '{ty}' " + f"FROM t_heap WHERE id = {i}")[0][0] for i in (3005, 9005)) + sql = f"SELECT id FROM %T WHERE {c} BETWEEN least({lo}, {hi}) AND greatest({lo}, {hi})" + got, want = _both(pgc_conn, sql) + expect.row_set(got, want, f"a range on {c} finds heap's rows") + r = _removed(pgc_conn, sql) + if c in NO_PRUNE: + expect.text(r, "0", f"a range on {c} skips no chunk groups: {NO_PRUNE[c]}") + else: + expect.text("skips" if r.isdigit() and int(r) >= 1 else f"removed {r}", "skips", + f"a range on {c} lets the zone maps skip chunk groups") + got, want = _both(pgc_conn, "SELECT id FROM %T WHERE j::text = (SELECT j::text FROM t_heap WHERE id = 4242)") + expect.row_set(got, want, "equality on json, as text, finds heap's rows") + got, want = _both(pgc_conn, "SELECT id FROM %T WHERE pt ~= point(4242, -4242)") + expect.row_set(got, want, "point equality (~=) finds heap's rows") + got, want = _both(pgc_conn, "SELECT id FROM %T WHERE tsv @@ to_tsquery('simple', 'term3')") + expect.row_set(got, want, "a tsvector match finds heap's rows") + got, want = _both(pgc_conn, "SELECT min(n)::text, max(n)::text, count(*) FILTER (WHERE n = 'NaN'), " + "count(*) FILTER (WHERE n = 'Infinity'), count(*) FILTER (WHERE n = '-Infinity'), " + "min(tsi)::text, max(tsi)::text, min(d)::text, max(d)::text FROM %T") + expect.row_set(got, want, "the aggregates over numeric NaN and Infinity and timestamp and date infinity are heap's") + got, want = _both(pgc_conn, "SELECT (cmp).a, (cmp).b, e::text, e > 'ok' FROM %T WHERE id % 97 = 0") + expect.row_set(got, want, "the composite's fields and the enum's order read back as heap's") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index dbd03a1c3..4466359f9 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -319,6 +319,7 @@ SUITES=( timeseries_recipes truncate_cleanup ttl_expire + type_matrix ungrouped_vector_agg unique_conc update_conc diff --git a/test/type_matrix.sh b/test/type_matrix.sh new file mode 100755 index 000000000..3c660af2b --- /dev/null +++ b/test/type_matrix.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# +# Types the differential matrix never stored, against heap (#1341). The encoder has +# fast paths for fixed widths 1, 2, 4 and 8; every other width and every varlena +# takes a generic path, and a zone map is built for each column whose type has a +# btree ordering. None of these types had been written to a columnar table: +# +# - fixed width, by reference, at unusual widths: timetz (12), macaddr (6), +# macaddr8 (8), name (64), point (16, no btree ordering); +# - the 1-byte "char"; +# - time, money, inet, cidr, bit, varbit, tsvector, xml, json (no equality); +# - an enum, a composite, a domain over int; +# - numeric NaN and +-Infinity, timestamp and date +-infinity. +# +# 12000 rows in row groups of 1000, each column NULL on its own pattern. The same +# rows go into heap and are copied to columnar. Then: every row, as text; each +# column's count and NULL count; a range on every ordered column, with constants +# read from heap so the zone maps can prune; equality on json (as text) and point +# (~=); and the aggregates over the special values. +# +# Usage: test/type_matrix.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +psql_run "CREATE TYPE tm_mood AS ENUM ('sad', 'ok', 'happy', 'elated'); + CREATE TYPE tm_pair AS (a int, b text); + CREATE DOMAIN tm_pos AS int CHECK (VALUE > 0);" +COLS="id int, tz timetz, mac macaddr, mac8 macaddr8, nm name, pt point, ch \"char\", + tm time, mo money, ip inet, cd cidr, bt bit(16), vb varbit, tsv tsvector, x xml, + j json, e tm_mood, cmp tm_pair, dom tm_pos, n numeric, tsi timestamp, d date" +# xml needs a server built with libxml. Without it the load below fails and every +# arm after it would fail for that reason, so the suite says so and stops +# (OffgridwithJD, review of #1426). +xml_ok="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres -d "$PGC_DB" -XAtq \ + -c "SELECT ''::xml IS NOT NULL" 2>&1 | tail -1)" # 2>&1, so a missing libxml is named +check "premise: the server supports xml (built with libxml)" "$([ "$xml_ok" = t ] && echo yes || echo "no ($xml_ok)")" "yes" +[ "$xml_ok" = t ] || pgc_summary +make_pair "$COLS" +q "SELECT pgcolumnar.set_options('t_col', stripe_row_limit => 1000)" >/dev/null +# g % k = 0 makes the column NULL; each column has its own k. +load_pair "SELECT g, + CASE WHEN g % 7 = 0 THEN NULL ELSE ((('00:00'::time + (g * 7) * interval '1 s')::text || '+' || lpad((g % 13)::text, 2, '0'))::timetz) END, + CASE WHEN g % 11 = 0 THEN NULL ELSE ('08:00:2b:' || lpad(to_hex(g / 256 % 256), 2, '0') || ':' || lpad(to_hex(g % 256), 2, '0') || ':01')::macaddr END, + CASE WHEN g % 13 = 0 THEN NULL ELSE ('08:00:2b:01:02:03:' || lpad(to_hex(g / 256 % 256), 2, '0') || ':' || lpad(to_hex(g % 256), 2, '0'))::macaddr8 END, + CASE WHEN g % 17 = 0 THEN NULL ELSE ('name_' || lpad(g::text, 6, '0'))::name END, + CASE WHEN g % 19 = 0 THEN NULL ELSE point(g, -g) END, + CASE WHEN g % 23 = 0 THEN NULL ELSE chr(65 + g % 26)::\"char\" END, + CASE WHEN g % 29 = 0 THEN NULL ELSE ('00:00'::time + g * interval '7 s') END, + CASE WHEN g % 31 = 0 THEN NULL ELSE (g * 1.25)::numeric::money END, + CASE WHEN g % 37 = 0 THEN NULL ELSE ('10.' || g / 256 % 256 || '.' || g % 256 || '.' || g % 7 || '/32')::inet END, + CASE WHEN g % 41 = 0 THEN NULL ELSE ('10.' || g / 256 % 256 || '.' || g % 256 || '.0/24')::cidr END, + CASE WHEN g % 43 = 0 THEN NULL ELSE g::bit(16) END, + CASE WHEN g % 47 = 0 THEN NULL ELSE (g::bit(16) || (g % 7)::bit(3))::varbit END, + CASE WHEN g % 53 = 0 THEN NULL ELSE to_tsvector('simple', 'word' || g || ' common term' || g % 9) END, + CASE WHEN g % 59 = 0 THEN NULL ELSE ('' || md5(g::text) || '')::xml END, + CASE WHEN g % 61 = 0 THEN NULL ELSE json_build_object('g', g, 's', md5(g::text)) END, + CASE WHEN g % 67 = 0 THEN NULL ELSE (enum_range(NULL::tm_mood))[1 + g % 4] END, + CASE WHEN g % 71 = 0 THEN NULL ELSE ROW(g, 'p' || g)::tm_pair END, + CASE WHEN g % 73 = 0 THEN NULL ELSE (g)::tm_pos END, + CASE WHEN g % 79 = 0 THEN NULL WHEN g % 1000 = 1 THEN 'NaN'::numeric WHEN g % 1000 = 2 THEN 'Infinity'::numeric + WHEN g % 1000 = 3 THEN '-Infinity'::numeric ELSE g / 7.0 END, + CASE WHEN g % 83 = 0 THEN NULL WHEN g = 500 THEN 'infinity'::timestamp WHEN g = 501 THEN '-infinity'::timestamp + ELSE '2026-01-01'::timestamp + g * interval '1 min' END, + CASE WHEN g % 89 = 0 THEN NULL WHEN g = 600 THEN 'infinity'::date WHEN g = 601 THEN '-infinity'::date + ELSE '2026-01-01'::date + g END + FROM generate_series(1, 12000) g" + +check "premise: both tables hold 12000 rows, the columnar one in at least twelve row groups" \ + "$(q "SELECT count(*) FROM t_heap")|$(q "SELECT count(*) FROM t_col")|$(q "SELECT CASE WHEN count(*) >= 12 THEN 'many' ELSE count(*)::text END + FROM pgcolumnar.row_group WHERE storage_id = pgcolumnar.get_storage_id('t_col')" 2>/dev/null | tail -1)" "12000|12000|many" + +ALL="id, tz::text, mac::text, mac8::text, nm::text, pt::text, ch::text, tm::text, mo::text, ip::text, cd::text, + bt::text, vb::text, tsv::text, x::text, j::text, e::text, cmp::text, dom::text, n::text, tsi::text, d::text" +diff_query "every row, as text, is heap's" "SELECT $ALL FROM %T" +parts=() +for c in tz mac mac8 nm pt ch tm mo ip cd bt vb tsv x j e cmp dom n tsi d; do + parts+=("count($c), count(*) FILTER (WHERE $c IS NULL)") +done +COUNTS="$(IFS=,; echo "${parts[*]}")" +diff_query "each column's count and NULL count is heap's" "SELECT $COUNTS FROM %T" + +# A range on every ordered column. The bounds are the values heap holds at ids 3005 +# and 9005, as literals, so the columnar scan can prune row groups by zone map. Not +# 3001 and 9001: id 3001 holds n's NaN, which made n's "range" NaN..NaN (OffgridwithJD, +# review of #1426). Neither id is NULL in any column. +# +# Each arm also says whether the zone maps skipped chunk groups, from EXPLAIN ANALYZE, +# so an arm that should prune cannot pass on a scan that read everything. Four measure +# 0 of 12 on PG17 and PG18, and are pinned at 0 with the reason: +declare -A NO_PRUNE=( + [ch]="every row group holds every value, since the column cycles through 26 letters" + [e]="every row group holds every value, since the column cycles through the 4 labels" + [n]="every row group holds NaN and both infinities, so each zone map spans every value" + [cd]="a cidr key is treated as cross-type against inet and dropped (#1433)") +removed() { # removed SQL -> the columnar scan's "Chunk Groups Removed by Filter" + q "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) ${1//%T/t_col}" | + sed -n 's/.*Columnar Chunk Groups Removed by Filter: \([0-9]*\).*/\1/p' | head -1 +} +for c in tz mac mac8 nm ch tm mo ip cd bt vb e dom n tsi d; do + # format_type keeps the typmod: pg_typeof would make bt's bounds bit(1) + # (OffgridwithJD, review of #1426). + ty="$(q "SELECT format_type(atttypid, atttypmod) FROM pg_attribute WHERE attrelid = 't_heap'::regclass AND attname = '$c'" | tail -1)" + lo="$(q "SELECT quote_literal($c::text) || '::' || '$ty' FROM t_heap WHERE id = 3005" | tail -1)" + hi="$(q "SELECT quote_literal($c::text) || '::' || '$ty' FROM t_heap WHERE id = 9005" | tail -1)" + sql="SELECT id FROM %T WHERE $c BETWEEN least($lo, $hi) AND greatest($lo, $hi)" + diff_query "a range on $c finds heap's rows" "$sql" + r="$(removed "$sql")" + if [ -n "${NO_PRUNE[$c]:-}" ]; then + check "a range on $c skips no chunk groups: ${NO_PRUNE[$c]}" "${r:-none}" "0" + else + check "a range on $c lets the zone maps skip chunk groups" \ + "$([ -n "$r" ] && [ "$r" -ge 1 ] && echo skips || echo "removed ${r:-none}")" "skips" + fi +done +diff_query "equality on json, as text, finds heap's rows" \ + "SELECT id FROM %T WHERE j::text = (SELECT j::text FROM t_heap WHERE id = 4242)" +diff_query "point equality (~=) finds heap's rows" "SELECT id FROM %T WHERE pt ~= point(4242, -4242)" +diff_query "a tsvector match finds heap's rows" "SELECT id FROM %T WHERE tsv @@ to_tsquery('simple', 'term3')" +diff_query "the aggregates over numeric NaN and Infinity and timestamp and date infinity are heap's" \ + "SELECT min(n)::text, max(n)::text, count(*) FILTER (WHERE n = 'NaN'), count(*) FILTER (WHERE n = 'Infinity'), + count(*) FILTER (WHERE n = '-Infinity'), min(tsi)::text, max(tsi)::text, min(d)::text, max(d)::text FROM %T" +diff_query "the composite's fields and the enum's order read back as heap's" \ + "SELECT (cmp).a, (cmp).b, e, e > 'ok' FROM %T WHERE id % 97 = 0" + +pgc_summary From c1441b4352b8bec546361eb329953772ba643aed Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 23:07:03 +0000 Subject: [PATCH 2/4] test: each type_matrix range arm requires heap's answer to hold rows, so bounds that come out wrong cannot pass on nothing (#1341) Review of #1426 by OffgridwithJD: with the old pg_typeof bounds put back, bt's range became BETWEEN '0' AND '0', found no rows on either table, and both its arms passed on two empty answers. Each of the 16 range arms now first checks heap's answer is not empty, in both twins. That mutation turns bt's new arm red (0 rows), and nothing else. Co-Authored-By: Claude Opus 5.5 --- test/check_ledger.tsv | 16 ++++++++++++++++ test/check_ledger_budget.txt | 15 +++++++++------ test/pytest/test_type_matrix.py | 4 ++++ test/type_matrix.sh | 6 ++++++ 4 files changed, 35 insertions(+), 6 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 25931eddd..b69d6584f 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -3326,36 +3326,52 @@ ttl_expire ttl_expire with the extent, expire retires the same groups 15;16;17;1 ttl_expire ttl_expire without the extent the same expire probes every group's zone map 15;16;17;18;19 never - ttl_expire ttl_expire without the extent, expire retires the aged groups 15;16;17;18;19 2026-10-14 expire returning 0 whatever it retired type_matrix type_matrix a range on bt finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on bt holds rows on heap, so its comparison reads something 15;16;17;18;19 2026-10-15 the range bounds cast with pg_typeof, which drops the typmod, so bt's become bit(1) and its range finds no rows type_matrix type_matrix a range on bt lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on cd finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on cd holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on cd skips no chunk groups: a cidr key is treated as cross-type against inet and dropped (#1433) 15;16;17;18;19 never - type_matrix type_matrix a range on ch finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on ch holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on ch skips no chunk groups: every row group holds every value, since the column cycles through 26 letters 15;16;17;18;19 never - type_matrix type_matrix a range on d finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on d holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on d lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on dom finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on dom holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on dom lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on e finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on e holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on e skips no chunk groups: every row group holds every value, since the column cycles through the 4 labels 15;16;17;18;19 never - type_matrix type_matrix a range on ip finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on ip holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on ip lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on mac finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mac holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on mac lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on mac8 finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mac8 holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on mac8 lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on mo finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on mo holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on mo lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on n finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on n holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on n skips no chunk groups: every row group holds NaN and both infinities, so each zone map spans every value 15;16;17;18;19 never - type_matrix type_matrix a range on nm finds heap's rows 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix a range on nm holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on nm lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen);the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on tm finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on tm holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on tm lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on tsi finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on tsi holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on tsi lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on tz finds heap's rows 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) +type_matrix type_matrix a range on tz holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on tz lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a range on vb finds heap's rows 15;16;17;18;19 never - +type_matrix type_matrix a range on vb holds rows on heap, so its comparison reads something 15;16;17;18;19 never - type_matrix type_matrix a range on vb lets the zone maps skip chunk groups 15;16;17;18;19 2026-10-15 the scan never skips a row group by zone map (pgcolumnar_native_group_can_match ignored) type_matrix type_matrix a tsvector match finds heap's rows 15;16;17;18;19 never - type_matrix type_matrix each column's count and NULL count is heap's 15;16;17;18;19 2026-10-15 a by-reference fixed-width value over 8 bytes is stored as its first 8 bytes and zero padding (PgColumnarEncodeValueByLen) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index ad80824b3..86492b7ca 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2762,7 +2762,7 @@ suites_not_covered 224 # 2557 + 847 == 3404 # # #1341, types the differential matrix never stored answer as heap (jdatcmd): -# 2557 -> 2581. type_matrix enters with all 41 checks from five-major runs (41/41 on +# 2557 -> 2596. type_matrix enters with all 57 checks from five-major runs (57/57 on # each of 15-19). 17 are dated on PG17 by two mutants (OffgridwithJD, reviews of # #1426): # - a by-reference fixed-width value over 8 bytes stored as its first 8 bytes and @@ -2770,11 +2770,14 @@ suites_not_covered 224 # bytes 8-11 matter), the nm range and its pruning, and point equality; # - the scan never skipping a row group by zone map: the 12 range arms that must # skip chunk groups, varbit's included now that its values increase with the row. +# A third, the range bounds cast with pg_typeof (which drops the typmod), dates bt's +# heap-not-empty arm: its range then finds no rows. Each of the 16 range arms has +# one, so no comparison can pass on two empty answers (review of #1426). # The rest arrive never, among them the four range arms pinned at 0 skipped groups, # with the reason in each name (cidr's names #1433), and the libxml premise. # -# ledger rows 3404 + 41 = 3445 -# never 2557 + 24 = 2581 -# dated 847 + 17 = 864 -# 2581 + 864 == 3445 -checks_never_observed_red 2581 +# ledger rows 3404 + 57 = 3461 +# never 2557 + 39 = 2596 +# dated 847 + 18 = 865 +# 2596 + 865 == 3461 +checks_never_observed_red 2596 diff --git a/test/pytest/test_type_matrix.py b/test/pytest/test_type_matrix.py index 9b72e510e..999c7bd23 100644 --- a/test/pytest/test_type_matrix.py +++ b/test/pytest/test_type_matrix.py @@ -118,6 +118,10 @@ def test_types_the_matrix_never_stored_answer_as_heap(pgc_conn, expect): lo, hi = (_rows(pgc_conn, f"SELECT quote_literal({c}::text) || '::' || '{ty}' " f"FROM t_heap WHERE id = {i}")[0][0] for i in (3005, 9005)) sql = f"SELECT id FROM %T WHERE {c} BETWEEN least({lo}, {hi}) AND greatest({lo}, {hi})" + # Heap's answer must not be empty: wrong bounds would make both answers empty. + n = _rows(pgc_conn, f"SELECT count(*) FROM ({sql.replace('%T', 't_heap')}) x")[0][0] + expect.text("yes" if n > 0 else f"no ({n} rows)", "yes", + f"a range on {c} holds rows on heap, so its comparison reads something") got, want = _both(pgc_conn, sql) expect.row_set(got, want, f"a range on {c} finds heap's rows") r = _removed(pgc_conn, sql) diff --git a/test/type_matrix.sh b/test/type_matrix.sh index 3c660af2b..392f2dc4e 100755 --- a/test/type_matrix.sh +++ b/test/type_matrix.sh @@ -106,6 +106,12 @@ for c in tz mac mac8 nm ch tm mo ip cd bt vb e dom n tsi d; do lo="$(q "SELECT quote_literal($c::text) || '::' || '$ty' FROM t_heap WHERE id = 3005" | tail -1)" hi="$(q "SELECT quote_literal($c::text) || '::' || '$ty' FROM t_heap WHERE id = 9005" | tail -1)" sql="SELECT id FROM %T WHERE $c BETWEEN least($lo, $hi) AND greatest($lo, $hi)" + # Heap's answer must not be empty: bounds that came out wrong (a lost typmod made + # bt's BETWEEN '0' AND '0') would make both answers empty, and the comparison and + # the skip arm would pass on nothing (OffgridwithJD, review of #1426). + n="$(q "SELECT count(*) FROM (${sql//%T/t_heap}) x" | tail -1)" + check "a range on $c holds rows on heap, so its comparison reads something" \ + "$([ -n "$n" ] && [ "$n" -gt 0 ] && echo yes || echo "no (${n:-none} rows)")" "yes" diff_query "a range on $c finds heap's rows" "$sql" r="$(removed "$sql")" if [ -n "${NO_PRUNE[$c]:-}" ]; then From 3dac39b1980bb106ccdc68a42c8013146720244b Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 22:56:42 +0000 Subject: [PATCH 3/4] test: an ungrouped aggregate applies a WHERE that references no column, red first (#1435) Three gating WHEREs, (SELECT false), now() < '2000-01-01' and current_user = 'pgc_nobody', each against heap, with a premise that the same aggregate with no WHERE is the vectorized node. On main each returns every row's aggregates (10000/1/10000 in the shell, 5000/1/5000 in the twin) where heap gives 0/-/-. Co-Authored-By: Claude Opus 5.5 --- test/native_agg.sh | 14 ++++++++++++++ test/pytest/test_native_agg.py | 26 ++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/test/native_agg.sh b/test/native_agg.sh index 299d955fa..bd738b2a4 100755 --- a/test/native_agg.sh +++ b/test/native_agg.sh @@ -76,6 +76,20 @@ check "count(*) uses the metadata agg node" \ check "sum/min/max uses the metadata agg node" \ "$(has_aggnode "$(plan 'SELECT sum(id), min(id), max(id) FROM n')")" "yes" +# A WHERE that references no column is a gating qual: planned as a one-time +# filter, not part of the per-row quals. The node answered from the zone maps alone +# and ignored it, so a false gate returned every row's aggregates (#1435). +GATED="SELECT count(*) || '/' || coalesce(min(id)::text, '-') || '/' || coalesce(max(id)::text, '-') FROM" +check "premise: count, min and max with no WHERE are the vectorized aggregate" \ + "$(has_aggnode "$(plan 'SELECT count(*), min(id), max(id) FROM n')")" "yes" +check "a WHERE (SELECT false) with no column in it gives heap's answer" \ + "$(q "$GATED n WHERE (SELECT false)")" "$(q "$GATED h WHERE (SELECT false)")" +check "a WHERE on now() with no column in it gives heap's answer" \ + "$(q "$GATED n WHERE now() < '2000-01-01'")" "$(q "$GATED h WHERE now() < '2000-01-01'")" +check "a WHERE on current_user with no column in it gives heap's answer" \ + "$(q "$GATED n WHERE current_user = 'pgc_nobody'")" "$(q "$GATED h WHERE current_user = 'pgc_nobody'")" +check "premise: heap's answer to a false gate is no rows" "$(q "$GATED h WHERE (SELECT false)")" "0/-/-" + # Supported aggregates answered from zone maps must equal the heap oracle. check "count(*)" "$(q 'SELECT count(*) FROM n;')" "$(q 'SELECT count(*) FROM h;')" check "count(col,nulls)" "$(q 'SELECT count(s) FROM n;')" "$(q 'SELECT count(s) FROM h;')" diff --git a/test/pytest/test_native_agg.py b/test/pytest/test_native_agg.py index 276f81762..bad469083 100644 --- a/test/pytest/test_native_agg.py +++ b/test/pytest/test_native_agg.py @@ -116,3 +116,29 @@ def both(sql): expect.text(got, want, "max(timestamptz) parity after deleting the newest rows") got, want = both("SELECT min(at) FROM {t}") expect.text(got, want, "min(timestamptz) parity after deleting the oldest rows") + + +GATED = "SELECT count(*) || '/' || coalesce(min(id)::text, '-') || '/' || coalesce(max(id)::text, '-') FROM" + + +def test_a_where_with_no_column_in_it_is_applied(pgc_conn, expect): + """A WHERE that references no column is a gating qual: planned as a one-time + filter, not part of the per-row quals. The ungrouped vectorized aggregate + answered from the zone maps alone and ignored it, so a false gate returned every + row's aggregates (#1435). Each arm is compared with heap.""" + with pgc_conn.cursor() as cur: + cur.execute("CREATE TABLE gn (id int) USING pgcolumnar") + cur.execute("CREATE TABLE gh (id int) USING heap") + cur.execute("SELECT pgcolumnar.set_options('gn', stripe_row_limit => 1000)") + cur.execute("INSERT INTO gn SELECT g FROM generate_series(1, 5000) g") + cur.execute("INSERT INTO gh SELECT g FROM generate_series(1, 5000) g") + cur.execute("EXPLAIN (COSTS OFF) SELECT count(*), min(id), max(id) FROM gn") + plan = "\n".join(r[0] for r in cur.fetchall()) + expect.text("yes" if "Columnar Vectorized Aggregates" in plan else f"no ({plan})", "yes", + "premise: count, min and max with no WHERE are the vectorized aggregate") + for name, where in (("(SELECT false)", "(SELECT false)"), ("now()", "now() < '2000-01-01'"), + ("current_user", "current_user = 'pgc_nobody'")): + expect.text(_one(pgc_conn, f"{GATED} gn WHERE {where}"), _one(pgc_conn, f"{GATED} gh WHERE {where}"), + f"a WHERE {'' if name.startswith('(') else 'on '}{name} with no column in it gives heap's answer") + expect.text(_one(pgc_conn, f"{GATED} gh WHERE (SELECT false)"), "0/-/-", + "premise: heap's answer to a false gate is no rows") From 3e0a7c0dd3ecac8c6dd15504e3a78d220d2716ec Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 15 Oct 2026 22:56:42 +0000 Subject: [PATCH 4/4] fix: the ungrouped vectorized aggregate refuses a query with a gating qual (#1435) A WHERE that references no column is a pseudoconstant qual, which extract_actual_clauses(..., false) leaves out of the node's quals. With no other qual, needsScan was false, the node answered from the zone maps, and the gating refusal in the scan-fold branch was never reached, so a false gate returned every row's aggregates. The refusal now runs before needsScan is decided, as the grouped node's does. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 ++++++ src/columnar_vector.c | 15 +++++++++++++++ test/check_ledger.tsv | 5 +++++ test/check_ledger_budget.txt | 13 ++++++++++++- test/pytest/TESTS.md | 1 + test/pytest/expected_tests.txt | 4 +++- 6 files changed, 42 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7db731a92..2f86e386f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -416,6 +416,12 @@ true until the next version shipped. over every row, so a caller limited by a policy could read values from rows it cannot see. It now raises `row-level security is in force`, as the other functions that read storage directly do. Found by OffgridwithJD in review. +- **An ungrouped aggregate on a columnar table applies a WHERE that references no + column.** `SELECT count(*), min(v) FROM t WHERE (SELECT false)`, or a WHERE on + `now()` or `current_user`, returned every row's aggregates instead of none. The + vectorized aggregate answered from the zone maps and never applied the one-time + filter. It now leaves such a query to the ordinary plan. Found by OffgridwithJD in + review (#1435). `test/native_agg.sh` and `test_native_agg.py`. - **A concurrent `UPDATE` of the same row now fails with 40001 on a table with a unique index, as documented, not 23505 (#1421).** Two sessions updating one row of diff --git a/src/columnar_vector.c b/src/columnar_vector.c index 3eb72b5b9..87b795118 100644 --- a/src/columnar_vector.c +++ b/src/columnar_vector.c @@ -1562,6 +1562,21 @@ PgColumnarCreateUpperPaths(PlannerInfo *root, UpperRelationKind stage, * zone-map answerable. A filter, or a sum/avg over int8/float/numeric, needs * the scan-fold path instead (#289). */ + /* + * A pseudoconstant (gating) qual -- a WHERE that references no column, such as + * (SELECT false) or now() < '2000-01-01' -- is a one-time filter, and + * extract_actual_clauses(..., false) leaves it out of quals. Refused HERE, + * before needsScan is decided: with no other qual the node answers from the + * zone maps and never reaches the refusal in the scan-fold branch below, so a + * false gate returned every row's aggregates (#1435). + */ + { + ListCell *gc; + + foreach(gc, factRel->baserestrictinfo) + if (lfirst_node(RestrictInfo, gc)->pseudoconstant) + return; + } if (pgcolumnar_quals_could_leak(factRel->baserestrictinfo)) return; quals = extract_actual_clauses(factRel->baserestrictinfo, false); diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index b69d6584f..7ec113978 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -2186,6 +2186,9 @@ maintenance_rollback maintenance_rollback premise: the table holds the rows it w maintenance_rollback maintenance_rollback recluster changed the row groups inside its transaction 15;16;17;18;19 never - maintenance_rollback maintenance_rollback vacuum changed the row groups inside its transaction 15;16;17;18;19 never - maintenance_rollback maintenance_rollback vacuum_sorted changed the row groups inside its transaction 15;16;17;18;19 never - +native_agg native_agg a WHERE (SELECT false) with no column in it gives heap's answer 15;16;17;18;19 2026-10-15 the ungrouped vectorized aggregate takes a query with a gating qual and answers from the zone maps (the tree before #1435) +native_agg native_agg a WHERE on current_user with no column in it gives heap's answer 15;16;17;18;19 2026-10-15 the ungrouped vectorized aggregate takes a query with a gating qual and answers from the zone maps (the tree before #1435) +native_agg native_agg a WHERE on now() with no column in it gives heap's answer 15;16;17;18;19 2026-10-15 the ungrouped vectorized aggregate takes a query with a gating qual and answers from the zone maps (the tree before #1435) native_agg native_agg an aggregate that needs zone maps reads them 15;16;17;18;19 never - native_agg native_agg avg(int4,nulls) 15;16;17;18;19 never - native_agg native_agg control: decoding ts reads the table's blocks 15;16;17;18;19 never - @@ -2215,6 +2218,8 @@ native_agg native_agg min(ts) reads under a tenth of the blocks a decode of ts r native_agg native_agg multi-agg 15;16;17;18;19 never - native_agg native_agg multi-agg beats a parallel plan 15;16;17;18;19 never - native_agg native_agg multi-agg plan has no Gather 15;16;17;18;19 never - +native_agg native_agg premise: count, min and max with no WHERE are the vectorized aggregate 15;16;17;18;19 never - +native_agg native_agg premise: heap's answer to a false gate is no rows 15;16;17;18;19 never - native_agg native_agg premise: the timestamptz table holds every row inserted 15;16;17;18;19 never - native_agg native_agg sum(bigint) fallback parity 15;16;17;18;19 never - native_agg native_agg sum(int4) 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 86492b7ca..8bbe8de29 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -2780,4 +2780,15 @@ suites_not_covered 224 # never 2557 + 39 = 2596 # dated 847 + 18 = 865 # 2596 + 865 == 3461 -checks_never_observed_red 2596 +# +# #1435, the ungrouped vectorized aggregate ignored a WHERE that references no column +# (jdatcmd): 2596 -> 2598. native_agg gains five checks from five-major runs (40/40 on +# each of 15-19). The three gating arms are dated on PG17 by the tree before the fix, +# where each false gate returned every row's aggregates (10000/1/10000 against +# heap's 0/-/-); the two premises arrive never. +# +# ledger rows 3461 + 5 = 3466 +# never 2596 + 2 = 2598 +# dated 865 + 3 = 868 +# 2598 + 868 == 3466 +checks_never_observed_red 2598 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index e3fb22797..314004377 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6487,6 +6487,7 @@ Not ported, and declared INCOMPLETE in `test_compare_to_bash.py`: the `int4`/`te | test | what it holds | | --- | --- | | `test_min_max_over_timestamptz_read_no_data` | the table holds every row; `max(ts)` plans the metadata node; a decode reads at least 10 blocks; `max` and `min` each read under a tenth of that; parity with heap before and after deleting rows at both ends | +| `test_a_where_with_no_column_in_it_is_applied` | a WHERE that references no column, such as `(SELECT false)`, `now() < ...` or `current_user = ...`, gives heap's answer, with a premise that the same aggregate with no WHERE is the vectorized node (#1435) | ## 86. test_format_downgrade.py: an alpha4 binary refuses what this build wrote diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 968ada983..f1b0fbcf0 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -725,4 +725,6 @@ guard_tests 438 # cluster_tests 600 -> 601 for #1341's test_rls_query_paths.py, one test, by collection # over the cluster files on this tree. # cluster_tests 601 -> 602 for #1341's test_type_matrix.py, one test, by collection -cluster_tests 602 +# cluster_tests 602 -> 603 for #1435's test_a_where_with_no_column_in_it_is_applied in +# test_native_agg.py, by collection over the cluster files on this tree. +cluster_tests 603