Skip to content

fix: an ungrouped aggregate applies a WHERE that references no column (#1435) - #1436

Merged
jdatcmd merged 4 commits into
mainfrom
fix/1435-gating-quals
Oct 15, 2026
Merged

jdatcmd merged 4 commits into
mainfrom
fix/1435-gating-quals

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1435, found by @OffgridwithJD reviewing #1429.

The bug: a wrong answer

On a columnar table, an ungrouped aggregate ignored a WHERE that references no column:

query columnar before heap
SELECT count(*), min(id), max(id) FROM t WHERE (SELECT false) every row's aggregates 0 rows
... WHERE now() < '2000-01-01' every row's aggregates 0 rows
... WHERE current_user = 'x' every row's aggregates 0 rows

Cause: a column-free WHERE is a pseudoconstant (gating) qual, and extract_actual_clauses(..., false) leaves it out of the node's quals. With no other qual, needsScan was false and the node answered from the zone maps. The node's own gating refusal sits in the scan-fold branch, which this case never reached.

The fix

In src/columnar_vector.c, the ungrouped node refuses a gating qual before needsScan is decided, as the grouped node does. The query goes to the ordinary plan, which applies the one-time filter. A query with no WHERE, or with a WHERE on a column, is unchanged.

Tests (red first)

test/native_agg.sh and a new test_a_where_with_no_column_in_it_is_applied in test_native_agg.py:

  • each of the three gating WHEREs gives heap's answer;
  • a premise that the same aggregate with no WHERE is the vectorized node, so the arms are aimed at it;
  • a premise that heap's answer to a false gate is 0/-/-, so the comparison can't pass on two wrong answers.

On main the three arms are red in both twins: got [10000/1/10000] want [0/-/-] in the shell, and got '5000/1/5000' in the twin.

Evidence

  • Green: native_agg 40/40 and the twin 15 passed, on each of PG15 to PG19, with no build warnings. On PG17, native_groupagg 94/94, int8_agg_int128 34/34 and batch_fold_explain 7/7.
  • Ledger: native_agg gains 5 rows. The three gating arms are dated by the tree before the fix; the two premises arrive never. The census is 3380 + 5 = 3385 rows, 2546 never, 839 dated, matching the tsv. The gate gives rc=0, cluster_tests is 601 by collection, the guard set passes 438/438, and docs_style passes.
  • Two commits: the tests (red on main), then the fix, its bookkeeping and the CHANGELOG entry.

🤖 Generated with Claude Code

The encoder has fast paths for fixed widths 1, 2, 4 and 8, and every
other width and every varlena takes a generic path. A new suite,
type_matrix, and its pytest twin store what no test had:

- by-reference fixed widths 6, 8, 12, 16 and 64: macaddr, macaddr8,
  timetz, point, name;
- the 1-byte "char";
- time, money, inet, cidr, bit, varbit, tsvector, xml, json;
- an enum, a composite, a domain;
- numeric NaN and +-Infinity, timestamp and date +-infinity.

12000 rows in row groups of 1000, each column NULL on its own pattern,
are generated once into heap and copied. Against heap: every row as text,
each column's counts, a range on all sixteen ordered columns with literal
bounds so zone maps can prune, json, point and tsvector matches, the
special-value aggregates, and the composite's fields.

24/24 on each of PG15-19, and the twin on each. Five checks are dated on
PG17 by a mutant that stores a by-reference value over 8 bytes as its
first 8 bytes. cluster_tests 565 -> 566.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N

Squashed for the rebase onto main; it replaces:
- test: type_matrix's pinned cidr arm names #1433, the cause OffgridwithJD found (#1341)
- test: type_matrix's rows re-dated, restated on main (#1341)
- test: type_matrix varies timetz's zone, asserts each range arm's pruning, and names a missing libxml (#1341)
- test: types the differential matrix never stored answer as heap (#1341)

Squashed for the rebase; it replaces:
- test: type_matrix's rows re-dated, restated on main 6f0bd12 (#1341)
- test: type_matrix's bit and varbit columns increase with the row, its bounds keep their typmod, and vb must skip (#1341)
- test: types the differential matrix never stored answer as heap (#1341)

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approved at 01640fd.

Red first, then green, on PG18:

tree native_agg.sh test_native_agg.py
bc40b67 (the tests only) 37/40: the three gating arms red, got [10000/1/10000] want [0/-/-] 1 failed, 1 passed: got '5000/1/5000' want '0/-/-'
01640fd (the fix) 40/40 2 passed

The .so differs between the two runs (9c1993a0 and 58d3e7f6).

Probes against heap on the fixed build, 200000 rows, with count/min/max/sum compared each time:

  • Gates: a false gate, a true gate (200000/1/200000/20000100000 on both sides) and a STABLE current_setting gate.
  • A gate beside a column qual: v > 10 AND (SELECT false), and the same with a true now() gate.
  • Other plan shapes: GROUP BY under a false gate (0 groups), HAVING (SELECT false) (0 rows), and a prepared WHERE $1 with false, both custom and forced-generic (0).
  • Settings: enable_ungrouped_vector_agg = on, and forced parallel with 4 workers, for false and true gates and GROUP BY. The parallel plan is Gather -> Partial Aggregate -> Result (One-Time Filter) -> Parallel Custom Scan.

All agree with heap. A query with no WHERE still plans Columnar Vectorized Aggregates: 3, so the refusal is no wider than the gate. Returning before any ungrouped path is built is the right place for it.

Found while probing, a separate bug, on main: #1437. Any join with a columnar fact and a gating qual fails with pgcolumnar runtime filter expected a core Hash Join, true gates and EXPLAIN included. enable_join_runtime_filter = off avoids it. It doesn't block this PR, which doesn't touch that path.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, this is now 79f514a, stacked on #1426 (996413c), which is on #1429, so the three merge in order without restating.

  • One real conflict: columnar_vector.c, where test: row-level security holds on every columnar read path (#1341) #1429's leak check and this fix both go right before the ungrouped node builds its quals. I kept both, the gating refusal first.
  • On the combined tree (PG17): native_agg 40/40, rls_query_paths 24/24, native_groupagg 94/94, type_matrix 41/41, and the twins pass.
  • Bookkeeping: the census is 3450 / 2583 / 867 by the tsv, cluster_tests is 603, and the gate gives rc=0. The two commits are kept, so red first still shows.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approved at 79f514a, stacked on #1426 (996413c).

  • The patch: identical to 01640fd's apart from one blank line.
  • The conflict: resolved as it should be. In columnar_vector.c the gating refusal comes first and #1429's pgcolumnar_quals_could_leak check follows it, and both just return.
  • The combined tree on PG17: native_agg 40/40 and rls_query_paths 24/24 in the shell, and both twins (test_native_agg.py and test_rls_query_paths.py) pass.

My PG18 red-first runs and probes from the 01640fd approval still cover the fix.

🤖 Generated with Claude Code

jdatcmd and others added 3 commits October 15, 2026 23:07
… so bounds that come out wrong cannot pass on nothing (#1341)

Review of #1426 by OffgridwithJD: with the old pg_typeof bounds put back, bt's range
became BETWEEN '0' AND '0', found no rows on either table, and both its arms passed
on two empty answers. Each of the 16 range arms now first checks heap's answer is not
empty, in both twins. That mutation turns bt's new arm red (0 rows), and nothing
else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n, red first (#1435)

Three gating WHEREs, (SELECT false), now() < '2000-01-01' and current_user =
'pgc_nobody', each against heap, with a premise that the same aggregate with no
WHERE is the vectorized node. On main each returns every row's aggregates
(10000/1/10000 in the shell, 5000/1/5000 in the twin) where heap gives 0/-/-.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… qual (#1435)

A WHERE that references no column is a pseudoconstant qual, which
extract_actual_clauses(..., false) leaves out of the node's quals. With no other
qual, needsScan was false, the node answered from the zone maps, and the gating
refusal in the scan-fold branch was never reached, so a false gate returned every
row's aggregates. The refusal now runs before needsScan is decided, as the grouped
node's does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd

jdatcmd commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD, restated as 3e0a7c0 onto #1426's new head (c1441b4), which added the heap-not-empty arms. Only the budget conflicted, and it's re-derived (3466 rows, 2598 never, 868 dated, matching the tsv). The patch is identical to 79f514a, which you approved, outside the budget and expected_tests.txt. cluster_tests is 603 by collection, and the gate gives rc=0.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd force-pushed the fix/1435-gating-quals branch from 79f514a to 3e0a7c0 Compare October 15, 2026 23:08

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approved at 3e0a7c0, on #1426's c1441b4.

  • The patch: the src, the tests and the CHANGELOG are identical to 79f514a's, and only the budget was re-derived.
  • Runs: my PG17 run of the combined tree and the PG18 red-first runs still cover it, since c1441b4 only adds type_matrix arms.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The ungrouped zone-map aggregate ignores a WHERE that references no column, and returns rows heap does not

2 participants