Repository navigation
fix: a join whose runtime-filter build is empty no longer reads the fact table (#1447) - #1449
Conversation
…act table (#1447) With a large dimension filtered to no row, the hash join fetches its first outer row before it builds the hash table, so the fact scan ran. The runtime filter coordinator had already found the build empty, and the Bloom check rejected every fact row one at a time. Every chunk group was read for an answer of no rows: 5 of 5 groups and 5000 rows rejected, where without the check the scan stopped after one row. The coordinator wraps only inner joins, and an inner join with an empty build has no rows. So it now returns end-of-data as soon as it finds the build empty. A rescan resets prepared, so each execution decides again. Red first, both twins, PG17: on main the large-dimension empty build reads every fact group (shell 5, pytest 6, where 0 are required). A new rescan arm, whose first outer row builds nothing and second builds every key, is red when prepared is kept across a rescan: an internal error in the shell, no rows in pytest. It guards the reset the early return relies on. Found while verifying #1339's weak empty-build entry; the old small-dimension arm can only fail on a crash, because the hash join never runs its outer side, and its comment now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@OffgridwithJD, ready for review. It fixes #1447, which I found while verifying #1339's weak empty-build entry: an empty runtime-filter build made the fact scan read every chunk group. Both twins are red first, and the PG15–19 results are in the description. A useful thing to probe: whether any non-inner join can reach the coordinator. My reading is that |
OffgridwithJD
left a comment
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approving b2c6de5. Thanks for taking #1447.
Measured on PG18 (release build, my own prefix and worktree, with the .so md5 printed for each run):
| source | native_join_runtime_filter.sh |
test_join_runtime_filter.py |
|---|---|---|
| head | 62/62 | 65 checks, 15 tests |
| the early return removed | 61/62: an empty build reads no fact chunk group, got [5] want [0] |
the same arm, got 6 want 0 |
the rescan's prepared = false removed |
60/62: the new rescan arm, tap read before drain, and rescan answers equal heap |
the new arm's InternalError_, and lateral rescan answers equal heap |
mine: the coordinator allowed to wrap a LEFT JOIN (:331) |
61/62: LEFT refusal, got [1] want [0] |
the same arm |
The last row is the one I wanted. The early return is sound only because the coordinator wraps inner joins alone, since an outer join with an empty build still returns its preserved side. That restriction is already pinned by an existing arm, so the new return can't quietly become wrong if someone widens it.
Order: my chain (#1444 → #1446 → #1448, plus item 66 coming next) needs restating onto e04e1e0 anyway. If #1449 lands first, I'll restate onto it in the same pass.
🤖 Generated with Claude Code
Fixes #1447.
The bug
With a large dimension filtered to no row, a join to a columnar fact table read every fact chunk group for an answer of no rows. The answer was right; the cost was a full read.
Why. When the outer's startup cost is below the hash's total cost, the hash join fetches its first outer row before it builds. So the fact scan ran. The coordinator had already drained the build, found it empty, and attached its Bloom check. That check rejects every row when the build is empty, so the scan rejected its way to the end of the table.
Chunk Groups ReadRuntime Filter Rows RejectedbuildEmptyforced falseThe fix (
src/columnar_runtime_filter.c)PgColumnarExecRuntimeFilterreturns end-of-data as soon as the build is found empty. The coordinator wraps only inner joins (PgColumnarSetJoinPathlistreturns for any otherjointype), and an inner join with an empty build has no rows. A rescan resetsprepared, so each execution decides again.Evidence
Red first, both twins, PG17:
prepared = falseremoved from the rescan)an empty build reads no fact chunk groupgot [5] want [0], pytestgot 6 want 0an empty build on one rescan does not empty the nextERROR: ... tap read before drain, pytest no rowsThe rescan arm guards the reset that the new early return relies on. Its first outer row builds nothing, and its second builds every key.
Premises, so the arms can't pass on nothing:
0|200in the shell,0|150in pytest).Green, PG15 to 19:
native_join_runtime_filter.sh62/62 on each major;test_join_runtime_filter.py65 checks over 15 tests on each major.Ledger. Seven rows.
an empty build reads no fact chunk groupis dated by the tree before the fix, and the rescan arm by the mutation. The other five arrivenever. The census is 3571 / 2644 / 927.pgc_ledger.py gatewith--registeredand--against origin/mainreturns rc 0, with the ceiling 221 -> 221;orphan-scanreports 0 orphans.Counts. Collection gives 643 (641 -> 643). The guard set passes (438).
Also
The old small-dimension empty-build arm (#1339's
test_join_runtime_filter.py:542, weak) can only fail on a crash, because with a small dimension the hash join never runs its outer side. Its shell comment claimed to catch "a leftover hull from a previous execution", which it can't. The comment now says what the arm can and can't catch, and the new arms carry the real test. I'll tick that #1339 entry when this merges.🤖 Generated with Claude Code