Repository navigation
fix: backport aggregate dynamic-filter NULL-bound recovery (#25579) - #41
Conversation
…c filters (apache#25579) The fix prevents one aggregate’s filter from discarding rows that another aggregate still needs in dynamic filters. ### Example: Query: `MIN(a), MAX(a), MIN(b), MAX(b)` ``` First batch: (1, NULL), (8, NULL) | v a has bounds [1, 8] b has no bounds yet | +---------+----------+ | | Before After | | a < 1 OR a > 8 filter = true | | v v Next batch: (2, 5), (3, 7) | | Both rows rejected Both rows accepted | | v v Result: Now b has bounds [5, 7] 1 8 NULL NULL | v Filtering can start: a < 1 OR a > 8 OR b < 5 OR b > 7 | v Correct result: 1 8 5 7 ``` The latter is correct 1. According to [Postgres](https://www.postgresql.org/docs/current/functions-aggregate.html): > - MIN: “Computes the minimum of the non-null input values.” > - MAX: “Computes the maximum of the non-null input values.” 2. According to [DataFusion](https://github.com/apache/datafusion/blob/6ff484a96d7eca4b1bf4a263d7190ada5af2343c/datafusion/sqllogictest/test_files/aggregate.slt#L5490-L5500) Executing `1, NULL, 3, 4, 5` expects `MIN = 1, MAX = 5` --------- Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com> Backport provenance: - Upstream PR: apache#25579 - Upstream commit: 0af6818 - Conflict resolution only supplies the missing test module/import context; the upstream logic, native test body and SQL assertions are preserved. - Original upstream tests are retained; fork-only adaptation, if any, follows separately. Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 390e174)
Move the existing native real-producer regressions from the former combined backport into the independent aggregate NULL-bound repair. No new test or product behavior is introduced by this history split. Signed-off-by: discord9 <discord9@163.com>
fengjiachun
left a comment
There was a problem hiding this comment.
LGTM. Reviewed bed91dc and verified that the production changes match upstream apache#25579. Waiting for every aggregate to have a non-NULL bound prevents premature pruning, and the typed-NULL handling allows bounds to recover.
Validation on this PR head: all five focused aggregate dynamic-filter unit tests passed, and the complete push_down_filter_regression.slt file passed. Reverting the production fix while retaining the tests caused four regressions to fail, confirming that they detect the bug.
Full workspace tests, Clippy, and GreptimeDB integration were not run.
WenyXu
left a comment
There was a problem hiding this comment.
Approved. Production hunks are byte-identical to upstream apache#25579 (0af68188): continue -> return Ok(lit(true)) and the is_null()-based short-circuit; the fork adds tests only.
Verified
shared_boundstarts atScalarValue::Nulland the min/max accumulators return a typed NULL until a non-NULL value arrives, sois_null()is an exact "no bound yet" proxy. Returninglit(true)while any aggregate still lacks a bound is both sufficient and necessary; it only costs selectivity for the current generation.ScalarValue::partial_cmpcompares theOptionpayloads (None < Some), so pre-patchscalar_min(Int32(None), Int32(5))returned the NULL and the shared bound stuck forever - the reported defect is real, and the new short-circuit only substitutes the other operand.DynamicFilterPhysicalExpr::updatehas no column-count or shape assertion and the predicate stays Boolean, solit(true) <-> OR-treetransitions are safe. Bounds are monotone, so a predicate that mixes bounds read at different instants remains conservative.- Tests:
cargo test -p datafusion-physical-plan aggregate_dynamic_filter5/5 (all five new tests),dynamic_filter22/22,cargo check -p datafusion-physical-plan --testsclean. The newagg_dyn_late_boundSLT block passes in both config modes; that file's later failure in my checkout is only the missingparquet-testingsubmodule.
No findings. Local toolchain note: rust-toolchain.toml pins 1.97.0, which was broken on my host, so runs used 1.98.1 with the preset CARGO_TARGET_DIR.
Scope: independent DataFusion backport of upstream apache#25579 — Independent aggregate apache#25579
backport. It requires none of the other 11 backports in JOIN/pruning PR #39 and has no #37
(cast/statistics) ancestry: both branches base on
e3221b8f7.Which issue does this PR close?
None. Upstream apache#25579 is already merged and keeps its own ownership; as a downstream backport this
carries no
Closesline.Rationale for this change
User-visible problem: with aggregate dynamic-filter pushdown on,
MIN/MAXover nullable columnscan silently return NULL — or drop rows another aggregate still needs — when one aggregate
publishes a bound before another has seen its first non-NULL value.
This is an aggregate producer fix and is deliberately independent of the TopK work. It is the
precondition for the GT generic statistics guard lift (
60f765) that restores TopK pruning forORDER BY ts, event_id LIMIT k: there a blanket... OR <child> IS NULLstatistics wrappergenerated for a non-time column with unknown statistics blocks SST time-range pruning, so the TopK
scans far more than it needs. This PR makes the aggregate filter contract safe for that lift — it
returns literal
TRUEuntil every aggregate has a non-NULL bound, and typed NULL bounds recoverinstead of sticking — but does not perform the lift.
Honest boundary: companion GT change
60f765(fix-topk-pruning-null-bound) replaces the guardedstatistics access with
.map(current)on the raw snapshot, dropping the blanketOR <child> IS NULLwrapper while retaining the decoded guard. That GT change is not containedhere, and this DataFusion PR does not by itself fix TopK pruning — it is its correctness
prerequisite.
What changes are included in this PR?
2 commits, 3 files, +344/−8:
datafusion/physical-plan/src/aggregates/aggregate_stream.rs: the producer returnslit(true)while any aggregate still lacks a bound (was
continue, letting the remaining bounds prune rowsanother aggregate needed);
scalar_cmp_null_short_circuitcompares byis_null()so typed NULLscalars are handled; adds
aggregate_dynamic_filter_recovers_from_null_bounds.datafusion/physical-plan/src/aggregates/mod.rs: real-producer regression tests (batch size 1through
FilterExecoverTestMemoryExec) for unknown columns, NULL→value selectivity, mixedmin/max, and a consumer observing the updated predicate.
datafusion/sqllogictest/test_files/push_down_filter_regression.slt: two-file case whereMIN(a)/MAX(a)bound beforebsees a value; results must match with pushdown on and off.Commit 1 is backport
ad8cef828(PR apache#25579 by Jayant Shrivastava, full SHA0af68188912349e9c865548458b37b0e66b5c31d), preserved with author, message and URL. Commit 2(
bed91dcd5) relocates the pre-existing local regression tests; it adds no new test or productbehavior.
Are these changes tested?
60f765+92d537) on a real 120-hour SST dataset:9 positive TopK cases pruned 73–94 SSTs, and the nullable case verified the complete 240k-row
multiset plus 16 ordered pages; the sort, aggregate, join and restart cases passed. That run is
not exact split-branch runtime (
92d537carried fix: restrict reverse cast constraint propagation to safe conversions #37 ancestry) and is not distributed oruniversal proof.
cargo check --locked --offline --all-targets -p datafusion-physical-plan -p datafusion-sqllogictest— PASS (exit 0,Finished dev profile … in 1m 19s). Compile-only for this split; no test or clippy re-run. No new benchmark framework or safety scaffold is added.Are there any user-facing changes?
Aggregate results over nullable columns are now correct under dynamic-filter pushdown (a column can
no longer come back NULL because its first non-NULL value was filtered out). No configuration option
changes.