Skip to content

fix: backport aggregate dynamic-filter NULL-bound recovery (#25579) - #41

Merged
discord9 merged 2 commits into
greptimedb-55.1-thin-forkfrom
fix/topk-aggregate-null-bound
Nov 10, 2026
Merged

discord9 merged 2 commits into
greptimedb-55.1-thin-forkfrom
fix/topk-aggregate-null-bound

Conversation

@discord9

@discord9 discord9 commented Nov 8, 2026

Copy link
Copy Markdown

Scope: independent DataFusion backport of upstream apache#25579 — Independent aggregate apache#25579
backport
. It requires none of the other 11 backports in JOIN/pruning PR #39 and has no #37
(cast/statistics) ancestry: both branches base on e3221b8f7.

Which issue does this PR close?

None. Upstream apache#25579 is already merged and keeps its own ownership; as a downstream backport this
carries no Closes line.

Rationale for this change

User-visible problem: with aggregate dynamic-filter pushdown on, MIN/MAX over nullable columns
can silently return NULL — or drop rows another aggregate still needs — when one aggregate
publishes a bound before another has seen its first non-NULL value.

This is an aggregate producer fix and is deliberately independent of the TopK work. It is the
precondition for the GT generic statistics guard lift (60f765) that restores TopK pruning for
ORDER BY ts, event_id LIMIT k: there a blanket ... OR <child> IS NULL statistics wrapper
generated for a non-time column with unknown statistics blocks SST time-range pruning, so the TopK
scans far more than it needs. This PR makes the aggregate filter contract safe for that lift — it
returns literal TRUE until every aggregate has a non-NULL bound, and typed NULL bounds recover
instead of sticking — but does not perform the lift.

Honest boundary: companion GT change 60f765 (fix-topk-pruning-null-bound) replaces the guarded
statistics access with .map(current) on the raw snapshot, dropping the blanket
OR <child> IS NULL wrapper while retaining the decoded guard. That GT change is not contained
here, and this DataFusion PR does not by itself fix TopK pruning — it is its correctness
prerequisite.

What changes are included in this PR?

2 commits, 3 files, +344/−8:

  • datafusion/physical-plan/src/aggregates/aggregate_stream.rs: the producer returns lit(true)
    while any aggregate still lacks a bound (was continue, letting the remaining bounds prune rows
    another aggregate needed); scalar_cmp_null_short_circuit compares by is_null() so typed NULL
    scalars are handled; adds aggregate_dynamic_filter_recovers_from_null_bounds.
  • datafusion/physical-plan/src/aggregates/mod.rs: real-producer regression tests (batch size 1
    through FilterExec over TestMemoryExec) for unknown columns, NULL→value selectivity, mixed
    min/max, and a consumer observing the updated predicate.
  • datafusion/sqllogictest/test_files/push_down_filter_regression.slt: two-file case where
    MIN(a)/MAX(a) bound before b sees a value; results must match with pushdown on and off.

Commit 1 is backport ad8cef828 (PR apache#25579 by Jayant Shrivastava, full SHA
0af68188912349e9c865548458b37b0e66b5c31d), preserved with author, message and URL. Commit 2
(bed91dcd5) relocates the pre-existing local regression tests; it adds no new test or product
behavior.

Are these changes tested?

  • Upstream unit tests plus the SLT case above; the local tests are moved, not new.
  • Historical bounded standalone evidence (GT 60f765 + 92d537) on a real 120-hour SST dataset:
    9 positive TopK cases pruned 73–94 SSTs, and the nullable case verified the complete 240k-row
    multiset plus 16 ordered pages; the sort, aggregate, join and restart cases passed. That run is
    not exact split-branch runtime (92d537 carried fix: restrict reverse cast constraint propagation to safe conversions #37 ancestry) and is not distributed or
    universal proof.
  • Compilation: cargo check --locked --offline --all-targets -p datafusion-physical-plan -p datafusion-sqllogictest — PASS (exit 0, Finished dev profile … in 1m 19s). Compile-only for this split; no test or clippy re-run. No new benchmark framework or safety scaffold is added.

Are there any user-facing changes?

Aggregate results over nullable columns are now correct under dynamic-filter pushdown (a column can
no longer come back NULL because its first non-NULL value was filtered out). No configuration option
changes.

jayshrivastava and others added 2 commits November 8, 2026 22:11
…c filters (apache#25579)

The fix prevents one aggregate’s filter from discarding rows that
another aggregate still needs in dynamic filters.

### Example:

Query: `MIN(a), MAX(a), MIN(b), MAX(b)`

```
  First batch: (1, NULL), (8, NULL)
                      |
                      v
             a has bounds [1, 8]
             b has no bounds yet
                      |
            +---------+----------+
            |                    |
          Before                After
            |                    |
     a < 1 OR a > 8         filter = true
            |                    |
            v                    v
  Next batch: (2, 5), (3, 7)
            |                    |
     Both rows rejected     Both rows accepted
            |                    |
            v                    v
     Result:             Now b has bounds [5, 7]
     1 8 NULL NULL                |
                                  v
                        Filtering can start:
                        a < 1 OR a > 8
                        OR b < 5 OR b > 7
                                  |
                                  v
                        Correct result: 1 8 5 7
```

The latter is correct

1. According to
[Postgres](https://www.postgresql.org/docs/current/functions-aggregate.html):
> - MIN: “Computes the minimum of the non-null input values.”
>  - MAX: “Computes the maximum of the non-null input values.”

2. According to
[DataFusion](https://github.com/apache/datafusion/blob/6ff484a96d7eca4b1bf4a263d7190ada5af2343c/datafusion/sqllogictest/test_files/aggregate.slt#L5490-L5500)
Executing `1, NULL, 3, 4, 5` expects `MIN = 1, MAX = 5`

---------

Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com>

Backport provenance:
- Upstream PR: apache#25579
- Upstream commit: 0af6818
- Conflict resolution only supplies the missing test module/import context; the upstream logic, native test body and SQL assertions are preserved.
- Original upstream tests are retained; fork-only adaptation, if any, follows separately.

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 390e174)
Move the existing native real-producer regressions from the former combined
backport into the independent aggregate NULL-bound repair. No new test or
product behavior is introduced by this history split.

Signed-off-by: discord9 <discord9@163.com>

@fengjiachun fengjiachun left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed bed91dc and verified that the production changes match upstream apache#25579. Waiting for every aggregate to have a non-NULL bound prevents premature pruning, and the typed-NULL handling allows bounds to recover.

Validation on this PR head: all five focused aggregate dynamic-filter unit tests passed, and the complete push_down_filter_regression.slt file passed. Reverting the production fix while retaining the tests caused four regressions to fail, confirming that they detect the bug.

Full workspace tests, Clippy, and GreptimeDB integration were not run.

@WenyXu WenyXu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Production hunks are byte-identical to upstream apache#25579 (0af68188): continue -> return Ok(lit(true)) and the is_null()-based short-circuit; the fork adds tests only.

Verified

  • shared_bound starts at ScalarValue::Null and the min/max accumulators return a typed NULL until a non-NULL value arrives, so is_null() is an exact "no bound yet" proxy. Returning lit(true) while any aggregate still lacks a bound is both sufficient and necessary; it only costs selectivity for the current generation.
  • ScalarValue::partial_cmp compares the Option payloads (None < Some), so pre-patch scalar_min(Int32(None), Int32(5)) returned the NULL and the shared bound stuck forever - the reported defect is real, and the new short-circuit only substitutes the other operand.
  • DynamicFilterPhysicalExpr::update has no column-count or shape assertion and the predicate stays Boolean, so lit(true) <-> OR-tree transitions are safe. Bounds are monotone, so a predicate that mixes bounds read at different instants remains conservative.
  • Tests: cargo test -p datafusion-physical-plan aggregate_dynamic_filter 5/5 (all five new tests), dynamic_filter 22/22, cargo check -p datafusion-physical-plan --tests clean. The new agg_dyn_late_bound SLT block passes in both config modes; that file's later failure in my checkout is only the missing parquet-testing submodule.

No findings. Local toolchain note: rust-toolchain.toml pins 1.97.0, which was broken on my host, so runs used 1.98.1 with the preset CARGO_TARGET_DIR.

@discord9
discord9 merged commit 45d2767 into greptimedb-55.1-thin-fork Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants