Skip to content

feat: integrate MIN/MAX reader filtering with Spark 2/n - #7657

Merged
sunchao merged 4 commits into
apache:mainfrom
pingzh:pingzh-minmax-spark-integration
Nov 19, 2026
Merged

sunchao merged 4 commits into
apache:mainfrom
pingzh:pingzh-minmax-spark-integration

Conversation

@pingzh

@pingzh pingzh commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part 2 of #5774, following the merged native reader-filter foundation in #7598. This PR does not close the issue.

Rationale for this change

For SELECT MIN(id) FROM parquet_table, each Spark task can tighten its bound as it reads input and skip later row groups that cannot improve its partial result. The final aggregate combines those partial results as usual.

What changes are included in this PR?

Add spark.comet.exec.aggregate.dynamicFilter.enabled, disabled by default. Capture the flag only for nonempty, ungrouped aggregates whose expressions all use raw-input Partial mode, and include it in Spark plan equality so execution and plan reuse preserve the setting chosen during planning.

When enabled, the Rust planner uses MinMaxReaderFilterExec for the narrow subset supported by #7598: one unfiltered MIN or MAX over a direct signed-integer column, without grouping, in raw-input Partial mode over an eligible native Parquet scan. Bounds remain local to each execution. Unsupported plans retain ordinary aggregation; Final/PartialMerge stages and JVM boundaries remain ineligible.

Only the enabled path removes a matching integer column's identity cast, allowing DataFusion to recognize the bound producer. The disabled path retains that cast and avoids DataFusion's otherwise unconsumed bound updates. Spark displays reader-filter attachment counters only on aggregates whose captured flag is enabled; pruning and byte metrics remain on scans.

Register the new Spark suite in Linux and macOS CI. Benchmarks and detailed tuning documentation remain for PR 3; the configuration description documents the current limits.

How are these changes tested?

  • CometMinMaxSuite: 9 tests passed on Spark 4.1.3. Filtering off/on matches Spark for all four signed integer types, multiple Spark partitions with AQE off/on, and unsupported plans. The eight-row-group fixture includes an all-null group and actual type minimum/maximum values after bounds improve, and asserts dynamic pruning plus row/byte reductions. Tests also cover captured flags, conditional metrics, canonicalization, DISTINCT and merge stages, and a Comet Partial above a JVM-to-Arrow boundary with no reader attachment.
  • min_max_dynamic_filter.sql: both config-matrix variants passed, covering empty, all-null, and duplicate/mixed input with separate MIN and MAX queries.
  • Native planner tests: 128 passed, including three new tests for serialized flag handling, pruning and metric ownership, ordinary stage/JVM fallback, retained real casts, and no dynamic expressions when disabled.
  • Existing dynamic-filter tests: 75 passed.
  • Workspace Clippy with warnings denied, Scala style checks, Spotless, Rust formatting, and whitespace checks passed.
  • Two independent source reviews completed for each of the two review-fix commits.
  • Source-archive license check (dev/release/check-rat.sh HEAD) passed. The raw text fixture from feat: txt file support #6784 has an exact-path RAT exemption, consistent with the existing data-fixture exclusions.

@github-actions github-actions Bot added enhancement New feature or request area:joins Join operators and dynamic filter pushdown labels Nov 16, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 0267e52 against merge base 36ab96a with five independent review scopes. No reproducible P1/P2 issue found in configuration capture, plan equality, aggregate eligibility, execution-local bounds, reader attachment or fallback behavior.

Verified paired Rust CI: 2,830 tests passed at base and 2,833 at head, including the three new planner tests; five skipped on each. Head Spark 3.5 test-source compilation passed. The new Spark suite's execution was not independently verified: the fork's native build failed downloading an unchanged Iceberg dependency over TLS, and downstream Spark jobs were skipped. Local JVM/native reruns were unavailable.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wiring reuses DataFusion's aggregate dynamic filter and the shared reader filter helper from #7598, and eligibility is narrow (one partial, ungrouped MIN or MAX over a direct signed integer column), so float, decimal and string ordering do not come up. Comments cover where the flag and metrics are set, and test coverage while pruning runs.

@sunchao
sunchao added this pull request to the merge queue Nov 18, 2026
@sunchao
sunchao removed this pull request from the merge queue due to a manual request Nov 18, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 081687d against merge base 36ab96a with five independent review scopes. No reproducible P1/P2 issue found in aggregate-mode eligibility, configuration capture, canonicalization/AQE, execution-local bounds, reader mapping, NULL/extreme integer handling, or metrics ownership.

The native implementation is unchanged from the earlier reviewed head, whose Rust run passed 2,833 tests. Current fork CI is still running, including Rust tests, Spark 3.5 strict compilation, and the Spark 4.1 native/JVM build: https://github.com/pingzh/datafusion-comet/actions/runs/44362462799. The author reports all nine revised Spark suite cases and both SQL matrix variants passing; I have not independently verified those exact-head execution logs. Normal CI checks remain outstanding.

@sunchao
sunchao added this pull request to the merge queue Nov 19, 2026
Merged via the queue into apache:main with commit 2b6956a Nov 19, 2026
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:joins Join operators and dynamic filter pushdown enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants