Repository navigation
feat: integrate MIN/MAX reader filtering with Spark 2/n - #7657
Conversation
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 0267e52 against merge base 36ab96a with five independent review scopes. No reproducible P1/P2 issue found in configuration capture, plan equality, aggregate eligibility, execution-local bounds, reader attachment or fallback behavior.
Verified paired Rust CI: 2,830 tests passed at base and 2,833 at head, including the three new planner tests; five skipped on each. Head Spark 3.5 test-source compilation passed. The new Spark suite's execution was not independently verified: the fork's native build failed downloading an unchanged Iceberg dependency over TLS, and downstream Spark jobs were skipped. Local JVM/native reruns were unavailable.
comphead
left a comment
There was a problem hiding this comment.
The wiring reuses DataFusion's aggregate dynamic filter and the shared reader filter helper from #7598, and eligibility is narrow (one partial, ungrouped MIN or MAX over a direct signed integer column), so float, decimal and string ordering do not come up. Comments cover where the flag and metrics are set, and test coverage while pruning runs.
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 081687d against merge base 36ab96a with five independent review scopes. No reproducible P1/P2 issue found in aggregate-mode eligibility, configuration capture, canonicalization/AQE, execution-local bounds, reader mapping, NULL/extreme integer handling, or metrics ownership.
The native implementation is unchanged from the earlier reviewed head, whose Rust run passed 2,833 tests. Current fork CI is still running, including Rust tests, Spark 3.5 strict compilation, and the Spark 4.1 native/JVM build: https://github.com/pingzh/datafusion-comet/actions/runs/44362462799. The author reports all nine revised Spark suite cases and both SQL matrix variants passing; I have not independently verified those exact-head execution logs. Normal CI checks remain outstanding.
Which issue does this PR close?
Part 2 of #5774, following the merged native reader-filter foundation in #7598. This PR does not close the issue.
Rationale for this change
For
SELECT MIN(id) FROM parquet_table, each Spark task can tighten its bound as it reads input and skip later row groups that cannot improve its partial result. The final aggregate combines those partial results as usual.What changes are included in this PR?
Add
spark.comet.exec.aggregate.dynamicFilter.enabled, disabled by default. Capture the flag only for nonempty, ungrouped aggregates whose expressions all use raw-input Partial mode, and include it in Spark plan equality so execution and plan reuse preserve the setting chosen during planning.When enabled, the Rust planner uses
MinMaxReaderFilterExecfor the narrow subset supported by #7598: one unfiltered MIN or MAX over a direct signed-integer column, without grouping, in raw-input Partial mode over an eligible native Parquet scan. Bounds remain local to each execution. Unsupported plans retain ordinary aggregation; Final/PartialMerge stages and JVM boundaries remain ineligible.Only the enabled path removes a matching integer column's identity cast, allowing DataFusion to recognize the bound producer. The disabled path retains that cast and avoids DataFusion's otherwise unconsumed bound updates. Spark displays reader-filter attachment counters only on aggregates whose captured flag is enabled; pruning and byte metrics remain on scans.
Register the new Spark suite in Linux and macOS CI. Benchmarks and detailed tuning documentation remain for PR 3; the configuration description documents the current limits.
How are these changes tested?
CometMinMaxSuite: 9 tests passed on Spark 4.1.3. Filtering off/on matches Spark for all four signed integer types, multiple Spark partitions with AQE off/on, and unsupported plans. The eight-row-group fixture includes an all-null group and actual type minimum/maximum values after bounds improve, and asserts dynamic pruning plus row/byte reductions. Tests also cover captured flags, conditional metrics, canonicalization, DISTINCT and merge stages, and a Comet Partial above a JVM-to-Arrow boundary with no reader attachment.min_max_dynamic_filter.sql: both config-matrix variants passed, covering empty, all-null, and duplicate/mixed input with separate MIN and MAX queries.dev/release/check-rat.sh HEAD) passed. The raw text fixture from feat: txt file support #6784 has an exact-path RAT exemption, consistent with the existing data-fixture exclusions.