Repository navigation
Conversation
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 2d6b1842f1485857aa118e2f8b8b11eeab7f440f with five independent passes covering correctness/Spark compatibility, performance and measurement validity, design/reuse, test maintainability/CI integration, and documentation/release compatibility. No substantive issues found in the benchmark or the documented behavior. The off/on cases use matched fixtures and settings, validate the direct scan/partial aggregate and per-task attachment, and separate untimed metrics from query timings. The eligibility, fallback limits, and metric descriptions match Comet and its pinned DataFusion implementation. This PR changes no production code or configuration defaults.
Please format docs/source/user-guide/latest/metrics.md and docs/source/user-guide/latest/tuning/operators.md. Both upstream Preflight and fork CI fail prettier --check "**/*.md" on those two files at this head. Preflight stops the workflow before downstream build/validation jobs can run. Please get the formatting fix through hosted CI before merging.
Validation was source inspection and exact-head hosted CI/log review. I did not run local tests or benchmarks.
Which issue does this PR close?
Part of #5774. Follows the merged reader support (#7598) and Spark integration (#7657).
TL;DR performance
Measured on 8.39M INT rows, uncompressed Parquet,
local[1], in both mode orders. Times below are filtering off → on.MAX, one task, improved 74 → 48 ms and 87 → 44 ms across the two runs (about 1.5–2.0× faster).MAXwith four tasks took 63 → 179 ms and 67 → 125 ms, with high variance despite pruning.Timings are sensitive to warmup and outliers; scan savings do not guarantee faster queries. The feature remains disabled by default. Full means, standard deviations, and counters are below.
Rationale for this change
MIN/MAX reader filtering can skip later Parquet row groups after a task observes a useful bound. The benefit depends on data order and task boundaries, while footer inspection and bound updates add overhead. This adds an off/on benchmark and documents eligibility, fallback limits, and metrics before considering broader enablement.
What changes are included in this PR?
CometMinMaxBenchmark: separate MIN and MAX queries over ascending, descending, deterministic random, and constant INT data, with one or four scan tasks by default. It checks Spark result parity, the direct scan/partial aggregate plan, actual task count, and per-task filter attachment before timing.The existing configuration remains disabled by default. Runtime behavior and supported scope are unchanged. The two commits separate the benchmark from documentation.
How are these changes tested?
-Ctarget-cpu=native -Clink-arg=-fuse-ld=bfdand the standard release profile/default features.test-compile, including formatting and style checks, passed. Packaged binary/source jar-content checks passed (the final check was rerun with JDK tools on PATH).git diff --checkpassed.Local release measurements
Environment: AMD EPYC-Genoa, 32 logical CPUs, about 98 GiB RAM, Ubuntu 24.04.4/Linux 6.8; Spark 4.1.3, OpenJDK 21.0.13.0.3, Rust 1.99.1. Spark used
local[1]and an 8 GiB benchmark JVM heap. Production code is fromd187e6823; the benchmark is in this PR'sd2f9fdfc6commit. Each fixture contained 8,388,608 rows and nine row groups per file. Four scan tasks ran serially underlocal[1].Equivalent standard invocations after the same release build:
Favorable ordering and constant values reduced emitted scan rows and requested data bytes by about 87.5%. The unfavorable ordered cases pruned no groups and expose filtering overhead. Random layouts pruned some groups but did not consistently improve elapsed time. These short local runs are sensitive to warmup and long-tail outliers; the first ascending-MIN timing even changes direction when case order reverses. This is evidence of scan-work reduction and overhead cases, not a general speedup claim or a basis to enable the feature by default.
All elapsed times and scan-work counters
Times are mean ± standard deviation in milliseconds, always shown as off → on, regardless of execution order. Every off execution emitted 8,388,608 rows and pruned zero groups. The byte column is
bytes_scannedin bytes, excluding metadata. All on attachment counts equaled the scan-task count; skipped counts were zero. Statistics, page-index, and decoder pruning counters were zero.Reader
scan_io_data_bytesequaledbytes_scanned.scan_io_metadata_byteswas 524,288 bytes per task in each mode (524,288 for one task and 2,097,152 for four); attachment does not eliminate footer inspection. Timings include planning and result collection, while fixture generation, Spark parity checks, plan validation, and metric reporting are outside the timed cases.