Repository navigation
Conversation
Reuse the grouped variance, covariance and Pearson kernels while preserving Spark regression state layouts, arithmetic and version-dependent null/error behavior. Add grouped update/merge/emission coverage and focused native and Spark benchmarks for issue apache#6886.
Compare interleaved keys with long same-key runs and exercise all six regression functions in the single-group control. Preserve the existing benchmark names and cardinality/partial-skip checks.
…ps-recovered # Conflicts: # native/spark-expr/benches/aggregate_stats.rs
Composing the grouped variance, covariance and correlation accumulators made grouped regr_* slower than the per-group accumulators when rows of one group arrive together: 9-23% for regr_sxy, regr_slope, regr_intercept and regr_r2 with a single group, and 11% for regr_r2 with 256-row runs. Replace the composition with one accumulator that keeps a state row per group in Spark's partial-buffer layout and folds every moment of a statistic in a single pass. It carries the state in a local while consecutive rows share a group. The arithmetic and its order are those of the per-row accumulators, so results stay bit-identical; the existing grouped tests compare the two. This leaves the shared variance, covariance and correlation kernels as they are on main. Also drop the test assertion that depended on how DataFusion splits a Vec on EmitTo::First, and the benchmark changes that the regr cases do not need.
|
A few things I left out that could be follow-ups:
|
comphead
left a comment
There was a problem hiding this comment.
I read the grouped kernels against the scalar accumulators and welford.rs and found no behavior difference. State layouts, FILTER and NULL handling, EmitTo::First, size() and the merge order all match. One test gap below. I did not run anything.
| regr_sxx(x, y) FILTER (WHERE keep), regr_syy(x, y) FILTER (WHERE keep), | ||
| regr_sxy(x, y) FILTER (WHERE keep), regr_slope(x, y) FILTER (WHERE keep), | ||
| regr_intercept(x, y) FILTER (WHERE keep), regr_r2(x, y) FILTER (WHERE keep) | ||
| FROM test_regr_filtered GROUP BY g ORDER BY g |
There was a problem hiding this comment.
The grouped path is only compared with Comet's own scalar accumulator for NaN, infinities and -0.0 (in the Rust tests), never with Spark. regr.sql has no NaN or Infinity input at all, while corr.sql has a NaN permutation table. Could you add a small grouped table with cast('NaN' as double), cast('Infinity' as double) and -0.0 in y and x, and run the six functions with GROUP BY and FILTER? That would check the new loop against Spark directly. I expect it to pass because the update and merge arithmetic is shared with the scalar path, but I have not run it.
Which issue does this PR close?
Closes #6886.
Rationale for this change
Grouped
regr_*aggregates used DataFusion'sGroupsAccumulatorAdapter, which keeps one boxed accumulator per group and slows down as the number of groups grows.What changes are included in this PR?
Regrgets aGroupsAccumulatorthat keeps one state row per group, in the partial buffer's column order, and updates all of a statistic's moments in one pass. Four small kernels (regr_sxx/regr_syy,regr_sxy,regr_slope/regr_intercept,regr_r2) define update, merge and result with the samewelfordhelpers the scalar accumulators use, so each moment keeps its floating-point operation order.The issue suggests composing the existing grouped variance, covariance and correlation accumulators. I tried that first. It was 9-23% slower than the adapter for
regr_sxy,regr_slope,regr_interceptandregr_r2when all rows share one group, because every row reloads the state it just stored andregr_r2walks the batch three times. The single pass still writes the state back for every row it processes. While consecutive processed rows share a group, the next one continues from a local instead of reloading the state it just stored.Partial state layouts are unchanged, and the variance, covariance and correlation code is untouched.
How are these changes tested?
Unit tests compare the grouped accumulator with the scalar one bit for bit (NaN by
is_nan()) across state, merge,FILTER, sliced inputs, runs of one group and partial emission, for both Spark rule sets.regr.sqladds groupedFILTERand grouped constant-inputregr_r2queries.Run locally:
agg_funcsunit tests, Clippy with-D warnings, andregr.sqlon Spark 4.1.aggregate_statsgets aregr_partialgroup, andCometAggregateExpressionBenchmarkgets six grouped queries. Partial aggregation of 262,144 rows on an Apple M3 Pro, adapter → this PR, median of three runs:regr_r2regr_sloperegr_sxyregr_sxxregr_intercepttracksregr_slope, andregr_syytracksregr_sxx. The sparse-NULL layout is left out of the table. None of the 32 cases is slower than the adapter.