Repository navigation
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for extending the reducer to computed keys. I compared against branch-1.2 and the rightKeyReduction config does not exist there, so nothing changes for users upgrading from the release. The default is off, so the new path is opt-in. I read the planner rewrite closely and did not find a correctness problem. The projection is built before the join key is rewritten, so an early return cannot leave join_on half-changed. The new SortExec keeps preserve_partitioning. The left-semi join with no filter needs no other right columns, so narrowing the right schema to the projected key is safe. CI is green, including the Spark 4.1 label run.
I have three asks on coverage and robustness.
-
The native test in
native/core/src/execution/planner/tests/semi_join_reduction.rsonly builds bareColumnkeys. The newProjectionExecbranch inreduce_before_sortis covered only throughCometJoinSuite. Could you add a native case with a non-Columninteger key, such as acoalesceover a bound reference and a literal? It could assert that the reducer sits under the sort, that a projection sits under the reducer, and that the join key becomesmembership_key. A negative case with a non-integer key would also help. -
The new
CometJoinSuitequery uses aBIGINTcolumn with a literal fallback. Could you add a variant like your motivating example,coalesce(r._1, r._2)over two nullable columns? It could include rows where both are null, so the projected key is null and must not match any left row. It would also be good to run it on anINTcolumn, since the Scala gate admits everyIntegralTypeand the native check acceptsInt8throughInt64. -
In
reduce_before_sort,key.data_type(&schema)?would fail the whole query if the key's type can't be resolved. The rest of that guard skips reduction when something doesn't fit. Should this returnOk(None)instead, so the function keeps its "check, then skip" contract?
|
Thanks @andygrove. I addressed all three points in 13fd925bb5.
Both native planner tests and all four membership reducer tests passed. All five focused join tests passed on Spark 3.5.9 and 4.1.3. I also refreshed the benchmark table in the PR description. |
comphead
left a comment
There was a problem hiding this comment.
Thanks @sunchao for extending the reducer to computed keys. I did not find a correctness problem for the gated shape: LeftSemi with no filter and one key, NullEqualsNothing keeps a NULL key from matching, and the reducer keeps one NULL row per batch.
The numbers in the description come from a workload that is not in the repo. Could CometSortMergeJoinBenchmark's right key reduction section get a coalesce case so the table can be reproduced? The unique-key rows also get faster with reduction on although every row bypasses the reducer, so I expect the gain there comes from projecting the sort input to one column, not from reduction. A projection-only variant, or the right_key_reduction_time metric, would show where the 2.4% and 7.7% slowdowns at 1,048,576 rows with 128 keys come from.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Thanks for extending the reducer to computed keys. I found no blocking concerns in the head commit 13fd925bb5. I read reduce_before_sort in native/core/src/execution/planner/join.rs against the Scala gate in CometSortMergeJoinExec.canReduceRightKey. The projection and the rewrite of join_on happen only after every skip check has passed, so an early return cannot leave the join half-changed. The Scala gate admits only Attribute and Literal operands, so casts and arithmetic never reach the native projection. A left-semi join with no filter reads no other right columns, so narrowing the right side to membership_key is safe. The new key.data_type(&schema) check now skips reduction instead of failing the plan, which addresses the earlier robustness question.
I compared against the latest release branch. The rightKeyReduction config is not there, and it defaults to false here, so a user upgrading sees no change in results, fallbacks or performance unless they opt in.
The remaining asks in the existing threads on SortExec fetch, the TINYINT and SMALLINT coverage, and the benchmark reproduction are the ones I would want addressed before merge. I have nothing to add beyond them.
Check fetch on the physical sort before rewriting it. Deduplicating [1, 1, 2] below a fetch of two would bring key 2 into the join. Exercise both nullable columns and a typed literal for TINYINT and SMALLINT, checking the key type, result, and reducer row counts. Compare the original sort, a post-shuffle projection, and projection with reduction over repeated and unique keys. Interleave the cases, check the native plan and result, and report time spent attempting reduction alongside action time.
Which issue does this PR close?
Part of #7189. This extends #7190.
Rationale for this change
A left-semi join such as
l.id = coalesce(r.primary_id, r.backup_id)only needs to know whether each right-side ID exists. The native sort currently receives every right row, even when many rows produce the same ID.What changes are included in this PR?
With
spark.comet.exec.sortMergeJoin.rightKeyReduction.enabled=true, the native right sort can now reduce one integercoalescekey over same-type columns and literals. A projection computes the key once before reduction. The sort and join read that projected column. The existing reducer still decides when to bypass a batch.The option defaults to false. As in #7190, it applies to an unfiltered left-semi join with a right sort directly over a Comet shuffle. Casts and arithmetic inside the coalesce fall outside the supported expression shape. If the planner cannot resolve the key’s type, it skips the reduction and continues planning the native join. The rewrite also checks
SortExec::fetch()on the physical sort. For sorted keys[1, 1, 2]and a fetch of two rows, deduplication before the fetch would incorrectly admit key2.How are these changes tested?
CometJoinSuitecovers all four signed integer widths. The two-columnINTcase matches keys5and7and exercises an all-NULL row. TheTINYINTandSMALLINTcases add a typed literal as the third operand, so the all-NULL row matches-1. Each case checks the planned key type, all 24,576 rows entering reduction, and fewer rows leaving it. TheBIGINTcase covers a literal fallback and duplicate left rows. The admission test also covers a cast inside the coalesce.13fd925bb5, the five focused join tests passed on Spark 3.5.9 and 4.1.3, along with both native planner tests and all four reducer tests. With the pre-PR JNI theINTtest returned the correct SQL result and failed the reducer-input assertion. Earlier validatione990b40a9c, the benchmark checked the expected count and sum[114688, 7516078080], native plan and row counts on Spark 3.5.9 and 4.1.3. Rust formatting, Maven Spotless andgit diff --checkpassed. The Rust and Comet test suites run in the fork. Fork CI · Apache CIBenchmark
CometSortMergeJoinBenchmarknow includes the four workloads below. All three modes shuffle the same two nullableINTcolumns, partitioned bycoalesce(a, b). Baseline carries both columns into the sort. Projection computes the key once after the shuffle and sorts every row of that one column. Reduction also removes repeated keys before sorting. Comparing reduction with projection shows the effect of enabling the reducer after narrowing the sort input. Benchmark sourceEach action time covers
spark.sql(query)throughcollect(). Within each JVM, two warmups per mode precede six measured rounds per workload. Each round runs all three modes, with each of the six possible orders used once. Three fresh JVMs run the workloads in table order and three reverse the workload and round orders. Each pair of times is forward / reverse on Spark 3.5.9. Each value is the median of three per-JVM medians. The last column isright_key_reduction_timewith reduction enabled, summed across tasks per action. It times attempts to deduplicate batches, including attempts that end in a bypass, and excludes projection and packing small batches. Timer scopeAcross the six Spark 3.5.9 JVMs, all 576 actions passed the result and plan checks and all 432 measured right sorts had zero spills.
Reproduce the workload
The probe has 131,072 rows. Every eighth key is above the build range and the rest use
(event_id * 48271) % 128. Build rowihas key(i * 48271) % distinct_keys, inbon every third row andaotherwise, with the other column NULL. The benchmark writes four Parquet files per input before timing. It uses four native shuffle partitions, direct reads, 8,192-row batches and the fair unified memory pool. AQE, broadcast and exchange reuse are disabled.The control uses
DISTRIBUTE BY coalesce(a, b)to place its projection after the shuffle. With constraint propagation enabled, Catalyst would infer an extra right-side null filter for the control's key alias. All three modes disable constraint propagation and specify the same probe filter. In the Spark 3.5.9 plans for the original and revised SQL at 1,048,576 rows and 128 keys, the scans, filters, hash partitioning and join matched for both flag settings. The right shuffle origin changed fromENSURE_REQUIREMENTStoREPARTITION_BY_COL. The control expresses the projection as a Spark-visible Comet operator. The enabled planner inserts a native projection. Both use DataFusionProjectionExec.The six JVMs invoked the benchmark main class directly with the Maven test classpath. They used JDK 17.0.21, four pinned AMD EPYC Milan CPUs, a 4 GiB heap, 1 GiB off-heap and the release JNI built with the checked-in lockfile. From the repository root with JDK 17 active, run the following commands on Linux, selecting four available CPUs for
taskset.Repeat the last two commands for each pair of JVMs. The output includes the six raw action times per mode, medians, sort row counts and spills.