You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Static planner drops a RepartitionExec with range partitioning instead of planning a range shuffle #2680
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
DataFusion 55 added Partitioning::Range with split points, and RepartitionExec can produce it. When a submitted physical plan contains one, the static planner removes it: DefaultDistributedPlanner keeps only hash repartitions as stage boundaries and drops every other RepartitionExec (planner.rs#L298-L301).
Dropping a round-robin repartition only costs parallelism, but a range repartition is part of what a plan means. Consider a plan that range-partitions and then sorts each partition, which is a distributed global sort. It comes back as partitions that are each sorted but not range-disjoint, and there is no error.
Ballista already has range shuffles (RangeShuffleWriterExec and RangeShuffleReaderExec), but only the AQE path builds them (aqe/adapter.rs#L95), from cuts it measures itself. Task slicing has a TODO to map Partitioning::Range onto them (task_builder.rs#L274).
Comet Native Executors (apache/datafusion-comet#7288) need this for Spark's range exchanges, as in ORDER BY. The driver can supply the bounds, as Spark computes them by sampling, but for now the prototype (apache/datafusion-comet#7289) has to sort in a single task.
Describe the solution you'd like
In the static planner, plan a RepartitionExec with Partitioning::Range as a range shuffle stage that uses the plan's split points, so that partition i of the consuming stage reads range i.
Restrict range shuffle readers per task by slicing their split points along with their partitions, as the TODO describes.
Until the first two land, return an error for a range repartition rather than dropping it.
Describe alternatives you've considered
Collapsing to one partition and sorting there, which is what the Comet prototype does. It's correct, but the whole sort runs in one task.
Activity
feat: run sorts, and with them TPC-H q1, on Comet Native Executors
feat: add comet-scheduler and comet-executor built on DataFusion Ball…
feat: add comet-scheduler and comet-executor built on DataFusion Ball…
feat: run sorts, and with them TPC-H q1, on Comet Native Executors
feat: add comet-scheduler and comet-executor built on DataFusion Ball…
feat: run sorts, and with them TPC-H q1, on Comet Native Executors
feat: add comet-scheduler and comet-executor built on DataFusion Ball…
feat: run sorts, and with them TPC-H q1, on Comet Native Executors
feat: add comet-scheduler and comet-executor built on DataFusion Ball…
feat: run sorts, and with them TPC-H q1, on Comet Native Executors
feat: add comet-scheduler and comet-executor for Comet Native Executo…
fix: plan a range RepartitionExec as a shuffle stage instead of dropp…
fix: plan a range RepartitionExec as a shuffle stage instead of dropp…