Repository navigation
Closed
Description
I went through the 147 open issues and checked each one against main (code search), the PRs that mention it, and its comment thread. Below are the ones that look safe to close, plus a few that need a quick check first. Nothing was rebuilt or benchmarked to confirm these, so please check before closing.
cc @milenkovicm, since you asked to be tagged on old issues that can be closed (#2849).
Already fixed
| Issue | Why it can be closed |
|---|---|
| #2193 NOT IN loses null semantics in SortMergeJoinExec | DataFusion fixed this in apache/datafusion#22810, and main is on DataFusion 56. #2188 and #2199 also handle NOT IN on the Ballista side. |
| #1960 Surface column statistics via ShuffleReaderExec | Done in the #2664 → #2671 stack. #2664 says "Closes #1960" but was closed rather than merged; its commits came in through #2671. state/aqe/mod.rs now reads output_column_stats. |
| #2063 AQE hangs when a replan cancels an in-flight stage | #2192 now removes stages a replan cancels and ignores their late task updates. The "there are stages to be cancelled but its not implemented" warning is gone from main. |
| #348 Implement broadcast join optimization | #342 (broadcast exchange) is closed, and #1900 and #1904 added broadcast joins. |
| #890 Add Iceberg table support | #2638 (Iceberg scans) is merged, and the remaining Iceberg work is tracked in #2174. |
| #1151 Push-based shuffle / all-at-once stage scheduling | Covered by the pipelined execution mode: #2657 is merged and the design issue #2652 is closed. |
| #408 Bubble execution mode | Same as #1151: covered by pipelined mode (#2652, #2657). |
| #332 TaskGroups (share memory between task slots) | #2038 added multi-partition tasks, so one task can run several partitions. |
#406 Control collect_stat for the table provider |
#796 added the setting, and Ballista no longer overrides DataFusion's collect_statistics. |
| #119 ExecutorReservation during task scheduling | ExecutorReservation was removed in #823. |
| #645 Source data cache layer | A 2024 comment notes that its only unchecked item, #833, was merged. |
| #2162 S3 prefetching like Spark | The comment on the issue says the benchmark gap behind it is resolved (721s vs 700s compared with Spark 3.4). |
| #2367 TPC-H "Failed to open partition file" | The cause was a stale compiled Python extension in the local checkout, not a Ballista bug. The CI follow-up is tracked in #2372. |
| #2712 Nightly Docker image build failed | Every nightly Docker build from Nov 12 to Nov 17 succeeded. |
| #2727 Run a stage once when several stages read it | #2765 (which closed #1953) runs a repeated subplan only once under AQE, matching stages by a hash of their subtree. It would be good to confirm this covers physical plans that Comet submits. |
Duplicates or covered by another issue
| Issue | Covered by |
|---|---|
| #553 Support multiple working dirs in Executor | Same title as #196, and #553's body is the empty template. |
| #663 Make task reset policy configurable | Same request as #189: a setting for how the scheduler recovers when an executor is lost. |
| #12 Prefer assigning a job's tasks to the same executor | Covered by #2319 (shuffle affinity, open PR #2430). |
| #456 Support deltalake | Covered by #1241 (open table formats). There is already an example extension at milenkovicm/ballista_delta. |
Ask the reporter before closing
- Job hangs instead of failing after executors are OOM-killed #2856 (job hangs after executors are OOM-killed): fix(scheduler): fail jobs whose executor-loss rollback fails instead of hanging them #2860 fixed one cause of the hang, found while looking into this issue, but it only closes Executor-loss recovery hangs every remaining job when one job's stage rollback fails #2859. A re-run would confirm whether anything is left.
- Static planner: null-aware anti join loses its probe-side coalesce under an aggregate #2849 (null-aware anti join loses its probe-side coalesce): fix(scheduler): run nested-loop joins that emit build rows in one task #2848 calls this a related gap rather than fixing it, so it is probably still open. The reporter should confirm.
Probably out of date
These weren't checked against the code. They are a judgment call based on age and the APIs they refer to.
- Ballista context should get file metadata from scheduler, not from local disk #22 refers to the old
BallistaContext, which no longer exists. - Scheduler will lost the registered executor when restart it in the
pushmode #90 is about the old sled state backend. - Client side may fail to get job status after invoking execute_query() #263 depends on scheduler behaviour from 2022 (Remove Keyspace::QueuedJobs #134).
- Revisit all the public methods in TaskManager, ExecutionGraph and ExecutorManager #282 is a broad 2022 cleanup request.
- Add Accumulator API #650 is a design discussion that stopped in 2023.
Can be closed once an open PR merges
| Issue | Open PR |
|---|---|
| #2882 | #2784 |
| #2840 | #2867 |
| #2839 | #2886 |
| #2804 | #2809 |
| #2767 | #2775 |
| #2691 | #2698 |
| #2622 | #2818 |
| #2578 | #2774 |
| #2454 | #2501 |
| #2319 | #2430 |
| #2854, #2866 | #2852 |
#2371 could also be closed as won't fix, since the discussion on #2395 says the current behaviour is expected.
Look fixed but are still open
- Static job planner to respect
datafusion....hash_join_single_partition_thresholdfor broadcast decision #1770: the static planner still usesballista.optimizer.broadcast_join_threshold_bytes(planner.rs), and feat: broadcast small build sides via non-zero CollectLeft thresholds #1900 was only part of this issue. - feat: make it user configurable to display plans as tree renderer #1627: the REST API part is done (feat: rest api supports plan tree rendering #1650), but the tree format for
EXPLAINis still waiting on a DataFusion protobuf change. - Compile the scheduler's event-log writer without the REST API #2745: fix: enable tracing with the scheduler's rest-api feature #2747 fixed the build with only the
rest-apifeature. This issue asks for the event-log writer to build without that feature.
Metadata
Metadata
Assignees
Labels
No labels
Activity
milenkovicm commentedon Nov 17, 2026
fair play to you @NoahKusaba thanks a lot
will go through the list
NoahKusaba commentedon Nov 18, 2026
Close this too if you don't need it anymore 😆
milenkovicm commentedon Nov 19, 2026
Thanks @NoahKusaba great cleanup. will close this now