Repository navigation
Description
Summary
DataFusion 55.0.0 was released on August 18. Six weeks later we still can't release Ballista 55.0.0, because our release includes the Python client, and the Python client can't ship until datafusion-python 55.0.0 is published. That release is being prepared in apache/datafusion-python#1696, but the vote hasn't started yet.
This isn't a complaint about datafusion-python, which has its own priorities, including the FFI query planner support that #2252 builds on. But now that our Rust side is ready soon after each DataFusion release, waiting for datafusion-python has become the main thing holding up our releases, and there's a lot on main that I'd like to ship. So I'd like to propose that we release the Python client separately from the Rust crates.
How far datafusion-python trails DataFusion
| DataFusion | Released | datafusion-python released | Lag |
|---|---|---|---|
| 44.0.0 | 2024-12-31 | 2025-02-12 | 43 days |
| 45.0.0 | 2025-02-07 | 2025-02-24 (45.2.0) | 17 days |
| 46.0.0 | 2025-03-07 | 2025-03-30 | 23 days |
| 47.0.0 | 2025-04-20 | 2025-05-28 | 38 days |
| 48.0.0 | 2025-06-11 | 2025-07-12 | 31 days |
| 49.0.0 | 2025-07-27 | 2025-08-29 | 33 days |
| 50.0.0 | 2025-09-16 | 2025-09-23 | 7 days |
| 51.0.0 | 2025-11-19 | 2026-01-09 | 51 days |
| 52.0.0 | 2026-01-12 | 2026-02-23 | 42 days |
| 53.0.0 | 2026-03-23 | 2026-04-13 | 21 days |
| 54.0.0 | 2026-06-08 | 2026-06-29 | 21 days |
| 55.0.0 | 2026-08-18 | not released yet | 42 days so far |
Dates are from crates.io and PyPI. We need both, so where they differ I used the later one.
Over the last eleven releases, the median lag was 31 days. The shortest was one week and the longest was over seven weeks.
This didn't hurt us much before, because our own DataFusion upgrades took about as long. For 54, our upgrade (#1906) landed on June 30, the day after datafusion-python 54.0.0 was published. Since #1920 we've tracked DataFusion main between releases, and this time we were on the released 55.0.0 crates eight days after DataFusion published them (#2375). So datafusion-python is now what we wait on, and with the current process our releases will usually trail DataFusion by at least a month.
What's waiting on main
310 commits have landed on main since branch-54 was cut on July 12, and 54.1.0 only picked up a handful of backported fixes. Some highlights:
- Correctness. Distributed
NOT INcould return wrong results (fix(scheduler): execute null-aware anti joins in one task #2188), and uncorrelatedNOT INsubqueries now get a plan that runs in parallel (feat(core): rewrite NOT IN subqueries into distributable anti join plans #2199). A set of AQE fixes means TPC-DS now verifies clean across all 99 queries with AQE on (fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default #2315). - Stability. Executors default to a bounded, auto-sized memory pool, so heavy queries spill instead of getting the executor OOM-killed (feat(executor): default to a bounded, auto-sized memory pool #2160). Busy executors keep heartbeating instead of being declared dead (fix(executor): keep heartbeating while all task slots are busy #2159), and jobs fail with a clear error instead of hanging when all executors are lost (fix(scheduler): fail running jobs when all executors are lost #2212).
- Performance. Adaptive query planning is on by default (fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default #2315), and it can now measure a join's build side before choosing the join (feat(scheduler): stage a join's build side before deciding the join #2434). Tasks can cover several partitions (Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor #2038), some window functions with no
PARTITION BYnow run in parallel instead of on a single core (feat: parallel prefix scan for UNBOUNDED PRECEDING window aggregates #2211, feat: parallel BWAG for the range-window shape (h2o Q8) #2223), and sharing the file statistics cache cuts total planning time for the 22 TPC-H queries at SF100 from 1.5 s to 0.13 s (perf(scheduler): share the file statistics cache across sessions #2498). - Operations. A history server for browsing completed jobs (feat(scheduler): add the history server #2265), and an OpenAPI description of the REST API (feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json #2397).
Previous discussion
We discussed this in #2370 in August:
- I listed the costs of splitting. We'd need separate tarball scripts and separate votes, it would be harder for Python users to help test and vote on the Rust release, and a Python release vote could turn up bugs in crates we'd already released, which would mean patch releases.
- @avantgardnerio asked whether Python users would have to wait to upgrade until the Python client caught up, and showed how a 54 client could get silently different results from a 55 cluster (the
preserve_nullsexample). - @milenkovicm noted that the Python bindings are "best effort" and a "technical preview" today and that we don't have many Python users yet, and suggested reviving the separate
datafusion-ballista-pythonrepo once the FFI integration is solid, so we could release Ballista without waiting for datafusion-python. - The consensus was to keep the single release "until proven otherwise". I think the 55.0.0 release is a good reason to look at it again.
Some older history is relevant too:
- In 2023 we moved the Python bindings to their own repo (Move Python bindings to separate repo? #635), partly because datafusion-python releases on a different schedule. They went unmaintained for about a year, and PyBallista - Python SQL client for Ballista #970 rebuilt them in this repo in 2024. That PR said the new bindings "will be versioned and released independently from the main project".
- In Ballista Python Issue(s) #1142 (2024), @milenkovicm proposed shipping the Rust release without the Python bindings until the Python integration was sorted out.
- Create PyPi Release Process #1120 then set the goal of keeping the Python client in step with the Rust releases, and docs(release): document PyPI publish process for python client #1610 documented the current process. We only started publishing the Python client to PyPI with 54.0.0 in July, so the coupling is fairly new.
Proposal
Release the Rust crates and the Python client separately, and keep both in this repo.
- Rust release. The same process as today, minus the Python wheels. We cut it as soon as we're ready after a DataFusion release.
- Python release. Its own RC tag (something like
python-55.0.0-rc1), source tarball and vote, once datafusion-python has published the matching version. It would build against theballistacrates we've already published to crates.io, so the wheels contain exactly the Rust code we voted on. The Python client keeps the version of the Ballista release it wraps, so Ballista 55.0.0 would be followed byballista55.0.0 on PyPI. - Same repo.
python/stays where it is. Keeping it next to the Rust code makes it much easier to test againstmain(CI: Python client tests do not run on Rust changes and do not test the working tree #2372), and the separate repo didn't work out last time.
For 55.0.0, this would mean starting the Rust release now and following up with the Python release once datafusion-python 55.0.0 is out.
What this means for users:
- Rust users get each release when it's ready, instead of waiting for datafusion-python.
- Python users get the Python client at the same time they would today, since today's release also waits for datafusion-python.
- Mixed versions. In the gap between the two releases, Python users should keep their cluster on the previous release. Since feat: scheduler REST & GRPC returns ballista version headers #2378 the client logs a warning when the scheduler's version doesn't match. Given CopyTo plans from a 54 client silently decode as the wrong file format on a 55 scheduler #2376 and the
preserve_nullsexample, I think a major version mismatch should be an error rather than a warning.
The concerns from #2370
- Separate scripts and votes. This is real, but it's mostly one-time work in
dev/releaseand the wheel build workflow, and I'm happy to do it. The PMC already runs separate votes for DataFusion, datafusion-python, Comet and Ballista, so one more vote isn't unusual. - Python users voting on the Rust release. They would vote on the Python release instead, which is the artifact they actually use.
- Bugs found late. If the Python release turns up a bug in the Rust crates, we'd do a patch release, the same as for any bug found after a release. I'd rather do that occasionally than hold 300 commits for six weeks.
- FFI. feat(python): integrate Ballista through DataFusion FFI #2252 would remove the Rust-level dependency on datafusion-python, which is great. As @milenkovicm pointed out, though, we'd probably still depend on a matching
datafusionPython release, so I see it as complementary rather than a fix for the release timing.
Alternatives
- Keep the single release. Every Ballista release waits for datafusion-python, a median of 31 days after DataFusion.
- Only split when we're blocked. Hold one vote when datafusion-python is already out, and split otherwise. Going by the table we'd end up splitting almost every time, so I'd rather have one process.
- Move the Python client back to its own repo. This was @milenkovicm's suggestion in Validate BALLISTA_PROTOCOL_VERSION for clients, not just executors #2370. It gives us the same release independence, but it makes testing the client against
mainharder, and it didn't go well last time.
Questions
- Is anyone opposed to releasing the Python client separately?
- Should a major version mismatch between the client and the scheduler be an error?
- Should we start the 55.0.0 Rust release now, or wait for datafusion-python 55.0.0 this time and switch to the new process for 56?
If there's support, I'm happy to update the release scripts, the wheel build workflow and the release docs.
Activity
villebro commentedon Sep 29, 2026
I support this split. While this would mean introducing yet another release for the PMC to vote on (not my problem 😛), I feel the long term benefit of having the Ballista and DataFusion Rust crate release cycles closer together far outweighs benefits of having a Python client available when the Ballista crate ships. Another benefit of decoupling is the fact that not all bug fixes or features necessarily apply to both: the Python client can need a change that doesn't require a new Rust crate and vice versa.
avantgardnerio commentedon Sep 29, 2026
I support this, because AFAIK, we still aren't able to test with DF main until we ship ballista-python 55?
Yes, I think a major version mismatch should be an error - I think I've shown doing otherwise is outside our control.
andygrove commentedon Sep 29, 2026
Good point. Yes, this is true.
I agree. I filed #2513 and am working on it separately to the change to the release process.
milenkovicm commentedon Sep 30, 2026
in our defence we can say this will be the best release ever :D
do we move python to datafusion-ballista-python at the end? it would make sense if we have different release process.
timsaucer commentedon Sep 30, 2026
I feel bad about the long delay in some of the
datafusion-pythonreleases. I also feel like it would help if more people were actively involved in pushing them forward. Since there is plenty of historical data to show the lag, I think this proposal makes good sense.andygrove commentedon Oct 1, 2026
I don't see much value in moving it out, personally. A separate release process doesn't need a separate repo: #2512 tags the Python releases on the same release branch and builds the source tarball from just python/. Keeping it in this repo also makes it easy to run the client's tests against a cluster built from main, which is how #2374 caught the wire break in #2376. And the last time we moved it out (#635), it went unmaintained for about a year before #970 brought it back.
andygrove commentedon Oct 1, 2026
No need to feel bad about anything. DF Python has its own priorities. I don't think it is fair for Ballista to put artificial pressure on other projects to ship faster.