Skip to content

Discussion: release the Python client separately from the Rust crates #2511

Closed

Description

@andygrove

Summary

DataFusion 55.0.0 was released on August 18. Six weeks later we still can't release Ballista 55.0.0, because our release includes the Python client, and the Python client can't ship until datafusion-python 55.0.0 is published. That release is being prepared in apache/datafusion-python#1696, but the vote hasn't started yet.

This isn't a complaint about datafusion-python, which has its own priorities, including the FFI query planner support that #2252 builds on. But now that our Rust side is ready soon after each DataFusion release, waiting for datafusion-python has become the main thing holding up our releases, and there's a lot on main that I'd like to ship. So I'd like to propose that we release the Python client separately from the Rust crates.

How far datafusion-python trails DataFusion

DataFusion Released datafusion-python released Lag
44.0.0 2024-12-31 2025-02-12 43 days
45.0.0 2025-02-07 2025-02-24 (45.2.0) 17 days
46.0.0 2025-03-07 2025-03-30 23 days
47.0.0 2025-04-20 2025-05-28 38 days
48.0.0 2025-06-11 2025-07-12 31 days
49.0.0 2025-07-27 2025-08-29 33 days
50.0.0 2025-09-16 2025-09-23 7 days
51.0.0 2025-11-19 2026-01-09 51 days
52.0.0 2026-01-12 2026-02-23 42 days
53.0.0 2026-03-23 2026-04-13 21 days
54.0.0 2026-06-08 2026-06-29 21 days
55.0.0 2026-08-18 not released yet 42 days so far

Dates are from crates.io and PyPI. We need both, so where they differ I used the later one.

Over the last eleven releases, the median lag was 31 days. The shortest was one week and the longest was over seven weeks.

This didn't hurt us much before, because our own DataFusion upgrades took about as long. For 54, our upgrade (#1906) landed on June 30, the day after datafusion-python 54.0.0 was published. Since #1920 we've tracked DataFusion main between releases, and this time we were on the released 55.0.0 crates eight days after DataFusion published them (#2375). So datafusion-python is now what we wait on, and with the current process our releases will usually trail DataFusion by at least a month.

What's waiting on main

310 commits have landed on main since branch-54 was cut on July 12, and 54.1.0 only picked up a handful of backported fixes. Some highlights:

Previous discussion

We discussed this in #2370 in August:

  • I listed the costs of splitting. We'd need separate tarball scripts and separate votes, it would be harder for Python users to help test and vote on the Rust release, and a Python release vote could turn up bugs in crates we'd already released, which would mean patch releases.
  • @avantgardnerio asked whether Python users would have to wait to upgrade until the Python client caught up, and showed how a 54 client could get silently different results from a 55 cluster (the preserve_nulls example).
  • @milenkovicm noted that the Python bindings are "best effort" and a "technical preview" today and that we don't have many Python users yet, and suggested reviving the separate datafusion-ballista-python repo once the FFI integration is solid, so we could release Ballista without waiting for datafusion-python.
  • The consensus was to keep the single release "until proven otherwise". I think the 55.0.0 release is a good reason to look at it again.

Some older history is relevant too:

Proposal

Release the Rust crates and the Python client separately, and keep both in this repo.

  • Rust release. The same process as today, minus the Python wheels. We cut it as soon as we're ready after a DataFusion release.
  • Python release. Its own RC tag (something like python-55.0.0-rc1), source tarball and vote, once datafusion-python has published the matching version. It would build against the ballista crates we've already published to crates.io, so the wheels contain exactly the Rust code we voted on. The Python client keeps the version of the Ballista release it wraps, so Ballista 55.0.0 would be followed by ballista 55.0.0 on PyPI.
  • Same repo. python/ stays where it is. Keeping it next to the Rust code makes it much easier to test against main (CI: Python client tests do not run on Rust changes and do not test the working tree #2372), and the separate repo didn't work out last time.

For 55.0.0, this would mean starting the Rust release now and following up with the Python release once datafusion-python 55.0.0 is out.

What this means for users:

The concerns from #2370

  • Separate scripts and votes. This is real, but it's mostly one-time work in dev/release and the wheel build workflow, and I'm happy to do it. The PMC already runs separate votes for DataFusion, datafusion-python, Comet and Ballista, so one more vote isn't unusual.
  • Python users voting on the Rust release. They would vote on the Python release instead, which is the artifact they actually use.
  • Bugs found late. If the Python release turns up a bug in the Rust crates, we'd do a patch release, the same as for any bug found after a release. I'd rather do that occasionally than hold 300 commits for six weeks.
  • FFI. feat(python): integrate Ballista through DataFusion FFI #2252 would remove the Rust-level dependency on datafusion-python, which is great. As @milenkovicm pointed out, though, we'd probably still depend on a matching datafusion Python release, so I see it as complementary rather than a fix for the release timing.

Alternatives

  • Keep the single release. Every Ballista release waits for datafusion-python, a median of 31 days after DataFusion.
  • Only split when we're blocked. Hold one vote when datafusion-python is already out, and split otherwise. Going by the table we'd end up splitting almost every time, so I'd rather have one process.
  • Move the Python client back to its own repo. This was @milenkovicm's suggestion in Validate BALLISTA_PROTOCOL_VERSION for clients, not just executors #2370. It gives us the same release independence, but it makes testing the client against main harder, and it didn't go well last time.

Questions

  • Is anyone opposed to releasing the Python client separately?
  • Should a major version mismatch between the client and the scheduler be an error?
  • Should we start the 55.0.0 Rust release now, or wait for datafusion-python 55.0.0 this time and switch to the new process for 56?

If there's support, I'm happy to update the release scripts, the wheel build workflow and the release docs.

Activity

  1. villebro commented on Sep 29, 2026

    @villebro
    Member

    I support this split. While this would mean introducing yet another release for the PMC to vote on (not my problem 😛), I feel the long term benefit of having the Ballista and DataFusion Rust crate release cycles closer together far outweighs benefits of having a Python client available when the Ballista crate ships. Another benefit of decoupling is the fact that not all bug fixes or features necessarily apply to both: the Python client can need a change that doesn't require a new Rust crate and vice versa.

  2. avantgardnerio commented on Sep 29, 2026

    @avantgardnerio
    Contributor

    I support this, because AFAIK, we still aren't able to test with DF main until we ship ballista-python 55?

    Yes, I think a major version mismatch should be an error - I think I've shown doing otherwise is outside our control.

  3. andygrove commented on Sep 29, 2026

    @andygrove
    MemberAuthor

    I support this, because AFAIK, we still aren't able to test with DF main until we ship ballista-python 55?

    Good point. Yes, this is true.

    Yes, I think a major version mismatch should be an error - I think I've shown doing otherwise is outside our control.

    I agree. I filed #2513 and am working on it separately to the change to the release process.

  4. milenkovicm commented on Sep 30, 2026

    @milenkovicm
    Contributor

    in our defence we can say this will be the best release ever :D

    do we move python to datafusion-ballista-python at the end? it would make sense if we have different release process.

  5. timsaucer commented on Sep 30, 2026

    @timsaucer
    Member

    I feel bad about the long delay in some of the datafusion-python releases. I also feel like it would help if more people were actively involved in pushing them forward. Since there is plenty of historical data to show the lag, I think this proposal makes good sense.

  6. self-assigned this
    on Oct 1, 2026
  7. andygrove commented on Oct 1, 2026

    @andygrove
    MemberAuthor

    do we move python to datafusion-ballista-python at the end? it would make sense if we have different release process.

    I don't see much value in moving it out, personally. A separate release process doesn't need a separate repo: #2512 tags the Python releases on the same release branch and builds the source tarball from just python/. Keeping it in this repo also makes it easy to run the client's tests against a cluster built from main, which is how #2374 caught the wire break in #2376. And the last time we moved it out (#635), it went unmaintained for about a year before #970 brought it back.

  8. andygrove commented on Oct 1, 2026

    @andygrove
    MemberAuthor

    I feel bad about the long delay in some of the datafusion-python releases. I also feel like it would help if more people were actively involved in pushing them forward. Since there is plenty of historical data to show the lag, I think this proposal makes good sense.

    No need to feel bad about anything. DF Python has its own priorities. I don't think it is fair for Ballista to put artificial pressure on other projects to ship faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions