You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Validate BALLISTA_PROTOCOL_VERSION for clients, not just executors #2370
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The scheduler validates BALLISTA_PROTOCOL_VERSION when an executor registers, but there is no equivalent check for clients. check_protocol_version in ballista/scheduler/src/scheduler_server/grpc.rs is only wired into the executor registration RPCs, and nothing in ballista/core/src/execution_plans/distributed_query.rs or the client crate sends or verifies a protocol version.
The result is that a client built against a different Ballista version connects happily, plans and runs a job successfully, and then fails much later with a confusing error when it tries to read the results.
This came up while investigating #2367, where the client asked the executor for .../{job}/3/0/data.arrow while the executor had written .../{job}/3/0/data-0.arrow. The shuffle file layout for the final stage changed after 54.0.0: the passthrough ShuffleWriterExec used to report file_id: None and write data.arrow, and now reports file_id: Some(task_id) and writes data-{task_id}.arrow. A client and cluster that disagree about that surface as:
Failed to open partition file at ".../data.arrow": Os { code: 2, kind: NotFound, ... }
which reads like data loss or a cleanup race rather than a version mismatch. The job even reports as Completed, so the first thing a user suspects is the shuffle machinery.
Describe the solution you'd like
Extend the existing protocol version handshake to the client:
Have the client send its compiled-in BALLISTA_PROTOCOL_VERSION on ExecuteQuery (or on a dedicated handshake RPC).
Have the scheduler reject a mismatch with a clear failed_precondition naming both versions, the same way it already does for executors.
Bump BALLISTA_PROTOCOL_VERSION whenever the on-disk shuffle layout or the client-visible wire contract changes, and document that rule next to the constant in ballista/core/src/lib.rs.
Describe alternatives you've considered
Only bumping the version on wire-format changes and leaving layout changes alone. That does not help here, since the proto messages (PartitionLocation, ShuffleWritePartition, FetchPartition) are byte-identical between 54.0.0 and main. What actually diverged is the meaning of file_id, so the version rule needs to cover the shuffle layout too.
Alternatively, make the failure message itself smarter (e.g. when a partition file is missing, check whether the sibling name under the other layout exists and say so). Useful as a diagnostic, but it papers over the mismatch instead of refusing it up front.
We now have a concrete case.#2367 turned out to be a ballista-core 52.0.0 Python extension talking to a current cluster. 52.0.0 predates file_id and is_sort_shuffle on FetchPartition (added in 8c6c864, #1527), so the client sent a fetch ticket with no file_id, the executor took the (None, false) branch of create_shuffle_path and looked for data.arrow while the writer had produced data-{task_id}.arrow. The job completed successfully and then the result fetch failed with NotFound. Four releases of skew, no error until the very last step, and the symptom pointed at the shuffle machinery rather than at the client. That is the failure mode this issue is about.
There is a policy question to settle first, and it conflicts with #2374. The Python client cannot track the workspace DataFusion version. pyballista re-exports datafusion-python types (python/src/lib.rs), so datafusion-python, datafusion and ballista-core all have to link the same datafusion, and crates.io datafusion-python is still at 54.0.0 while the workspace is on the 55.0.0-rc3 tag. The practical consequence is that the Python client normally lags the cluster by a DataFusion release cycle, so a supported Python user is routinely running a client that is a release or two behind.
That means a strict equality check on BALLISTA_PROTOCOL_VERSION, like the one executors get today, would reject the normal Python setup. #2374 adds a CI job that runs the pinned 54.0.0 client against a cluster built from the branch, precisely because that is what users run, and a strict check would turn that job red by design.
So this probably wants something more nuanced than the executor check, for example:
a minimum supported client version rather than exact equality, with the version bumped only when something client-visible actually breaks, or
a capability or feature negotiation at connect time, so the scheduler can tell whether the client understands file_id rather than inferring it from a version number.
Either way the win is the same and worth having: fail at connect with a message naming both versions, instead of at the last fetch with a missing file.
a supported Python user is routinely running a client that is a release or two behind
Should python users wait to upgrade Ballista until the python client catches up? Or is that not an operational reality?
Even if we don't enforce a strict version, could DF make a breaking change in their protocol - even in cases where a Ballista release made no protocol changes at all?
Pardon my ignorance, I haven't done much with Python & DF/Ballista together.
According to Claude, this is the probable outcome:
UnnestOptions.preserve_nulls → replaced by null_handling at field 3. The 54 client's field 1 is dropped; DF55 defaults to Preserve (proto-models/src/from_proto.rs:70,
common/src/unnest.rs:123). And proto3 omits false, so preserve_nulls=false — the DROP case — silently becomes PRESERVE. Different rows, no error.
do we have way to provide backward compatibility of protocols if we rely on datafusion which may have breaking changes between releases? do we want to enforce backward compatibility on datafusion side ?
I think we haven't really made a decision on what the compatibility should like look. I think it would be fine for Ballista and PyBallista to have different release cycles and no guarantee of compatibility between mixed versions, but it would be good to have queries fail with mixed versions rather than produce incorrect results
I'm going around in circles thinking about this. We currently have one vote for both Ballista and PyBallista for the same version, and consequences are:
We have to wait for datafusion-python release before we can release ballista
Python users can help test the release and vote on it
If we split into separate release cycles for ballista vs PyBallista:
We will need separate create-tarball/release-tarball scripts for each and separate votes for each
It will be more difficult for Python users to vote on ballista releases
When we start the vote on PyBallista, there could be issues with the already released Ballista crates. This would generate additional work for patch releases and more release votes
Sounds like maybe we just stick with option #1, and help out @timsaucer with the FFI stuff if we can? IIUC, that should remove our dependency on datafusion-python entirely.
Sounds like maybe we just stick with option #1, and help out @timsaucer with the FFI stuff if we can? IIUC, that should remove our dependency on datafusion-python entirely.
I'm not sure if we can avoid df py dependency, if we do not depend on df py rust code we might still depend on python .
Also, once with integrate with py df it's our concern to handle version mismatch, py df will not care about this issue the same way df does not care about it at the moment
Once we get solid DataFusion python integration would it make sense reacting DataFusion ballista python repository? This would mean we go with option 2 for release but we already have most of the infrastructure ready and we could release ballista without df python being released
(Not sure why I like resurrecting dead repositories 😀)
Activity
andygrove commentedon Aug 25, 2026
Two notes for whoever picks this up.
We now have a concrete case. #2367 turned out to be a
ballista-core52.0.0 Python extension talking to a current cluster. 52.0.0 predatesfile_idandis_sort_shuffleonFetchPartition(added in 8c6c864, #1527), so the client sent a fetch ticket with nofile_id, the executor took the(None, false)branch ofcreate_shuffle_pathand looked fordata.arrowwhile the writer had produceddata-{task_id}.arrow. The job completed successfully and then the result fetch failed withNotFound. Four releases of skew, no error until the very last step, and the symptom pointed at the shuffle machinery rather than at the client. That is the failure mode this issue is about.There is a policy question to settle first, and it conflicts with #2374. The Python client cannot track the workspace DataFusion version.
pyballistare-exports datafusion-python types (python/src/lib.rs), sodatafusion-python,datafusionandballista-coreall have to link the samedatafusion, and crates.iodatafusion-pythonis still at 54.0.0 while the workspace is on the55.0.0-rc3tag. The practical consequence is that the Python client normally lags the cluster by a DataFusion release cycle, so a supported Python user is routinely running a client that is a release or two behind.That means a strict equality check on
BALLISTA_PROTOCOL_VERSION, like the one executors get today, would reject the normal Python setup. #2374 adds a CI job that runs the pinned 54.0.0 client against a cluster built from the branch, precisely because that is what users run, and a strict check would turn that job red by design.So this probably wants something more nuanced than the executor check, for example:
file_idrather than inferring it from a version number.Either way the win is the same and worth having: fail at connect with a message naming both versions, instead of at the last fetch with a missing file.
milenkovicm commentedon Aug 25, 2026
Can scheduler return version headers, so client can just check for scheduler version? I think I have PR ready to cover this case
avantgardnerio commentedon Aug 25, 2026
Should python users wait to upgrade Ballista until the python client catches up? Or is that not an operational reality?
Even if we don't enforce a strict version, could DF make a breaking change in their protocol - even in cases where a Ballista release made no protocol changes at all?
Pardon my ignorance, I haven't done much with Python & DF/Ballista together.
avantgardnerio commentedon Aug 25, 2026
ie, between DF 54 & 55rc3, it appears 3 fields were lost:
According to Claude, this is the probable outcome:
milenkovicm commentedon Aug 25, 2026
do we have way to provide backward compatibility of protocols if we rely on datafusion which may have breaking changes between releases? do we want to enforce backward compatibility on datafusion side ?
milenkovicm commentedon Aug 25, 2026
i dont think we have too many python users to worry about at this moment, and current python bindings are "best effort" / "technical preview"
andygrove commentedon Aug 26, 2026
I think we haven't really made a decision on what the compatibility should like look. I think it would be fine for Ballista and PyBallista to have different release cycles and no guarantee of compatibility between mixed versions, but it would be good to have queries fail with mixed versions rather than produce incorrect results
andygrove commentedon Aug 26, 2026
I'm going around in circles thinking about this. We currently have one vote for both Ballista and PyBallista for the same version, and consequences are:
If we split into separate release cycles for ballista vs PyBallista:
andygrove commentedon Aug 26, 2026
cc @timsaucer since he is a potential user of PyBallista
avantgardnerio commentedon Aug 26, 2026
Sounds like maybe we just stick with option #1, and help out @timsaucer with the FFI stuff if we can? IIUC, that should remove our dependency on datafusion-python entirely.
milenkovicm commentedon Aug 26, 2026
Until proven otherwise option 1 makes sense.
Release train may get more complicated if/when we bring iceberg
avantgardnerio commentedon Aug 26, 2026
Upon in depth review of Tim's changes, I think it wouldn't solve
preserve_nulls- I think #1 is the way to go. (which is what we do already).milenkovicm commentedon Aug 26, 2026
I'm not sure if we can avoid df py dependency, if we do not depend on df py rust code we might still depend on python .
milenkovicm commentedon Aug 26, 2026
Also, once with integrate with py df it's our concern to handle version mismatch, py df will not care about this issue the same way df does not care about it at the moment
milenkovicm commentedon Aug 26, 2026
Once we get solid DataFusion python integration would it make sense reacting DataFusion ballista python repository? This would mean we go with option 2 for release but we already have most of the infrastructure ready and we could release ballista without df python being released
(Not sure why I like resurrecting dead repositories 😀)