Skip to content

Validate BALLISTA_PROTOCOL_VERSION for clients, not just executors #2370

Closed

Description

@andygrove

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The scheduler validates BALLISTA_PROTOCOL_VERSION when an executor registers, but there is no equivalent check for clients. check_protocol_version in ballista/scheduler/src/scheduler_server/grpc.rs is only wired into the executor registration RPCs, and nothing in ballista/core/src/execution_plans/distributed_query.rs or the client crate sends or verifies a protocol version.

The result is that a client built against a different Ballista version connects happily, plans and runs a job successfully, and then fails much later with a confusing error when it tries to read the results.

This came up while investigating #2367, where the client asked the executor for .../{job}/3/0/data.arrow while the executor had written .../{job}/3/0/data-0.arrow. The shuffle file layout for the final stage changed after 54.0.0: the passthrough ShuffleWriterExec used to report file_id: None and write data.arrow, and now reports file_id: Some(task_id) and writes data-{task_id}.arrow. A client and cluster that disagree about that surface as:

Failed to open partition file at ".../data.arrow": Os { code: 2, kind: NotFound, ... }

which reads like data loss or a cleanup race rather than a version mismatch. The job even reports as Completed, so the first thing a user suspects is the shuffle machinery.

Describe the solution you'd like

Extend the existing protocol version handshake to the client:

  • Have the client send its compiled-in BALLISTA_PROTOCOL_VERSION on ExecuteQuery (or on a dedicated handshake RPC).
  • Have the scheduler reject a mismatch with a clear failed_precondition naming both versions, the same way it already does for executors.
  • Bump BALLISTA_PROTOCOL_VERSION whenever the on-disk shuffle layout or the client-visible wire contract changes, and document that rule next to the constant in ballista/core/src/lib.rs.

Describe alternatives you've considered

Only bumping the version on wire-format changes and leaving layout changes alone. That does not help here, since the proto messages (PartitionLocation, ShuffleWritePartition, FetchPartition) are byte-identical between 54.0.0 and main. What actually diverged is the meaning of file_id, so the version rule needs to cover the shuffle layout too.

Alternatively, make the failure message itself smarter (e.g. when a partition file is missing, check whether the sibling name under the other layout exists and say so). Useful as a diagnostic, but it papers over the mismatch instead of refusing it up front.

Additional context

Related: #2367.

Activity

  1. andygrove commented on Aug 25, 2026

    @andygrove
    MemberAuthor

    Two notes for whoever picks this up.

    We now have a concrete case. #2367 turned out to be a ballista-core 52.0.0 Python extension talking to a current cluster. 52.0.0 predates file_id and is_sort_shuffle on FetchPartition (added in 8c6c864, #1527), so the client sent a fetch ticket with no file_id, the executor took the (None, false) branch of create_shuffle_path and looked for data.arrow while the writer had produced data-{task_id}.arrow. The job completed successfully and then the result fetch failed with NotFound. Four releases of skew, no error until the very last step, and the symptom pointed at the shuffle machinery rather than at the client. That is the failure mode this issue is about.

    There is a policy question to settle first, and it conflicts with #2374. The Python client cannot track the workspace DataFusion version. pyballista re-exports datafusion-python types (python/src/lib.rs), so datafusion-python, datafusion and ballista-core all have to link the same datafusion, and crates.io datafusion-python is still at 54.0.0 while the workspace is on the 55.0.0-rc3 tag. The practical consequence is that the Python client normally lags the cluster by a DataFusion release cycle, so a supported Python user is routinely running a client that is a release or two behind.

    That means a strict equality check on BALLISTA_PROTOCOL_VERSION, like the one executors get today, would reject the normal Python setup. #2374 adds a CI job that runs the pinned 54.0.0 client against a cluster built from the branch, precisely because that is what users run, and a strict check would turn that job red by design.

    So this probably wants something more nuanced than the executor check, for example:

    • a minimum supported client version rather than exact equality, with the version bumped only when something client-visible actually breaks, or
    • a capability or feature negotiation at connect time, so the scheduler can tell whether the client understands file_id rather than inferring it from a version number.

    Either way the win is the same and worth having: fail at connect with a message naming both versions, instead of at the last fetch with a missing file.

  2. milenkovicm commented on Aug 25, 2026

    @milenkovicm
    Contributor

    Can scheduler return version headers, so client can just check for scheduler version? I think I have PR ready to cover this case

  3. avantgardnerio commented on Aug 25, 2026

    @avantgardnerio
    Contributor

    a supported Python user is routinely running a client that is a release or two behind

    Should python users wait to upgrade Ballista until the python client catches up? Or is that not an operational reality?

    Even if we don't enforce a strict version, could DF make a breaking change in their protocol - even in cases where a Ballista release made no protocol changes at all?

    Pardon my ignorance, I haven't done much with Python & DF/Ballista together.

  4. avantgardnerio commented on Aug 25, 2026

    @avantgardnerio
    Contributor

    ie, between DF 54 & 55rc3, it appears 3 fields were lost:

      fields: 689 -> 749
      REMOVED (3):
        ListingTableScanNode#8   bool collect_stat        reserved_in_new=True
        ListingTableScanNode#9   uint32 target_partitions reserved_in_new=True
        UnnestOptions#1          bool preserve_nulls      reserved_in_new=True
      RETYPED (0)
      RENAMED (0)
    

    According to Claude, this is the probable outcome:

    UnnestOptions.preserve_nulls → replaced by null_handling at field 3. The 54 client's field 1 is dropped; DF55 defaults to Preserve (proto-models/src/from_proto.rs:70,
        common/src/unnest.rs:123). And proto3 omits false, so preserve_nulls=false — the DROP case — silently becomes PRESERVE. Different rows, no error.
    
      ┌────────┬─────────┬───────────┐
      │ client │ cluster │ semantics │
      ├────────┼─────────┼───────────┤
      │ 54     │ 54      │ Drop      │
      ├────────┼─────────┼───────────┤
      │ 55     │ 55      │ Drop      │
      ├────────┼─────────┼───────────┤
      │ 54     │ 55      │ Preserve  │
      └────────┴─────────┴───────────┘
    
  5. milenkovicm commented on Aug 25, 2026

    @milenkovicm
    Contributor

    do we have way to provide backward compatibility of protocols if we rely on datafusion which may have breaking changes between releases? do we want to enforce backward compatibility on datafusion side ?

  6. milenkovicm commented on Aug 25, 2026

    @milenkovicm
    Contributor

    i dont think we have too many python users to worry about at this moment, and current python bindings are "best effort" / "technical preview"

  7. andygrove commented on Aug 26, 2026

    @andygrove
    MemberAuthor

    I think we haven't really made a decision on what the compatibility should like look. I think it would be fine for Ballista and PyBallista to have different release cycles and no guarantee of compatibility between mixed versions, but it would be good to have queries fail with mixed versions rather than produce incorrect results

  8. andygrove commented on Aug 26, 2026

    @andygrove
    MemberAuthor

    I'm going around in circles thinking about this. We currently have one vote for both Ballista and PyBallista for the same version, and consequences are:

    • We have to wait for datafusion-python release before we can release ballista
    • Python users can help test the release and vote on it

    If we split into separate release cycles for ballista vs PyBallista:

    • We will need separate create-tarball/release-tarball scripts for each and separate votes for each
    • It will be more difficult for Python users to vote on ballista releases
    • When we start the vote on PyBallista, there could be issues with the already released Ballista crates. This would generate additional work for patch releases and more release votes
  9. andygrove commented on Aug 26, 2026

    @andygrove
    MemberAuthor

    cc @timsaucer since he is a potential user of PyBallista

  10. avantgardnerio commented on Aug 26, 2026

    @avantgardnerio
    Contributor

    Sounds like maybe we just stick with option #1, and help out @timsaucer with the FFI stuff if we can? IIUC, that should remove our dependency on datafusion-python entirely.

  11. milenkovicm commented on Aug 26, 2026

    @milenkovicm
    Contributor

    Until proven otherwise option 1 makes sense.

    Release train may get more complicated if/when we bring iceberg

  12. avantgardnerio commented on Aug 26, 2026

    @avantgardnerio
    Contributor

    Upon in depth review of Tim's changes, I think it wouldn't solve preserve_nulls - I think #1 is the way to go. (which is what we do already).

  13. milenkovicm commented on Aug 26, 2026

    @milenkovicm
    Contributor

    Sounds like maybe we just stick with option #1, and help out @timsaucer with the FFI stuff if we can? IIUC, that should remove our dependency on datafusion-python entirely.

    I'm not sure if we can avoid df py dependency, if we do not depend on df py rust code we might still depend on python .

  14. milenkovicm commented on Aug 26, 2026

    @milenkovicm
    Contributor

    Also, once with integrate with py df it's our concern to handle version mismatch, py df will not care about this issue the same way df does not care about it at the moment

  15. milenkovicm commented on Aug 26, 2026

    @milenkovicm
    Contributor

    Once we get solid DataFusion python integration would it make sense reacting DataFusion ballista python repository? This would mean we go with option 2 for release but we already have most of the infrastructure ready and we could release ballista without df python being released

    (Not sure why I like resurrecting dead repositories 😀)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions