Skip to content

Run a job's final commit step on the scheduler, not an executor #2622

Open

Description

@NoahKusaba

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Table formats such as Iceberg and Delta finish a write with one commit. Every writer task's data files are gathered in one place and committed through the catalog, exactly once, after all writes have succeeded.

DataFusion represents this commit as an ordinary plan node at the root of the INSERT plan. For Iceberg (#2217) that plan is:

IcebergCommitExec          1 partition: one catalog commit, outputs a `count` row
  CoalescePartitionsExec
    IcebergWriteExec       N partitions: writes Parquet files, outputs their metadata

Ballista sends every plan node to an executor, so the commit runs as a one-task final stage on some executor. That has three costs, all of which come from where the commit runs rather than from the table format:

  1. Executors need the catalog. The plan has to carry the catalog's configuration, credentials included, so that an executor can rebuild the catalog client and commit.
  2. Commits must be idempotent. Ballista retries failed tasks, so a commit task can run twice. feat: Add Iceberg Support #2217 has to give each commit an id and refuse to run a commit without one.
  3. Every executor has to cache catalog clients, with logic to evict and retry stale ones.

Spark avoids all three because its write API ends on the driver. Writer tasks send small commit messages back, and the driver calls BatchWrite.commit(messages) once. Ballista has nowhere to run code on the scheduler at the end of a job: succeed_job only saves the job's final state.

Describe the solution you'd like

A way for a job's final step to run on the scheduler, after all of its stages have succeeded. Its small result becomes the job's result.

A minimal version, which I have a working prototype of:

  • A config hook to mark the final step: SessionConfigExt::with_ballista_scheduler_root(predicate). When a job's root operator matches, the scheduler removes it before splitting the plan into stages, and the rest runs on executors as usual.
  • Running it when the job ends: once the job succeeds, the scheduler reads the final stage's output (here, the data files' metadata) with a ShuffleReaderExec and runs the removed root on it.
  • Returning the result: the root's output comes back to the client in a new field, SuccessfulJob.inline_result, as Arrow IPC bytes. That's a few rows for a commit.

For Iceberg this is a one-line opt-in that matches IcebergCommitExec. It needs no change to datafusion-iceberg or to the plan.

Points for discussion:

  • Scope: the hook could run any root operator, but it is only meant for small final steps such as commits. Should the API be narrowed to that, for example a JobFinalizer trait with a capped result size?
  • Job state: the prototype reports a finished job as running until the commit is done. An explicit Finalizing state in the job graph would be cleaner.
  • Cancellation: if a job is cancelled while its commit runs, the commit may still go through. Spark has the same behaviour; here it should at least be documented.

Describe alternatives you've considered

  • Keep committing on an executor (what feat: Add Iceberg Support #2217 does). It works, but has the three costs above.
  • A Spark-style commit protocol: writer tasks return commit messages in their task status, and a scheduler-side hook commits them. This avoids writing the metadata to shuffle files and reading it back, but it needs a new writer/committer API in Ballista core, and the write node would have to report its output through task status.
  • Run the final stage on the scheduler as an ordinary stage that the scheduler executes itself. This is closer to how the job graph already works, but it changes both graph implementations, and the scheduler would have to serve that stage's output.

Additional context

  • Use case: Iceberg writes in feat: Add Iceberg Support #2217, part of [EPIC] Production-grade Apache Iceberg scans #2174. Any sink that ends with a single commit would benefit, Delta Lake and Hudi included.
  • Prototype: I'm happy to open it as a draft PR for discussion. It covers both the adaptive and the static planner and both pull and push job status. It includes a standalone test and a Docker test against a real Iceberg REST catalog.

Activity

  1. NoahKusaba commented on Oct 18, 2026

    @NoahKusaba
    ContributorAuthor

    I'll follow up once Iceberg Scans are addressed

  2. changed the title [-]Run a job's final commit step on the scheduler, not an executor[/-] [+][Iceberg] Run a job's final commit step on the scheduler, not an executor[/+] on Oct 18, 2026
  3. milenkovicm commented on Oct 20, 2026

    @milenkovicm
    Contributor

    one note @NoahKusaba at the moment catalog support is on client side, scheduler session does not have a catalog

  4. NoahKusaba commented on Oct 21, 2026

    @NoahKusaba
    ContributorAuthor

    3. one note @NoahKusaba at the moment catalog support is on client side, scheduler session does not have a catalog

    I guess if we want to keep that design, commits can run on the client then?

    On the Ballista side, this needs a client-side hook that splits an INSERT into the distributed writes and a local commit, with the client reading the writes' output through the normal result path.

    I thought it might be better to have scheduler do commits, as it would likely be closer to the executors / data-lake in a real production setting and reduce I/O cost.

    Running commits on the executor seems like an extra IO for executors -> Scheduler -> executor (requires more upstream changes too).

  5. milenkovicm commented on Oct 22, 2026

    @milenkovicm
    Contributor

    I'm not prescribing solution just mentioning the fact

  6. milenkovicm commented on Oct 22, 2026

    @milenkovicm
    Contributor

    Adding an extra physical plan node would be an easy option.

    Maybe we should put pros and cons table and a bit of design documentation before we commit to solution

  7. NoahKusaba commented on Oct 27, 2026

    @NoahKusaba
    ContributorAuthor

    Thanks @milenkovicm, agreed. Here's a short design note. It replaces my earlier client-commit suggestion.

    Catalogs on the scheduler

    The scheduler can get catalogs through the existing SchedulerConfig::with_override_session_builder, with no core change. #2638 does this with iceberg_session_builder(catalogs), which Flight SQL clients need anyway: they send SQL for the scheduler to plan, so they have no catalog of their own. Executors never need one.

    That leaves the question of where the commit runs.

    Options

    Trade-offs
    A. Commit on the client The client commits through its own catalog, but must stay connected until the job ends. Doesn't work for Flight SQL clients, which have no catalog. Slowest: the file lists travel out of the cluster to the client before the commit.
    B. Final step on the scheduler Writers save their file lists as normal stage output. Once every writer is done, the scheduler fetches the lists from the executors, then commits. A small change, contained in the scheduler. Faster than A: the file lists stop at the scheduler instead of continuing to the client.
    C. Spark-style commit messages Each writer puts its file list in its task-status message, so the scheduler already has every list when the last writer finishes and needs no fetch. But task status must carry output, which adds load to status messages and scheduler memory, needs a size cap, and changes executor, scheduler and proto. Fastest: it skips B's final fetch from the executors, but the scheduler holds every writer's file list in memory until the job ends.
  8. NoahKusaba commented on Oct 29, 2026

    @NoahKusaba
    ContributorAuthor

    I think I'm going to pursue the Spark design, it's the more performant one and there's no reason to not support the same design if Spark can handle it fine.

  9. added 3 commits that reference this issue on Nov 8, 2026
    232b074
    9ececb4
    209c2e8
  10. milenkovicm commented on Nov 13, 2026

    @milenkovicm
    Contributor

    Would it make sense to make this generic, non iceberg specific ?

  11. changed the title [-][Iceberg] Run a job's final commit step on the scheduler, not an executor[/-] [+]Run a job's final commit step on the scheduler, not an executor[/+] on Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions