Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
8824ff7
feat(iceberg): scan Iceberg tables in Ballista
NoahKusaba Oct 21, 2026
34286f2
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Oct 21, 2026
3e64e7a
refactor(iceberg): rename the crate to ballista-iceberg
NoahKusaba Oct 22, 2026
8ef54d6
feat(iceberg): pin catalog-backed tables when a query is planned
NoahKusaba Oct 26, 2026
abfeaea
feat(iceberg): mount Iceberg catalogs on the scheduler for SQL clients
NoahKusaba Oct 27, 2026
09de52a
Merge remote-tracking branch 'upstream/main' into feat/iceberg-standa…
NoahKusaba Oct 27, 2026
6ed3d0b
fix(iceberg): refuse metadata tables with a clear error
NoahKusaba Oct 27, 2026
4749413
feat(iceberg): wrap the scheduler's own session builder
NoahKusaba Oct 27, 2026
43477b2
docs(iceberg): say why PinnedTable isn't a static table provider
NoahKusaba Oct 27, 2026
5df0109
feat(iceberg): take and return the scheduler's SessionBuilder shape
NoahKusaba Oct 27, 2026
ec064ce
fix(iceberg): keep the session ID in register_iceberg
NoahKusaba Oct 27, 2026
de73e6b
Frame Iceberg payloads with an "iceberg" prefix
NoahKusaba Oct 30, 2026
f7af624
cleanup
NoahKusaba Oct 30, 2026
41673e7
Merge remote-tracking branch 'upstream/main' into feat/iceberg-standa…
NoahKusaba Oct 31, 2026
86bc19c
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Oct 31, 2026
1105c59
Move ballista-iceberg out of the root workspace, add with_iceberg
NoahKusaba Nov 2, 2026
d8b1e5d
Merge remote-tracking branch 'origin/feat/iceberg-standalone-scans' i…
NoahKusaba Nov 2, 2026
86f4237
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Nov 2, 2026
74521ca
Limit the Iceberg workflow's token to reading contents
NoahKusaba Nov 2, 2026
29a7f22
Merge remote-tracking branch 'origin/feat/iceberg-standalone-scans' i…
NoahKusaba Nov 2, 2026
44dac0e
Merge remote-tracking branch 'upstream/main' into feat/iceberg-standa…
NoahKusaba Nov 3, 2026
2519c7a
Compose the Iceberg codecs after the session's codecs
NoahKusaba Nov 3, 2026
97d465d
Fall back to Ballista's codecs in IcebergLogicalCodec::after
NoahKusaba Nov 3, 2026
e7d19b9
Correct the Iceberg docs on how the codecs are added
NoahKusaba Nov 3, 2026
9025298
Re-pin a view's Iceberg tables each time a query names it
NoahKusaba Nov 3, 2026
0b39988
Move view re-pinning to a follow-up PR
NoahKusaba Nov 6, 2026
eb78175
Pin ballista-iceberg's Ballista crates to a commit on main
NoahKusaba Nov 6, 2026
8f731f5
Refuse to encode an Iceberg scan with more than one partition
NoahKusaba Nov 6, 2026
f11230e
Document the catalog wrapper, the partition guard and the Ballista pin
NoahKusaba Nov 6, 2026
b0ec8b6
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Nov 6, 2026
a75a721
Format ballista-iceberg's Cargo.toml with taplo
NoahKusaba Nov 6, 2026
932f2c1
Merge remote-tracking branch 'origin/feat/iceberg-standalone-scans' i…
NoahKusaba Nov 6, 2026
0c2d0de
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Nov 7, 2026
56cd7bd
Send Iceberg tables without their metadata
NoahKusaba Nov 8, 2026
aa0473f
Say when the Iceberg table cache rebuilds a version again
NoahKusaba Nov 8, 2026
abcf538
Document that encrypted Iceberg tables aren't supported
NoahKusaba Nov 8, 2026
7755c0f
Document calling with_iceberg once, with every catalog
NoahKusaba Nov 8, 2026
cd54fd5
Correct which operators above an Iceberg scan run in parallel
NoahKusaba Nov 8, 2026
2cc76a3
Merge remote-tracking branch 'upstream/main' into feat/iceberg-standa…
NoahKusaba Nov 11, 2026
8c19489
Pin ballista-iceberg to a Ballista commit with #2721
NoahKusaba Nov 11, 2026
42c909b
Mark Iceberg support experimental, document split-union limits, bump …
NoahKusaba Nov 14, 2026
46e84fb
Merge branch 'main' into feat/iceberg-standalone-scans
NoahKusaba Nov 14, 2026
e5ec432
Correct the multi-partition guard's reason and a stale Rust version note
NoahKusaba Nov 14, 2026
50a7dac
Say a split UNION fails too, not only UNION ALL
NoahKusaba Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/iceberg.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# ballista-iceberg is its own workspace, outside Ballista's, and depends on a
# pinned commit of Ballista, so it doesn't block Ballista's build or release;
# this workflow checks it separately.
name: Ballista Iceberg

concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
cancel-in-progress: true

on:
push:
branches: [ "main" ]
paths:
- "integrations/iceberg/**"
- ".github/workflows/iceberg.yml"
- "!**/*.md"
pull_request:
paths:
- "integrations/iceberg/**"
- ".github/workflows/iceberg.yml"
- "!**/*.md"
workflow_dispatch:

permissions:
contents: read

defaults:
run:
working-directory: integrations/iceberg

jobs:
test:
name: check and test ballista-iceberg
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@v7.0.1
with:
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
- uses: swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 #2.9.3
with:
workspaces: integrations/iceberg
- name: Check formatting
run: cargo fmt --check
- name: Run clippy
run: |
rustup component add clippy
cargo clippy --all-targets --all-features --locked -- -D warnings
- name: Run tests
run: cargo test --all-features --locked
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# under the License.

[workspace]
exclude = ["dev/msrvcheck", "python"]
exclude = ["dev/msrvcheck", "integrations/iceberg", "python"]
members = [
"ballista-cli",
"ballista/api-types",
Expand Down
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Table of content
user-guide/versioning
user-guide/extending-components
user-guide/extensions-example
user-guide/iceberg

.. toctree::
:maxdepth: 1
Expand Down
3 changes: 2 additions & 1 deletion docs/source/user-guide/flightsql.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@ front of the scheduler and leave the frontend as-is.
- Embedders can pre-populate the catalog for every session through the
scheduler's `SessionBuilder`, so users do not have to re-run
`CREATE EXTERNAL TABLE` on each connection. See
[Extending Ballista components](extending-components.md).
[Extending Ballista components](extending-components.md), and
[Apache Iceberg](iceberg.md#sql-clients) for mounting Iceberg catalogs.

## Limitations

Expand Down
179 changes: 179 additions & 0 deletions docs/source/user-guide/iceberg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache Iceberg (experimental)

The `ballista-iceberg` crate (`integrations/iceberg`) lets Ballista scan
[Apache Iceberg](https://iceberg.apache.org/) tables. On top of
[iceberg-datafusion](https://github.com/apache/datafusion-iceberg), it adds the
codecs that send Iceberg tables and scans between the client, the scheduler and
the executors.

The crate is not published yet. It depends on unreleased revisions of
iceberg-datafusion and iceberg-rust, and needs Rust 1.95. It is its own
workspace, with its own `Cargo.lock` and CI, outside Ballista's, and depends
on a pinned commit of Ballista rather than the crates beside it, so a
DataFusion upgrade in Ballista that iceberg-rust doesn't support yet doesn't
block Ballista's build or release. Build and test it from
`integrations/iceberg`.

Iceberg support is experimental. Its behavior and API may change.

## Usage

Ready the session with `register_iceberg`, then mount an Iceberg catalog:

```rust
use std::sync::Arc;

use ballista::prelude::{SessionConfigExt, SessionContextExt};
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::{SessionConfig, SessionContext};
use ballista_iceberg::register_iceberg;
use iceberg_datafusion::IcebergCatalogProvider;

let state = SessionStateBuilder::new()
.with_config(SessionConfig::new_with_ballista())
.with_default_features()
.build();
let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?;

ctx.register_catalog("rest", Arc::new(IcebergCatalogProvider::try_new(catalog).await?));
ctx.sql("SELECT * FROM rest.ns.t").await?.show().await?;
```

A single table registered as an `IcebergCatalogTableProvider` works the same
way. Either kind reads the table's latest version in each query.

For time travel, register an `IcebergStaticTableProvider`, which keeps one
version: `IcebergStaticTableProvider::try_new_from_table_snapshot(table, snapshot_id)`.

The example registers both on a table in a temporary local warehouse, and shows
a commit reaching the live table but not the static one:

```shell
cd integrations/iceberg
cargo run --example standalone-iceberg-scan
```

## SQL clients

Flight SQL clients (JDBC, ADBC, BI tools) send SQL to the scheduler, which plans
it, so they hold no catalog. To let them query Iceberg tables, mount catalogs
on the scheduler with `SchedulerConfigExt::with_iceberg` (the `scheduler`
feature), which also adds the Iceberg codecs:

```rust
use ballista_iceberg::SchedulerConfigExt;

let rest = IcebergCatalogProvider::try_new(catalog).await?;
let config = SchedulerConfig::default()
.with_flight_sql(true)
.with_iceberg([("rest", Arc::new(rest) as _)]);
```

Executors add the codecs with `ExecutorProcessConfigExt::with_iceberg` (the
`executor` feature):

```rust
use ballista_iceberg::ExecutorProcessConfigExt;

let config = ExecutorProcessConfig::default().with_iceberg();
```

Both keep any codecs the config already has, and the scheduler keeps its
session builder, mounting the catalogs on each session it builds. Call each
once, passing the scheduler every catalog: a second call adds the codecs
again, so the scheduler's would no longer match the executors'. A client can
then run `SELECT * FROM rest.ns.t`. Tables are pinned the same way as on the
client, while the scheduler plans the query.

## How it works

Ballista sends the client's plan to the scheduler, and each stage's plan to
the executors, as bytes. The codecs replace each live Iceberg table with one
version of it: the location of its metadata file and its serialized `FileIO`.
Metadata files never change, so every node rebuilds exactly that version.

- **A catalog-backed table** is pinned to the version its catalog serves when
the client plans the query. DataFusion looks up each table a query names
once, so every scan of it, such as each side of a self-join, reads that
version, and each new query reads the latest one.
- **A static table** is sent as the version it holds.
- **A scan** is sent as the version the scheduler planned it against.

Only the location is sent, not the metadata, so a plan's size doesn't grow
with the table's history. Each node reads a version's metadata file when it
first rebuilds it, and keeps up to 16 rebuilt versions.

Only the node that plans a query talks to a catalog: the client, or the
scheduler for SQL clients. Executors never need catalog configuration.

## Combining with other table formats

@NoahKusaba NoahKusaba Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @wirybeaver, we need to establish a convention that lets different table-providers interop with each other, so for example, a single query could read Hudi and Iceberg tables together.

This is the standard I'm trying to establish: have HudiLogicalExtensionCodec::new(inner) wrap config.ballista_logical_extension_codec() instead of hard-coding BallistaLogicalExtensionCodec, and forward every method to inner, including the UDF ones.

Let me know if this works for you.

@wirybeaver wirybeaver Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NoahKusaba, the codec composition convention makes sense, including forwarding the UDF methods. I think we should establish the build and release boundary first, before landing individual format integrations, including the codec composition.

This was the motivation behind my proposal in #1241 and the split between #2388 and #2389: a minimal, SDK-free, opt-in build gate on the Ballista side, with the actual format integration owning its dependencies separately. The same convention should apply to Iceberg, Hudi, Lance, and Delta.

Your putting Iceberg in a separate crate is a useful step, but keeping it in Ballista’s root workspace and lockfile still couples dependency resolution. For example, Ballista may upgrade DataFusion before a format’s adapter supports that version. We should be able to release Ballista without waiting for every format integration to catch up.

I’d suggest establishing the minimal gate first, with format SDKs and implementations outside Ballista’s main workspace, root lockfile, and release graph, then adding Iceberg against that boundary. Each integration could maintain its own compatibility requirements and release cadence.

The gate itself would not make incompatible DataFusion versions interoperable; it would keep a format adapter’s compatibility constraints from becoming a prerequisite for Ballista’s build and release. Thus, the Codec composition comes to the rescue. Codec composition would then provide interoperability among integrations built against compatible versions.

@wirybeaver wirybeaver Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, could you also take another look at whether it would make sense to reopen #2388 and #2389, with revisions as needed?

Since the earlier discussion, hudi-rs has matured further, including the merged DataFusion 55 / Arrow 59 compatibility work in apache/hudi-rs#698. I also now have a standalone Ballista–Hudi integration exercising the existing extension points.

With Iceberg integration progressing too, I think this is a useful time to revisit the shared build and release convention. #2388 could serve as the starting point for the minimal SDK-free gate, and #2389 could demonstrate the separately maintained format implementation. Establishing that boundary first would give Iceberg, Hudi, Lance, and Delta a consistent integration model while keeping their dependency and release schedules separate from Ballista’s.

I understand the earlier maintainer-bandwidth concern still applies. Would you be open to revisiting those PRs around this build and release boundary?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still see no reason to bring Hudi as dependency

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would standard composable codec work with multiple codecs?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milenkovicm, I would separate this into two priorities:

P0: Establish the build and release boundary for Iceberg

Composable codecs address interoperability between integrations built against compatible dependencies. They do not establish an independent build or release boundary.

Suppose Ballista upgrades its DataFusion/Arrow stack while the Iceberg adapter still exposes types from the previous versions. Cargo can resolve both versions, but the integration may fail to compile where those types cross the adapter boundary.

This PR includes Iceberg in the root workspace and inherits Ballista’s dependencies, so that failure would affect workspace-wide CI. Keeping the build green would then require updating the adapter alongside Ballista or explicitly excluding it. That coordination burden is my concern; it does not mean the core crates are inherently unpublishable.

I suggest establishing an explicit opt-in build gate and independent release policy first: the integration should have its own workspace, lockfile, and CI, and remain on a compatible Ballista release until it is ready to upgrade. A feature flag alone is insufficient. Whether this requires an additional core gate or can use the existing extension points is something we can settle around that requirement.

We can focus entirely on Iceberg for this step.

P2: Consider other formats later

Hudi’s record-location indexes support efficient key-based upserts, while Lance’s scalar indexes can accelerate merge-insert operations. These offer useful capabilities for CDC and frequent-update workloads, and users may want Ballista to query those datasets.

We can defer those integrations. The immediate question is how to integrate Iceberg without making its dependency upgrade schedule part of Ballista’s core release obligations.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the P0/P2 discussion, I have made the proposed boundary concrete in two PRs:

The demo is stacked on the gate branch, so its diff isolates the integration changes. Its path dependencies exercise the unpublished gate; independently released adapters would select compatible released Ballista versions. This separates dependency resolution and release ownership, without claiming compatibility between different DataFusion/Arrow types.


`register_iceberg` and `with_iceberg` add the Iceberg codecs after the codecs
already set, with `ComposedLogicalExtensionCodec` and DataFusion's
`ComposedPhysicalExtensionCodec` (see
[Combining Codecs](extending-components.md#combining-codecs)). To read tables
of another format too, such as Delta, set its codecs first: on the session's
`SessionConfig` before calling `register_iceberg`, and on `SchedulerConfig` and
`ExecutorProcessConfig` before calling `with_iceberg`. Clients, the scheduler
and executors must all set the same codecs in the same order.

## Limitations

- **Reads only.** `INSERT INTO` an Iceberg table fails with "isn't supported
on Ballista yet".
- **No metadata tables.** Tables such as `t$snapshots` can't be sent to the
scheduler yet.
- **One task per scan.** Iceberg scans do not split into partitions yet, so one
executor reads the whole table. Operators after the first shuffle still run
in parallel. A scan reporting more than one partition fails to encode.
- **A `UNION` or `UNION ALL` can fail or return wrong rows when the adaptive
planner splits its stage between tasks.** That happens with
`ballista.scheduler.max_partitions_per_task` set, or when the executor has
fewer free task slots than the union has partitions. A union of Iceberg scans
fails with an error that suggests `SchedulerConfig::partition_sliceable_resolver`,
which standalone mode can't set;
[#2852](https://github.com/apache/datafusion-ballista/pull/2852) fixes it. A union
whose branches each merge their input into one partition, as with an
`ORDER BY` over the union, a `LIMIT` in a branch or aggregates without
`GROUP BY`, returns some rows more than once, whatever the table format
([#2854](https://github.com/apache/datafusion-ballista/issues/2854)). Setting
`ballista.planner.adaptive.enabled` to `false` avoids both.
- **No encrypted tables.** A table with `encryption.key-id` set fails to load
on the scheduler and executors, which have no key management client.
- **Storage credentials travel in plain text.** Each table's serialized
`FileIO` includes them, so the links between client, scheduler and executors
must be trusted.
- **A view keeps the version from when it was created.** DataFusion stores a
view as a planned query, and tables are pinned when a query is planned. The
same goes for a `DataFrame` or a Flight SQL prepared statement run more than
once: each run reads the version from when it was planned. To read newer
commits, recreate the view or plan the query again.
- **A separately deployed scheduler and executors** must link
`ballista-iceberg` and be readied with `with_iceberg` (see
[SQL clients](#sql-clients)). The tests cover standalone mode and an
in-process scheduler serving Flight SQL.
- **`ctx.catalog(name)` returns a wrapper.** `register_iceberg` wraps each
catalog to pin its tables, so the result doesn't downcast to the type you
registered. Keep the `Arc` you registered to reach it.
- **A mounted catalog lists its tables once.** `IcebergCatalogProvider` reads
the table names when it is built, so a table created afterwards isn't
visible until the catalog is mounted again.
Loading
Loading