Repository navigation
feat(iceberg): scan Iceberg tables in Ballista #2638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8824ff7
34286f2
3e64e7a
8ef54d6
abfeaea
09de52a
6ed3d0b
4749413
43477b2
5df0109
ec064ce
de73e6b
f7af624
41673e7
86bc19c
1105c59
d8b1e5d
86f4237
74521ca
29a7f22
44dac0e
2519c7a
97d465d
e7d19b9
9025298
0b39988
eb78175
8f731f5
f11230e
b0ec8b6
a75a721
932f2c1
0c2d0de
56cd7bd
aa0473f
abcf538
7755c0f
cd54fd5
2cc76a3
8c19489
42c909b
46e84fb
e5ec432
50a7dac
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
|
|
||
| # ballista-iceberg is its own workspace, outside Ballista's, and depends on a | ||
| # pinned commit of Ballista, so it doesn't block Ballista's build or release; | ||
| # this workflow checks it separately. | ||
| name: Ballista Iceberg | ||
|
|
||
| concurrency: | ||
| group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} | ||
| cancel-in-progress: true | ||
|
|
||
| on: | ||
| push: | ||
| branches: [ "main" ] | ||
| paths: | ||
| - "integrations/iceberg/**" | ||
| - ".github/workflows/iceberg.yml" | ||
| - "!**/*.md" | ||
| pull_request: | ||
| paths: | ||
| - "integrations/iceberg/**" | ||
| - ".github/workflows/iceberg.yml" | ||
| - "!**/*.md" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| defaults: | ||
| run: | ||
| working-directory: integrations/iceberg | ||
|
|
||
| jobs: | ||
| test: | ||
| name: check and test ballista-iceberg | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: amd64/rust | ||
| steps: | ||
| - uses: actions/checkout@v7.0.1 | ||
| with: | ||
| fetch-depth: 1 | ||
| - name: Setup Rust toolchain | ||
| uses: ./.github/actions/setup-builder | ||
| - uses: swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 #2.9.3 | ||
| with: | ||
| workspaces: integrations/iceberg | ||
| - name: Check formatting | ||
| run: cargo fmt --check | ||
| - name: Run clippy | ||
| run: | | ||
| rustup component add clippy | ||
| cargo clippy --all-targets --all-features --locked -- -D warnings | ||
| - name: Run tests | ||
| run: cargo test --all-features --locked | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| <!--- | ||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
| --> | ||
|
|
||
| # Apache Iceberg (experimental) | ||
|
|
||
| The `ballista-iceberg` crate (`integrations/iceberg`) lets Ballista scan | ||
| [Apache Iceberg](https://iceberg.apache.org/) tables. On top of | ||
| [iceberg-datafusion](https://github.com/apache/datafusion-iceberg), it adds the | ||
| codecs that send Iceberg tables and scans between the client, the scheduler and | ||
| the executors. | ||
|
|
||
| The crate is not published yet. It depends on unreleased revisions of | ||
| iceberg-datafusion and iceberg-rust, and needs Rust 1.95. It is its own | ||
| workspace, with its own `Cargo.lock` and CI, outside Ballista's, and depends | ||
| on a pinned commit of Ballista rather than the crates beside it, so a | ||
| DataFusion upgrade in Ballista that iceberg-rust doesn't support yet doesn't | ||
| block Ballista's build or release. Build and test it from | ||
| `integrations/iceberg`. | ||
|
|
||
| Iceberg support is experimental. Its behavior and API may change. | ||
|
|
||
| ## Usage | ||
|
|
||
| Ready the session with `register_iceberg`, then mount an Iceberg catalog: | ||
|
|
||
| ```rust | ||
| use std::sync::Arc; | ||
|
|
||
| use ballista::prelude::{SessionConfigExt, SessionContextExt}; | ||
| use datafusion::execution::SessionStateBuilder; | ||
| use datafusion::prelude::{SessionConfig, SessionContext}; | ||
| use ballista_iceberg::register_iceberg; | ||
| use iceberg_datafusion::IcebergCatalogProvider; | ||
|
|
||
| let state = SessionStateBuilder::new() | ||
| .with_config(SessionConfig::new_with_ballista()) | ||
| .with_default_features() | ||
| .build(); | ||
| let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?; | ||
|
|
||
| ctx.register_catalog("rest", Arc::new(IcebergCatalogProvider::try_new(catalog).await?)); | ||
| ctx.sql("SELECT * FROM rest.ns.t").await?.show().await?; | ||
| ``` | ||
|
|
||
| A single table registered as an `IcebergCatalogTableProvider` works the same | ||
| way. Either kind reads the table's latest version in each query. | ||
|
|
||
| For time travel, register an `IcebergStaticTableProvider`, which keeps one | ||
| version: `IcebergStaticTableProvider::try_new_from_table_snapshot(table, snapshot_id)`. | ||
|
|
||
| The example registers both on a table in a temporary local warehouse, and shows | ||
| a commit reaching the live table but not the static one: | ||
|
|
||
| ```shell | ||
| cd integrations/iceberg | ||
| cargo run --example standalone-iceberg-scan | ||
| ``` | ||
|
|
||
| ## SQL clients | ||
|
|
||
| Flight SQL clients (JDBC, ADBC, BI tools) send SQL to the scheduler, which plans | ||
| it, so they hold no catalog. To let them query Iceberg tables, mount catalogs | ||
| on the scheduler with `SchedulerConfigExt::with_iceberg` (the `scheduler` | ||
| feature), which also adds the Iceberg codecs: | ||
|
|
||
| ```rust | ||
| use ballista_iceberg::SchedulerConfigExt; | ||
|
|
||
| let rest = IcebergCatalogProvider::try_new(catalog).await?; | ||
| let config = SchedulerConfig::default() | ||
| .with_flight_sql(true) | ||
| .with_iceberg([("rest", Arc::new(rest) as _)]); | ||
| ``` | ||
|
|
||
| Executors add the codecs with `ExecutorProcessConfigExt::with_iceberg` (the | ||
| `executor` feature): | ||
|
|
||
| ```rust | ||
| use ballista_iceberg::ExecutorProcessConfigExt; | ||
|
|
||
| let config = ExecutorProcessConfig::default().with_iceberg(); | ||
| ``` | ||
|
|
||
| Both keep any codecs the config already has, and the scheduler keeps its | ||
| session builder, mounting the catalogs on each session it builds. Call each | ||
| once, passing the scheduler every catalog: a second call adds the codecs | ||
| again, so the scheduler's would no longer match the executors'. A client can | ||
| then run `SELECT * FROM rest.ns.t`. Tables are pinned the same way as on the | ||
| client, while the scheduler plans the query. | ||
|
|
||
| ## How it works | ||
|
|
||
| Ballista sends the client's plan to the scheduler, and each stage's plan to | ||
| the executors, as bytes. The codecs replace each live Iceberg table with one | ||
| version of it: the location of its metadata file and its serialized `FileIO`. | ||
| Metadata files never change, so every node rebuilds exactly that version. | ||
|
|
||
| - **A catalog-backed table** is pinned to the version its catalog serves when | ||
| the client plans the query. DataFusion looks up each table a query names | ||
| once, so every scan of it, such as each side of a self-join, reads that | ||
| version, and each new query reads the latest one. | ||
| - **A static table** is sent as the version it holds. | ||
| - **A scan** is sent as the version the scheduler planned it against. | ||
|
|
||
| Only the location is sent, not the metadata, so a plan's size doesn't grow | ||
| with the table's history. Each node reads a version's metadata file when it | ||
| first rebuilds it, and keeps up to 16 rebuilt versions. | ||
|
|
||
| Only the node that plans a query talks to a catalog: the client, or the | ||
| scheduler for SQL clients. Executors never need catalog configuration. | ||
|
|
||
| ## Combining with other table formats | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hey @wirybeaver, we need to establish a convention that lets different table-providers interop with each other, so for example, a single query could read Hudi and Iceberg tables together. This is the standard I'm trying to establish: have HudiLogicalExtensionCodec::new(inner) wrap config.ballista_logical_extension_codec() instead of hard-coding BallistaLogicalExtensionCodec, and forward every method to inner, including the UDF ones. Let me know if this works for you.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @NoahKusaba, the codec composition convention makes sense, including forwarding the UDF methods. I think we should establish the build and release boundary first, before landing individual format integrations, including the codec composition. This was the motivation behind my proposal in #1241 and the split between #2388 and #2389: a minimal, SDK-free, opt-in build gate on the Ballista side, with the actual format integration owning its dependencies separately. The same convention should apply to Iceberg, Hudi, Lance, and Delta. Your putting Iceberg in a separate crate is a useful step, but keeping it in Ballista’s root workspace and lockfile still couples dependency resolution. For example, Ballista may upgrade DataFusion before a format’s adapter supports that version. We should be able to release Ballista without waiting for every format integration to catch up. I’d suggest establishing the minimal gate first, with format SDKs and implementations outside Ballista’s main workspace, root lockfile, and release graph, then adding Iceberg against that boundary. Each integration could maintain its own compatibility requirements and release cadence. The gate itself would not make incompatible DataFusion versions interoperable; it would keep a format adapter’s compatibility constraints from becoming a prerequisite for Ballista’s build and release. Thus, the Codec composition comes to the rescue. Codec composition would then provide interoperability among integrations built against compatible versions.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @andygrove, could you also take another look at whether it would make sense to reopen #2388 and #2389, with revisions as needed? Since the earlier discussion, hudi-rs has matured further, including the merged DataFusion 55 / Arrow 59 compatibility work in apache/hudi-rs#698. I also now have a standalone Ballista–Hudi integration exercising the existing extension points. With Iceberg integration progressing too, I think this is a useful time to revisit the shared build and release convention. #2388 could serve as the starting point for the minimal SDK-free gate, and #2389 could demonstrate the separately maintained format implementation. Establishing that boundary first would give Iceberg, Hudi, Lance, and Delta a consistent integration model while keeping their dependency and release schedules separate from Ballista’s. I understand the earlier maintainer-bandwidth concern still applies. Would you be open to revisiting those PRs around this build and release boundary?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I still see no reason to bring Hudi as dependency
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would standard composable codec work with multiple codecs?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @milenkovicm, I would separate this into two priorities: P0: Establish the build and release boundary for IcebergComposable codecs address interoperability between integrations built against compatible dependencies. They do not establish an independent build or release boundary. Suppose Ballista upgrades its DataFusion/Arrow stack while the Iceberg adapter still exposes types from the previous versions. Cargo can resolve both versions, but the integration may fail to compile where those types cross the adapter boundary. This PR includes Iceberg in the root workspace and inherits Ballista’s dependencies, so that failure would affect workspace-wide CI. Keeping the build green would then require updating the adapter alongside Ballista or explicitly excluding it. That coordination burden is my concern; it does not mean the core crates are inherently unpublishable. I suggest establishing an explicit opt-in build gate and independent release policy first: the integration should have its own workspace, lockfile, and CI, and remain on a compatible Ballista release until it is ready to upgrade. A feature flag alone is insufficient. Whether this requires an additional core gate or can use the existing extension points is something we can settle around that requirement. We can focus entirely on Iceberg for this step. P2: Consider other formats laterHudi’s record-location indexes support efficient key-based upserts, while Lance’s scalar indexes can accelerate merge-insert operations. These offer useful capabilities for CDC and frequent-update workloads, and users may want Ballista to query those datasets. We can defer those integrations. The immediate question is how to integrate Iceberg without making its dependency upgrade schedule part of Ballista’s core release obligations.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Following up on the P0/P2 discussion, I have made the proposed boundary concrete in two PRs:
The demo is stacked on the gate branch, so its diff isolates the integration changes. Its path dependencies exercise the unpublished gate; independently released adapters would select compatible released Ballista versions. This separates dependency resolution and release ownership, without claiming compatibility between different DataFusion/Arrow types. |
||
|
|
||
| `register_iceberg` and `with_iceberg` add the Iceberg codecs after the codecs | ||
| already set, with `ComposedLogicalExtensionCodec` and DataFusion's | ||
| `ComposedPhysicalExtensionCodec` (see | ||
| [Combining Codecs](extending-components.md#combining-codecs)). To read tables | ||
| of another format too, such as Delta, set its codecs first: on the session's | ||
| `SessionConfig` before calling `register_iceberg`, and on `SchedulerConfig` and | ||
| `ExecutorProcessConfig` before calling `with_iceberg`. Clients, the scheduler | ||
| and executors must all set the same codecs in the same order. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **Reads only.** `INSERT INTO` an Iceberg table fails with "isn't supported | ||
| on Ballista yet". | ||
| - **No metadata tables.** Tables such as `t$snapshots` can't be sent to the | ||
| scheduler yet. | ||
| - **One task per scan.** Iceberg scans do not split into partitions yet, so one | ||
| executor reads the whole table. Operators after the first shuffle still run | ||
| in parallel. A scan reporting more than one partition fails to encode. | ||
| - **A `UNION` or `UNION ALL` can fail or return wrong rows when the adaptive | ||
| planner splits its stage between tasks.** That happens with | ||
| `ballista.scheduler.max_partitions_per_task` set, or when the executor has | ||
| fewer free task slots than the union has partitions. A union of Iceberg scans | ||
| fails with an error that suggests `SchedulerConfig::partition_sliceable_resolver`, | ||
| which standalone mode can't set; | ||
| [#2852](https://github.com/apache/datafusion-ballista/pull/2852) fixes it. A union | ||
| whose branches each merge their input into one partition, as with an | ||
| `ORDER BY` over the union, a `LIMIT` in a branch or aggregates without | ||
| `GROUP BY`, returns some rows more than once, whatever the table format | ||
| ([#2854](https://github.com/apache/datafusion-ballista/issues/2854)). Setting | ||
| `ballista.planner.adaptive.enabled` to `false` avoids both. | ||
| - **No encrypted tables.** A table with `encryption.key-id` set fails to load | ||
| on the scheduler and executors, which have no key management client. | ||
| - **Storage credentials travel in plain text.** Each table's serialized | ||
| `FileIO` includes them, so the links between client, scheduler and executors | ||
| must be trusted. | ||
| - **A view keeps the version from when it was created.** DataFusion stores a | ||
| view as a planned query, and tables are pinned when a query is planned. The | ||
| same goes for a `DataFrame` or a Flight SQL prepared statement run more than | ||
| once: each run reads the version from when it was planned. To read newer | ||
| commits, recreate the view or plan the query again. | ||
| - **A separately deployed scheduler and executors** must link | ||
| `ballista-iceberg` and be readied with `with_iceberg` (see | ||
| [SQL clients](#sql-clients)). The tests cover standalone mode and an | ||
| in-process scheduler serving Flight SQL. | ||
| - **`ctx.catalog(name)` returns a wrapper.** `register_iceberg` wraps each | ||
| catalog to pin its tables, so the result doesn't downcast to the type you | ||
| registered. Keep the `Arc` you registered to reach it. | ||
| - **A mounted catalog lists its tables once.** `IcebergCatalogProvider` reads | ||
| the table names when it is built, so a table created afterwards isn't | ||
| visible until the catalog is mounted again. | ||
Uh oh!
There was an error while loading. Please reload this page.