Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ballista/core/src/serde/composed_codec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,14 @@ impl ComposedLogicalExtensionCodec {
}

/// Combines the codecs' errors, so the owning codec's error isn't hidden.
///
/// TODO: when moving this to `datafusion-proto`, return the error of a codec
/// that recognised the node (one other than `NotImplemented`) unchanged, and
/// join the errors only when every codec declined. Joining them into one string
/// loses the owning codec's error chain, so `DataFusionError::find_root` can't
/// reach it and callers can only match its text. For example, reading an
/// Iceberg table that can't be loaded reports `no composed codec could encode
/// it: codec 0: … not implemented …; codec 1: <the load error>`.
fn collect(mut errors: Vec<DataFusionError>) -> DataFusionError {
match errors.len() {
0 => DataFusionError::NotImplemented("Empty list of composed codecs".to_owned()),
Expand Down
17 changes: 12 additions & 5 deletions docs/source/user-guide/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,13 @@ Metadata files never change, so every node rebuilds exactly that version.
the client plans the query. DataFusion looks up each table a query names
once, so every scan of it, such as each side of a self-join, reads that
version, and each new query reads the latest one.
- **A view**, from `CREATE VIEW` or a `DataFrame`'s `into_view`, stores the
plan it was created with, so each query that names it pins its
catalog-backed tables again, and it reads their latest versions.
- **A table its catalog can't load**, because it was dropped or the catalog
can't be reached, fails a query only when the query reads it, as in
DataFusion. The table, and any view over it, can still be described, dropped
or replaced, and a view reads the table again once it loads.
- **A static table** is sent as the version it holds.
- **A scan** is sent as the version the scheduler planned it against.

Expand Down Expand Up @@ -162,11 +169,11 @@ and executors must all set the same codecs in the same order.
- **Storage credentials travel in plain text.** Each table's serialized
`FileIO` includes them, so the links between client, scheduler and executors
must be trusted.
- **A view keeps the version from when it was created.** DataFusion stores a
view as a planned query, and tables are pinned when a query is planned. The
same goes for a `DataFrame` or a Flight SQL prepared statement run more than
once: each run reads the version from when it was planned. To read newer
commits, recreate the view or plan the query again.
- **A reused `DataFrame` or prepared statement keeps its version.** Tables are
pinned when a query is planned, so a `DataFrame` or a Flight SQL prepared
statement run more than once reads the version from when it was planned. To
read newer commits, plan the query again, or register the `DataFrame` as a
view with `into_view` and query that.
- **A separately deployed scheduler and executors** must link
`ballista-iceberg` and be readied with `with_iceberg` (see
[SQL clients](#sql-clients)). The tests cover standalone mode and an
Expand Down
217 changes: 193 additions & 24 deletions integrations/iceberg/src/catalog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,45 @@

//! Pins catalog-backed tables to one version while a query is planned.

use std::borrow::Cow;
use std::collections::HashMap;
use std::sync::Arc;

use async_trait::async_trait;
use datafusion::arrow::datatypes::SchemaRef;
use datafusion::catalog::view::ViewTable;
use datafusion::catalog::{
CatalogProvider, CatalogProviderList, SchemaProvider, Session, TableProvider,
};
use datafusion::common::{Result, not_impl_err};
use datafusion::datasource::TableType;
use datafusion::common::tree_node::{Transformed, TreeNodeRecursion};
use datafusion::common::{DataFusionError, Result, not_impl_err};
use datafusion::datasource::source::DataSourceExec;
use datafusion::datasource::{DefaultTableSource, TableType, provider_as_source};
use datafusion::logical_expr::dml::InsertOp;
use datafusion::logical_expr::{Expr, TableProviderFilterPushDown};
use datafusion::logical_expr::{
Expr, LogicalPlan, TableProviderFilterPushDown, TableScan,
};
use datafusion::physical_plan::ExecutionPlan;
use futures::future::join_all;
use iceberg::TableIdent;
use iceberg::table::Table;
use iceberg_datafusion::read::IcebergDataSource;
use iceberg_datafusion::{IcebergCatalogTableProvider, to_datafusion_error};

/// A session's catalogs, serving each catalog-backed Iceberg table as a
/// [`PinnedTable`].
/// [`PinnedTable`], and each view with its Iceberg tables pinned again, as a
/// [`RepinnedView`].
///
/// DataFusion looks up each table a query names once, while planning it, so
/// every scan of a table in the query reads the same version, and encoding the
/// plan needs no I/O.
/// plan needs no I/O. A view stores the plan it was created with, so its tables
/// are pinned again each time a query names it, and it reads the latest
/// versions, as it would in DataFusion.
///
/// A lookup never fails because a table can't be loaded: the [`PinnedTable`]
/// holds the error instead, and a query fails with it only when it reads the
/// table, so the table, or a view over it, can still be described, dropped or
/// replaced.
#[derive(Debug)]
pub(crate) struct PinningCatalogList(pub(crate) Arc<dyn CatalogProviderList>);

Expand Down Expand Up @@ -109,18 +125,20 @@ impl SchemaProvider for PinningSchema {
let Some(table) = self.0.table(name).await? else {
return Ok(None);
};
let Some(provider) = table.downcast_ref::<IcebergCatalogTableProvider>() else {
return Ok(Some(table));
if let Some(provider) = table.downcast_ref::<IcebergCatalogTableProvider>() {
let table = load(provider).await;
return Ok(Some(Arc::new(PinnedTable::pin(provider, table))));
}
let repinned = match table.get_logical_plan() {
Some(plan) => match repin(&plan).await? {
Some(plan) => ViewTable::new(plan, None),
None => return Ok(Some(table)),
},
None => return Ok(Some(table)),
};
let pinned = provider
.catalog()
.load_table(provider.table_ident())
.await
.map_err(to_datafusion_error)?;
Ok(Some(Arc::new(PinnedTable {
table: pinned,
snapshot_id: None,
schema: provider.schema(),
Ok(Some(Arc::new(RepinnedView {
original: table,
repinned,
})))
}

Expand All @@ -145,8 +163,132 @@ impl SchemaProvider for PinningSchema {
}
}

/// A catalog-backed table: its catalog, by address, and its identifier.
type TableKey = (usize, TableIdent);

fn key(provider: &IcebergCatalogTableProvider) -> TableKey {
let catalog = Arc::as_ptr(provider.catalog()).cast::<()>() as usize;
(catalog, provider.table_ident().clone())
}

/// A loaded table, or the error loading it returned, shared by every scan of it.
type Loaded = std::result::Result<Table, Arc<DataFusionError>>;

/// The version of `provider`'s table its catalog serves now, or why it couldn't
/// be loaded.
async fn load(provider: &IcebergCatalogTableProvider) -> Loaded {
provider
.catalog()
.load_table(provider.table_ident())
.await
.map_err(|error| Arc::new(to_datafusion_error(error)))
}

/// The catalog-backed table a scan's pinned table was pinned from, if any.
fn origin(scan: &TableScan) -> Option<&IcebergCatalogTableProvider> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One case I wanted to check with you. Say t's catalog is briefly unreachable and someone runs CREATE VIEW w AS SELECT * FROM v meanwhile. Then v resolves to a RepinnedView holding the error, and since it has no plan, w stores a plain TableScan of that RepinnedView. Later lookups of w find no PinnedTable here, so repin returns None and w keeps failing with the old error, even once t loads fine again and v itself works. Would it make sense for repin to also recognise a scan over a RepinnedView and re-pin its original's plan? Or is this rare enough to just note in the docs?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, reproduced. Rather than special-casing RepinnedView scans in repin, a table that can't be loaded is now a PinnedTable holding the error, which still knows its catalog table. So a view is always inlined, w stores v's plan, and its next lookup reloads t. This also fixes #2882, since lookups no longer fail. Covered by should_read_a_view_created_while_its_table_was_gone.

let source = scan.source.downcast_ref::<DefaultTableSource>()?;
let pinned = source.table_provider.downcast_ref::<PinnedTable>()?;
pinned.origin.as_ref()
}

/// Returns a view's `plan` with each catalog-backed Iceberg table it reads
/// pinned to the version its catalog serves now, loading each table once, or
/// `None` if it reads none. A table that can't be loaded is pinned to the error,
/// so the plan still knows which table to load next time. DataFusion inlines a
/// view when it plans a scan of it, so the plan of a view over other views
/// already holds their tables, and needs no recursion.
async fn repin(plan: &LogicalPlan) -> Result<Option<LogicalPlan>> {
let mut providers = HashMap::new();
plan.apply_with_subqueries(|node| {
if let LogicalPlan::TableScan(scan) = node
&& let Some(origin) = origin(scan)
{
providers
.entry(key(origin))
.or_insert_with(|| origin.clone());
}
Ok(TreeNodeRecursion::Continue)
})?;
if providers.is_empty() {
return Ok(None);
}
let loads = providers
.into_iter()
.map(|(key, provider)| async move { (key, load(&provider).await) });
let tables: HashMap<_, _> = join_all(loads).await.into_iter().collect();
let plan = plan
.clone()
.transform_up_with_subqueries(|node| {
let LogicalPlan::TableScan(mut scan) = node else {
return Ok(Transformed::no(node));
};
let Some(origin) = origin(&scan) else {
return Ok(Transformed::no(LogicalPlan::TableScan(scan)));
};
let table = tables[&key(origin)].clone();
scan.source = provider_as_source(Arc::new(PinnedTable::pin(origin, table)));
Ok(Transformed::yes(LogicalPlan::TableScan(scan)))
})?
.data;
Ok(Some(plan))
}

/// A view with each catalog-backed Iceberg table it reads pinned again.
///
/// A view is any table DataFusion inlines: one whose provider has a logical
/// plan, such as a [`ViewTable`] from `CREATE VIEW` or a `DataFrame`'s
/// `into_view`. It keeps the original view's type, schema and definition, and
/// replaces only its plan.
#[derive(Debug)]
struct RepinnedView {
original: Arc<dyn TableProvider>,
/// The re-pinned plan, held as a [`ViewTable`] to plan a scan of it.
repinned: ViewTable,
}

#[async_trait]
impl TableProvider for RepinnedView {
fn schema(&self) -> SchemaRef {
self.original.schema()
}

fn table_type(&self) -> TableType {
self.original.table_type()
}

fn get_table_definition(&self) -> Option<&str> {
self.original.get_table_definition()
}

fn get_logical_plan(&self) -> Option<Cow<'_, LogicalPlan>> {
self.repinned.get_logical_plan()
}

fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> Result<Vec<TableProviderFilterPushDown>> {
self.repinned.supports_filters_pushdown(filters)
}

async fn scan(
&self,
state: &dyn Session,
projection: Option<&Vec<usize>>,
filters: &[Expr],
limit: Option<usize>,
) -> Result<Arc<dyn ExecutionPlan>> {
self.repinned.scan(state, projection, filters, limit).await
}
}

/// One version of an Iceberg table, read with the schema the client planned
/// against.
/// against, or the error loading it from its catalog returned.
///
/// A table that couldn't be loaded still knows the catalog-backed table it was
/// pinned from, so a view over it reads it once its catalog can load it again.
/// Pushing filters into it, scanning it or encoding it returns the error, as
/// reading a missing table does in DataFusion.
///
/// Not an [`IcebergStaticTableProvider`]: that derives its schema from the
/// table, which may have evolved since planning, and is built asynchronously,
Expand All @@ -155,9 +297,33 @@ impl SchemaProvider for PinningSchema {
/// [`IcebergStaticTableProvider`]: iceberg_datafusion::IcebergStaticTableProvider
#[derive(Debug)]
pub(crate) struct PinnedTable {
pub(crate) table: Table,
pub(crate) table: Loaded,
pub(crate) snapshot_id: Option<i64>,
pub(crate) schema: SchemaRef,
/// The catalog-backed table it was pinned from, to pin it again. Not sent
/// between nodes.
pub(crate) origin: Option<IcebergCatalogTableProvider>,
}

impl PinnedTable {
/// `table`, a version of `provider` loaded from its catalog or the error
/// loading it returned, read with the provider's schema, which it caches
/// when it is created.
fn pin(provider: &IcebergCatalogTableProvider, table: Loaded) -> Self {
Self {
table,
snapshot_id: None,
schema: provider.schema(),
origin: Some(provider.clone()),
}
}

/// The pinned version, or the error loading it returned.
pub(crate) fn table(&self) -> Result<&Table> {
self.table
.as_ref()
.map_err(|error| DataFusionError::Shared(Arc::clone(error)))
}
}

#[async_trait]
Expand All @@ -177,19 +343,22 @@ impl TableProvider for PinnedTable {
filters: &[Expr],
limit: Option<usize>,
) -> Result<Arc<dyn ExecutionPlan>> {
let source = IcebergDataSource::builder(self.table.clone(), self.schema.clone())
.with_snapshot_id(self.snapshot_id)
.with_projection(projection)
.with_filters(filters)
.with_fetch(limit)
.build()?;
let source =
IcebergDataSource::builder(self.table()?.clone(), self.schema.clone())
.with_snapshot_id(self.snapshot_id)
.with_projection(projection)
.with_filters(filters)
.with_fetch(limit)
.build()?;
Ok(DataSourceExec::from_data_source(source))
}

fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> Result<Vec<TableProviderFilterPushDown>> {
// Fails a filtered query while it's planned, before it's sent.
self.table()?;
Ok(vec![TableProviderFilterPushDown::Inexact; filters.len()])
}

Expand Down
4 changes: 2 additions & 2 deletions integrations/iceberg/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@
//! - Reads only.
//! - Each scan runs as a single task.
//! - No metadata tables such as `t$snapshots`.
//! - A view, or a `DataFrame` or prepared statement reused across runs, reads
//! the version from when it was planned.
//! - A `DataFrame` or prepared statement reused across runs reads the version
//! from when it was planned.
//! - No encrypted tables.
//! - Storage credentials travel in plain text.
//! - A `UNION` or `UNION ALL` can fail, or with an `ORDER BY` return rows more
Expand Down
5 changes: 3 additions & 2 deletions integrations/iceberg/src/logical_codec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,10 @@ impl LogicalExtensionCodec for IcebergLogicalCodec {
) -> Result<Arc<dyn TableProvider>> {
let ProviderWire { table, snapshot_id } = wire::decode(buf)?;
Ok(Arc::new(PinnedTable {
table: table.load()?,
table: Ok(table.load()?),
snapshot_id,
schema,
origin: None,
}))
}

Expand All @@ -108,7 +109,7 @@ impl LogicalExtensionCodec for IcebergLogicalCodec {
}
} else if let Some(pinned) = node.downcast_ref::<PinnedTable>() {
ProviderWire {
table: TableWire::new(&pinned.table)?,
table: TableWire::new(pinned.table()?)?,
snapshot_id: pinned.snapshot_id,
}
} else if node.is::<IcebergCatalogTableProvider>() {
Expand Down
Loading
Loading