You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Iceberg: DROP TABLE fails on a registered table its catalog can't load #2882
With ballista-iceberg (#2638), a catalog-backed Iceberg table registered in an ordinary schema with register_table can't be dropped with SQL once its catalog can no longer load it, for example after the table is dropped from the catalog or while the catalog is unreachable.
PinningSchema::table loads an IcebergCatalogTableProvider on every lookup and returns the error if the load fails. DataFusion's DROP TABLE looks the table up first (SessionContext::find_and_deregister), so the error makes it report the table as missing:
DROP TABLE t fails with Table 't' doesn't exist.
DROP TABLE IF EXISTS t succeeds without dropping it, and t stays registered.
To Reproduce
let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?;let t = IcebergCatalogTableProvider::try_new(catalog.clone(), namespace,"t").await?;
ctx.register_table("t",Arc::new(t))?;
catalog.drop_table(&TableIdent::from_strs(["ns","t"])?).await?;
ctx.sql("DROP TABLE t").await;// Err: Table 't' doesn't exist.
ctx.sql("DROP TABLE IF EXISTS t").await;// Ok, but t is still registered
Expected behavior
DROP TABLE removes the table, as it does in single-process DataFusion, where IcebergCatalogTableProvider only loads its table in scan(). Reading the table should still fail with the load error.
ctx.deregister_table("t") works, since it doesn't look the table up. #2784 documents that as the workaround.
Additional context
#2784 fixes the same problem for views: a view whose tables can't be loaded is returned as a RepinnedView that holds the error, so it can be dropped or replaced and fails only when read. Fix options:
A similar wrapper for base tables that reports the original's type and schema and fails on scan and encode. PinnedTable is what the codec and the view re-pinning look for, so they would have to see through the wrapper.
Activity