Skip to content

Iceberg: DROP TABLE fails on a registered table its catalog can't load #2882

Open

Description

@NoahKusaba

Describe the bug

With ballista-iceberg (#2638), a catalog-backed Iceberg table registered in an ordinary schema with register_table can't be dropped with SQL once its catalog can no longer load it, for example after the table is dropped from the catalog or while the catalog is unreachable.

PinningSchema::table loads an IcebergCatalogTableProvider on every lookup and returns the error if the load fails. DataFusion's DROP TABLE looks the table up first (SessionContext::find_and_deregister), so the error makes it report the table as missing:

  • DROP TABLE t fails with Table 't' doesn't exist.
  • DROP TABLE IF EXISTS t succeeds without dropping it, and t stays registered.

To Reproduce

let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?;
let t = IcebergCatalogTableProvider::try_new(catalog.clone(), namespace, "t").await?;
ctx.register_table("t", Arc::new(t))?;
catalog.drop_table(&TableIdent::from_strs(["ns", "t"])?).await?;

ctx.sql("DROP TABLE t").await;           // Err: Table 't' doesn't exist.
ctx.sql("DROP TABLE IF EXISTS t").await; // Ok, but t is still registered

Expected behavior

DROP TABLE removes the table, as it does in single-process DataFusion, where IcebergCatalogTableProvider only loads its table in scan(). Reading the table should still fail with the load error.

ctx.deregister_table("t") works, since it doesn't look the table up. #2784 documents that as the workaround.

Additional context

#2784 fixes the same problem for views: a view whose tables can't be loaded is returned as a RepinnedView that holds the error, so it can be dropped or replaced and fails only when read. Fix options:

  • A similar wrapper for base tables that reports the original's type and schema and fails on scan and encode. PinnedTable is what the codec and the view re-pinning look for, so they would have to see through the wrapper.
  • Pin when a query runs instead of at lookup (Iceberg: pin tables when a query runs, not when it's looked up #2881), after which lookups do no I/O and this goes away.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions