You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Iceberg: pin tables when a query runs, not when it's looked up #2881
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
ballista-iceberg (#2638) pins each catalog-backed Iceberg table to one version when a query looks it up, in PinningSchema::table, so the plan can be encoded without I/O. #2784 extends that to views by re-pinning a view's stored plan on each lookup. Choosing the version at lookup still leaves gaps that single-process DataFusion doesn't have, since there IcebergCatalogTableProvider loads its table in scan():
A reused DataFrame, or a Flight SQL prepared statement run more than once, reads the version from when it was planned. Neither looks its tables up again.
A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups gives them different versions.
Pin when a query runs: in the async QueryPlanner::create_physical_plan, walk the whole plan once, load each distinct catalog-backed table once, and swap in a pinned version of each scan. Lookups would then do no I/O, every scan of a table in a query would read one version, and each run of a reused DataFrame or prepared statement would read the latest version. The re-pinning in PinningSchema::table and RepinnedView would go away.
This needs a hook in Ballista's core first. SessionStateExt::upgrade_for_ballista (ballista/core/src/extension.rs) only lets ballista_query_planner() replace BallistaQueryPlanner, so an integration can't wrap it. Either a composable query planner, or a hook that runs on the logical plan before DistributedQueryExec is built, would do. The scheduler plans Flight SQL queries itself, so it needs the same hook.
Activity