Skip to content

Iceberg: pin tables when a query runs, not when it's looked up #2881

Open

Description

@NoahKusaba

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

ballista-iceberg (#2638) pins each catalog-backed Iceberg table to one version when a query looks it up, in PinningSchema::table, so the plan can be encoded without I/O. #2784 extends that to views by re-pinning a view's stored plan on each lookup. Choosing the version at lookup still leaves gaps that single-process DataFusion doesn't have, since there IcebergCatalogTableProvider loads its table in scan():

Describe the solution you'd like

Pin when a query runs: in the async QueryPlanner::create_physical_plan, walk the whole plan once, load each distinct catalog-backed table once, and swap in a pinned version of each scan. Lookups would then do no I/O, every scan of a table in a query would read one version, and each run of a reused DataFrame or prepared statement would read the latest version. The re-pinning in PinningSchema::table and RepinnedView would go away.

This needs a hook in Ballista's core first. SessionStateExt::upgrade_for_ballista (ballista/core/src/extension.rs) only lets ballista_query_planner() replace BallistaQueryPlanner, so an integration can't wrap it. Either a composable query planner, or a hook that runs on the logical plan before DistributedQueryExec is built, would do. The scheduler plans Flight SQL queries itself, so it needs the same hook.

Describe alternatives you've considered

Additional context

Follow-up to #2638 and #2784.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions