Skip to content

DataFusion: paimon_incremental_query table function #506

Description

@Pandas886

Describe the feature

Expose batch incremental reads in DataFusion SQL via a table function, roughly:

SELECT * FROM paimon_incremental_query(
  'table_name',
  start_snapshot_exclusive,
  end_snapshot_inclusive
  [, 'auto' | 'delta' | 'changelog' | 'diff']
);

Also support the $audit_log style suffix so row kinds show up as a first column when needed.

Motivation

The Rust core incremental API is useful, but most people will want to poke at this from SQL. DataFusion should only parse args and adapt the execution plan — snapshot planning stays in paimon core.

Depends on

Core incremental Delta / Changelog / Diff support landing first (or the PR can sit on top of that stack).

Out of scope

  • Reimplementing snapshot planning inside the DataFusion crate
  • Lookup/compact changelog generation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions