Skip to content

[lake] Let the lake tiering factory declare record batch write support #5222

Open

Description

@mikepapadim

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

The tiering job writes an append-only ARROW log table to the lake batch by batch (SupportsRecordBatchWrite) only when the table's lake format is Paimon. TieringSplitReader#useRecordBatchPath compares the format directly:

&& tableInfo.getTableConfig().getDataLakeFormat().orElse(null)
        == DataLakeFormat.PAIMON;

So a lake writer of another format that implements SupportsRecordBatchWrite is never given batches. For example, #4047 / #4048 add SupportsRecordBatchWrite to IcebergLakeWriter, but Iceberg tables would still be tiered row by row after that PR.

Solution

Let the tiering factory declare it, the same way LakeTieringFactory#supportsPartitionMarkDone() does for mark-done:

// LakeTieringFactory (@PublicEvolving), default false
default boolean supportsRecordBatchWrite(TableInfo tableInfo) {
    return false;
}

PaimonLakeTieringFactory returns true, and TieringSplitReader calls the factory instead of comparing the format. Paimon keeps its batch path, and the other lakes stay on the row path until they opt in. Iceberg could then return true once #4048 lands.

Anything else?

This is about 20 lines of main code. I have it on a branch with tests and will open a PR.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions