You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[lake] Let the lake tiering factory declare record batch write support #5222
I searched in the issues and found nothing similar.
Motivation
The tiering job writes an append-only ARROW log table to the lake batch by batch (SupportsRecordBatchWrite) only when the table's lake format is Paimon. TieringSplitReader#useRecordBatchPath compares the format directly:
So a lake writer of another format that implements SupportsRecordBatchWrite is never given batches. For example, #4047 / #4048 add SupportsRecordBatchWrite to IcebergLakeWriter, but Iceberg tables would still be tiered row by row after that PR.
Solution
Let the tiering factory declare it, the same way LakeTieringFactory#supportsPartitionMarkDone() does for mark-done:
PaimonLakeTieringFactory returns true, and TieringSplitReader calls the factory instead of comparing the format. Paimon keeps its batch path, and the other lakes stay on the row path until they opt in. Iceberg could then return true once #4048 lands.
Anything else?
This is about 20 lines of main code. I have it on a branch with tests and will open a PR.
Activity