Repository navigation
[lake] Let the lake tiering factory declare record batch write support - #5223
mikepapadim wants to merge 2 commits into
Conversation
The tiering job wrote ARROW log tables batch by batch only when the table's lake format was Paimon, by comparing DataLakeFormat.PAIMON in TieringSplitReader. A lake writer of another format that implements SupportsRecordBatchWrite was never given batches. LakeTieringFactory#supportsRecordBatchWrite(TableInfo), false by default, now declares it, the same way supportsPartitionMarkDone() does for mark-done. PaimonLakeTieringFactory returns true, so Paimon tiering is unchanged; other lakes keep the row path until they opt in. fluss-flink-common gets unshaded Arrow as a test dependency, so the record batch path can be tested there.
|
|
||
| private void tierLogTableAndVerifyRows( | ||
| TablePath tablePath, TestingLakeTieringFactory lakeTieringFactory) throws Exception { | ||
| long tableId = createTable(tablePath, DEFAULT_LOG_TABLE_DESCRIPTOR); |
There was a problem hiding this comment.
Could we make the positive test use a non-Paimon lake format? DEFAULT_LOG_TABLE_DESCRIPTOR is configured for Paimon, so the previous hard-coded format check would also take the batch path here. The negative test covers the fallback, but we still need a test showing that a non-Paimon factory can opt in to batch writes.
There was a problem hiding this comment.
Thanks, good point. The test cluster only accepts tables whose lake format matches the cluster's (Paimon here), so I couldn't create an Iceberg table end to end. Instead, 3036ca5 extracts the decision into TieringSplitReader#useRecordBatchPath(TableInfo, LakeTieringFactory, boolean) and adds testNonPaimonLakeFormatCanOptInToRecordBatchWrite. It uses an Iceberg TableInfo and a factory that enables batch writes for Iceberg only, and checks that the Iceberg log table takes the batch path while a Paimon table does not, and that the Arrow and primary-key conditions still apply. With the old hard-coded Paimon check in place, this test fails.
Extract the record batch path decision of TieringSplitReader into a static method, and check it on an Iceberg TableInfo with a factory that declares batch support for Iceberg only. The test cluster creates only Paimon tables, so the existing positive test could not tell the capability check from a hard-coded Paimon check; this one fails with the latter.
Jackeyzhe
left a comment
There was a problem hiding this comment.
Thanks for addressed it, LGTM
Generated-by: Claude Code (Claude Opus 5.5) following the guidelines
Purpose
Linked issue: close #5222
The tiering job sends an append-only ARROW log table to the lake batch by batch only when the table's lake format is Paimon.
TieringSplitReader#useRecordBatchPathcomparesDataLakeFormat.PAIMONdirectly. Because of that, a lake writer of another format that implementsSupportsRecordBatchWritenever receives batches. One example is the Iceberg writer in #4048.This PR has the lake tiering factory declare the capability instead, the same way
LakeTieringFactory#supportsPartitionMarkDone()already does for partition mark-done.Brief change log
LakeTieringFactory(@PublicEvolving): adddefault boolean supportsRecordBatchWrite(TableInfo tableInfo), which returnsfalse. When a factory returnstruefor a table, every lake writer it creates for that table must implementSupportsRecordBatchWrite.PaimonLakeTieringFactory: returnstrue, so Paimon tiering behaves as before.TieringSplitReader#useRecordBatchPath: asks the factory instead of comparing the lake format. The other conditions stay the same: unshaded Arrow is available, the table is append-only, and the log format is ARROW.fluss-flink-commonpom: addsarrow-vectorandarrow-memory-netty(${arrow.version}, the versionsfluss-commonandfluss-clientalready declare) with test scope only, so the record batch path can be tested in this module. Nothing that ships changes.Iceberg, Lance and Hudi keep the row path. Each can opt in by overriding the method once its writer supports batches. For Iceberg that would be after #4048.
Tests
New cases in
TieringSplitReaderTest, both using a testing factory whose writer accepts rows and record batches:testArrowLogTableUsesRecordBatchWriteWhenFactorySupportsIt: the factory returnstrue, and every bucket is written throughwrite(RecordBatch)with no row writes. This test fails on main, because the writer receives rows there.testArrowLogTableWritesRowsWhenFactoryDoesNotSupportRecordBatchWrite: the factory keeps the default, and the table is written row by row.Also run:
org.apache.fluss.flink.tiering.**tests influss-flink-common: 99 tests, 0 failuresorg.apache.fluss.lake.paimon.tiering.**tests andPaimonTieringITCaseinfluss-lake-paimon(Paimon's record batch path): 72 tests, 0 failuresAPI and Format
The change adds one default method to
LakeTieringFactory(@PublicEvolving). Existing implementations compile and behave unchanged. There is no storage format change.Documentation
None. The behaviour is unchanged for all existing lake formats.