Skip to content

[lake] Let the lake tiering factory declare record batch write support - #5223

Open
mikepapadim wants to merge 2 commits into
apache:mainfrom
mikepapadim:lake-tiering-batch-capability
Open

mikepapadim wants to merge 2 commits into
apache:mainfrom
mikepapadim:lake-tiering-batch-capability

Conversation

@mikepapadim

Copy link
Copy Markdown

Generated-by: Claude Code (Claude Opus 5.5) following the guidelines

  • No generative AI tools used
  • Yes: Claude Code (Claude Opus 5.5)

Purpose

Linked issue: close #5222

The tiering job sends an append-only ARROW log table to the lake batch by batch only when the table's lake format is Paimon. TieringSplitReader#useRecordBatchPath compares DataLakeFormat.PAIMON directly. Because of that, a lake writer of another format that implements SupportsRecordBatchWrite never receives batches. One example is the Iceberg writer in #4048.

This PR has the lake tiering factory declare the capability instead, the same way LakeTieringFactory#supportsPartitionMarkDone() already does for partition mark-done.

Brief change log

  • LakeTieringFactory (@PublicEvolving): add default boolean supportsRecordBatchWrite(TableInfo tableInfo), which returns false. When a factory returns true for a table, every lake writer it creates for that table must implement SupportsRecordBatchWrite.
  • PaimonLakeTieringFactory: returns true, so Paimon tiering behaves as before.
  • TieringSplitReader#useRecordBatchPath: asks the factory instead of comparing the lake format. The other conditions stay the same: unshaded Arrow is available, the table is append-only, and the log format is ARROW.
  • fluss-flink-common pom: adds arrow-vector and arrow-memory-netty (${arrow.version}, the versions fluss-common and fluss-client already declare) with test scope only, so the record batch path can be tested in this module. Nothing that ships changes.

Iceberg, Lance and Hudi keep the row path. Each can opt in by overriding the method once its writer supports batches. For Iceberg that would be after #4048.

Tests

New cases in TieringSplitReaderTest, both using a testing factory whose writer accepts rows and record batches:

  • testArrowLogTableUsesRecordBatchWriteWhenFactorySupportsIt: the factory returns true, and every bucket is written through write(RecordBatch) with no row writes. This test fails on main, because the writer receives rows there.
  • testArrowLogTableWritesRowsWhenFactoryDoesNotSupportRecordBatchWrite: the factory keeps the default, and the table is written row by row.

Also run:

  • org.apache.fluss.flink.tiering.** tests in fluss-flink-common: 99 tests, 0 failures
  • org.apache.fluss.lake.paimon.tiering.** tests and PaimonTieringITCase in fluss-lake-paimon (Paimon's record batch path): 72 tests, 0 failures

API and Format

The change adds one default method to LakeTieringFactory (@PublicEvolving). Existing implementations compile and behave unchanged. There is no storage format change.

Documentation

None. The behaviour is unchanged for all existing lake formats.

The tiering job wrote ARROW log tables batch by batch only when the
table's lake format was Paimon, by comparing DataLakeFormat.PAIMON in
TieringSplitReader. A lake writer of another format that implements
SupportsRecordBatchWrite was never given batches.

LakeTieringFactory#supportsRecordBatchWrite(TableInfo), false by
default, now declares it, the same way supportsPartitionMarkDone()
does for mark-done. PaimonLakeTieringFactory returns true, so Paimon
tiering is unchanged; other lakes keep the row path until they opt in.

fluss-flink-common gets unshaded Arrow as a test dependency, so the
record batch path can be tested there.

private void tierLogTableAndVerifyRows(
TablePath tablePath, TestingLakeTieringFactory lakeTieringFactory) throws Exception {
long tableId = createTable(tablePath, DEFAULT_LOG_TABLE_DESCRIPTOR);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make the positive test use a non-Paimon lake format? DEFAULT_LOG_TABLE_DESCRIPTOR is configured for Paimon, so the previous hard-coded format check would also take the batch path here. The negative test covers the fallback, but we still need a test showing that a non-Paimon factory can opt in to batch writes.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good point. The test cluster only accepts tables whose lake format matches the cluster's (Paimon here), so I couldn't create an Iceberg table end to end. Instead, 3036ca5 extracts the decision into TieringSplitReader#useRecordBatchPath(TableInfo, LakeTieringFactory, boolean) and adds testNonPaimonLakeFormatCanOptInToRecordBatchWrite. It uses an Iceberg TableInfo and a factory that enables batch writes for Iceberg only, and checks that the Iceberg log table takes the batch path while a Paimon table does not, and that the Arrow and primary-key conditions still apply. With the old hard-coded Paimon check in place, this test fails.

Extract the record batch path decision of TieringSplitReader into a
static method, and check it on an Iceberg TableInfo with a factory that
declares batch support for Iceberg only. The test cluster creates only
Paimon tables, so the existing positive test could not tell the
capability check from a hard-coded Paimon check; this one fails with
the latter.

@Jackeyzhe Jackeyzhe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressed it, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[lake] Let the lake tiering factory declare record batch write support

2 participants