Repository navigation
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
On one machine, the lake writer accounts for most of a tiering round for an append-only ARROW log table tiered to Iceberg. I profiled TieringSplitReader against an in-process cluster: one bucket, a TPC-H lineitem-like table with 16 columns, and ZSTD Parquet output. IcebergLakeWriter.write + complete took 52% of the round at 8 MB of Arrow data and 81% at 512 MB (7.2 s of 8.9 s, about 11 CPU-seconds). The same data tiered to Paimon takes 3.7 s. The method and full numbers are in the linked discussion.
There are two separate costs:
- Row conversion (≈1.7 s of the 8.9 s). Iceberg tiering always reads rows. [lake] Let the lake tiering factory declare record batch write support #5222 / [lake] Let the lake tiering factory declare record batch write support #5223 let a lake declare
supportsRecordBatchWrite, and [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4047 / [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4048 implement it for Iceberg. - Encoding and compression (≈7.2 s). This is parquet-java's work. On Paimon, the Arrow batch path is not faster at encoding than the row path (3.16 s vs 3.08 s for 512 MB), so a columnar path alone does not reduce it.
Direct Arrow-to-Parquet conversion has been asked for in #437 / #2963, and #2963 is blocked on arrow-java#904. Fluss currently offers no way to try a different Parquet writer for Iceberg tiering without changing fluss-lake-iceberg-common.
Solution
An optional, device-neutral extension point in fluss-lake-iceberg-common for writing an append-only table's Arrow batches to Iceberg data files. Nothing changes unless an implementation is configured:
@PublicEvolving
public interface IcebergArrowFileWriterFactory {
String identifier();
boolean supports(Table table, TableInfo tableInfo);
IcebergArrowFileWriter create(Table table, TableInfo tableInfo, OutputFileFactory files,
@Nullable PartitionKey partition, long targetFileSizeBytes);
}
@PublicEvolving
public interface IcebergArrowFileWriter extends Closeable {
void write(ArrowBatchData batch) throws IOException;
List<DataFile> complete() throws IOException;
}- Implementations are discovered with
ServiceLoaderfrom the Iceberg plugin's class loader and selected per catalog with an Iceberg lake option (for exampletiering.arrow-writer=<identifier>). There is no default implementation, and the option is unset by default. IcebergLakeTieringFactory#supportsRecordBatchWrite(TableInfo)([lake] Let the lake tiering factory declare record batch write support #5223) returns true only when a writer is configured andsupportsthe table.IcebergLakeWriterpasses it the batches and adds itsDataFiles to the existing write result, so the committer is unchanged.- Both sides of the Arrow boundary have to be explicit, given the classpath issue found in [client] Expose Arrow IPC batches with optional Java adapter #4006. The batches are the unshaded
ArrowBatchDatathat the record batch path already hands to Paimon. Ownership stays with the caller: the writer copies what it needs duringwrite, and the batch is closed afterwritereturns, as it is today. - Table-level fallback: if
supportsreturns false (an unsupported type, legacy schema columns), the table is tiered row by row as today.
The goal is to let alternative Parquet writers be tried outside the Fluss tree, for example an Arrow-native CPU writer once arrow-java#904 lands, or a GPU writer. As an example of what such a writer can do: a cuDF-based GPU writer encoded the same 512 MB as ZSTD Parquet in 0.22 s on an RTX 4090. I'm not proposing to add any GPU or native dependency to Fluss. That code would stay outside the tree, behind this interface.
It would build on #5223 and is about 240 lines (interfaces, ServiceLoader lookup, the option, and wiring in IcebergLakeTieringFactory / IcebergLakeWriter) plus tests with a Testing* writer.
Anything else?
Open questions for reviewers:
- Is
fluss-lake-iceberg-commonthe right home, or should this be lake-neutral influss-common(SupportsRecordBatchWriteplus a file-writer SPI that Paimon could also use)? ServiceLoaderplus a lake option, or a class-name option such astiering.arrow-writer.class?- Does this need a FIP?
Willingness to contribute
- I'm willing to submit a PR!
Activity
mikepapadim commentedon Nov 13, 2026
Method, full numbers and limits for the motivation above: #5227