Skip to content

[lake/iceberg] Pluggable Arrow file writer for tiering append-only tables #5226

Open

Description

@mikepapadim

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

On one machine, the lake writer accounts for most of a tiering round for an append-only ARROW log table tiered to Iceberg. I profiled TieringSplitReader against an in-process cluster: one bucket, a TPC-H lineitem-like table with 16 columns, and ZSTD Parquet output. IcebergLakeWriter.write + complete took 52% of the round at 8 MB of Arrow data and 81% at 512 MB (7.2 s of 8.9 s, about 11 CPU-seconds). The same data tiered to Paimon takes 3.7 s. The method and full numbers are in the linked discussion.

There are two separate costs:

  1. Row conversion (≈1.7 s of the 8.9 s). Iceberg tiering always reads rows. [lake] Let the lake tiering factory declare record batch write support #5222 / [lake] Let the lake tiering factory declare record batch write support #5223 let a lake declare supportsRecordBatchWrite, and [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4047 / [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4048 implement it for Iceberg.
  2. Encoding and compression (≈7.2 s). This is parquet-java's work. On Paimon, the Arrow batch path is not faster at encoding than the row path (3.16 s vs 3.08 s for 512 MB), so a columnar path alone does not reduce it.

Direct Arrow-to-Parquet conversion has been asked for in #437 / #2963, and #2963 is blocked on arrow-java#904. Fluss currently offers no way to try a different Parquet writer for Iceberg tiering without changing fluss-lake-iceberg-common.

Solution

An optional, device-neutral extension point in fluss-lake-iceberg-common for writing an append-only table's Arrow batches to Iceberg data files. Nothing changes unless an implementation is configured:

@PublicEvolving
public interface IcebergArrowFileWriterFactory {
    String identifier();
    boolean supports(Table table, TableInfo tableInfo);
    IcebergArrowFileWriter create(Table table, TableInfo tableInfo, OutputFileFactory files,
                                  @Nullable PartitionKey partition, long targetFileSizeBytes);
}

@PublicEvolving
public interface IcebergArrowFileWriter extends Closeable {
    void write(ArrowBatchData batch) throws IOException;
    List<DataFile> complete() throws IOException;
}
  • Implementations are discovered with ServiceLoader from the Iceberg plugin's class loader and selected per catalog with an Iceberg lake option (for example tiering.arrow-writer=<identifier>). There is no default implementation, and the option is unset by default.
  • IcebergLakeTieringFactory#supportsRecordBatchWrite(TableInfo) ([lake] Let the lake tiering factory declare record batch write support #5223) returns true only when a writer is configured and supports the table. IcebergLakeWriter passes it the batches and adds its DataFiles to the existing write result, so the committer is unchanged.
  • Both sides of the Arrow boundary have to be explicit, given the classpath issue found in [client] Expose Arrow IPC batches with optional Java adapter #4006. The batches are the unshaded ArrowBatchData that the record batch path already hands to Paimon. Ownership stays with the caller: the writer copies what it needs during write, and the batch is closed after write returns, as it is today.
  • Table-level fallback: if supports returns false (an unsupported type, legacy schema columns), the table is tiered row by row as today.

The goal is to let alternative Parquet writers be tried outside the Fluss tree, for example an Arrow-native CPU writer once arrow-java#904 lands, or a GPU writer. As an example of what such a writer can do: a cuDF-based GPU writer encoded the same 512 MB as ZSTD Parquet in 0.22 s on an RTX 4090. I'm not proposing to add any GPU or native dependency to Fluss. That code would stay outside the tree, behind this interface.

It would build on #5223 and is about 240 lines (interfaces, ServiceLoader lookup, the option, and wiring in IcebergLakeTieringFactory / IcebergLakeWriter) plus tests with a Testing* writer.

Anything else?

Open questions for reviewers:

  • Is fluss-lake-iceberg-common the right home, or should this be lake-neutral in fluss-common (SupportsRecordBatchWrite plus a file-writer SPI that Paimon could also use)?
  • ServiceLoader plus a lake option, or a class-name option such as tiering.arrow-writer.class?
  • Does this need a FIP?

Willingness to contribute

  • I'm willing to submit a PR!

Activity

  1. mikepapadim commented on Nov 13, 2026

    @mikepapadim
    Author

    Method, full numbers and limits for the motivation above: #5227

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions