Skip to content

[lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4047

Open

Description

@qzyu999

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Paimon's lake writer implements SupportsRecordBatchWrite (landed in #3417/#3418), enabling direct Arrow batch to Parquet writes for append-only tables. This avoids the row-by-row serialization overhead through GenericRecord objects.

The Iceberg lake writer (IcebergLakeWriter) currently writes row-by-row via GenericAppenderFactory, resulting in significantly lower throughput for append-only log table tiering compared to the Paimon path.

Since Fluss uses Arrow as its internal columnar format, and both Iceberg and Parquet have native Arrow support (iceberg-arrow module, parquet-arrow module), we can bridge the Arrow batch directly to Iceberg Parquet files.

Solution

  1. Make IcebergLakeWriter implement SupportsRecordBatchWrite
  2. Create an Arrow batch writer that takes ArrowBatchData and writes to Iceberg Parquet files
  3. Only enabled for append-only tables (same restriction as Paimon)
  4. Preserve Iceberg file metrics tracking for proper table maintenance

Related: #437 (generic arrow-to-parquet issue)

Upstream Issues for True Zero-Copy Path

The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors and producing GenericRecord objects. This avoids binary log deserialization but still materializes row objects because Iceberg and parquet-java lack columnar write APIs.

Upstream issues filed to enable true zero-copy Arrow-to-Parquet writes:

  1. apache/parquet-java#3733 — Add ParquetWriter<VectorSchemaRoot> to the parquet-arrow module. This is the foundational primitive.
  2. apache/iceberg#17748 — Add FileAppender<VectorSchemaRoot> to Iceberg's arrow module. Depends on (or works alongside) the parquet-java change.

Once these land, Fluss can replace the IcebergArrowBatchHelper (GenericRecord bridge) with a direct call to Iceberg's vectorized appender.

Anything else?

All necessary primitives exist in upstream Iceberg (iceberg-arrow 1.10.1) and Apache Parquet (parquet-arrow). No upstream changes needed for the initial implementation; upstream changes enable the optimized path.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

  1. qzyu999 commented on Aug 20, 2026

    @qzyu999
    ContributorAuthor

    Upstream Context

    The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors directly and producing Iceberg GenericRecord objects. This avoids binary log deserialization but still materializes row objects because parquet-java lacks a batch/columnar write API.

    For true zero-copy Arrow-to-Parquet writing (matching what Paimon achieves with paimon-arrow), the gap is in the underlying Parquet Java library:

    Once parquet-java exposes batch write methods on ValuesWriter, the Iceberg and Fluss layers can be updated to pass Arrow column buffers directly without row materialization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions