You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4047
I searched in the issues and found nothing similar.
Motivation
Paimon's lake writer implements SupportsRecordBatchWrite (landed in #3417/#3418), enabling direct Arrow batch to Parquet writes for append-only tables. This avoids the row-by-row serialization overhead through GenericRecord objects.
The Iceberg lake writer (IcebergLakeWriter) currently writes row-by-row via GenericAppenderFactory, resulting in significantly lower throughput for append-only log table tiering compared to the Paimon path.
Since Fluss uses Arrow as its internal columnar format, and both Iceberg and Parquet have native Arrow support (iceberg-arrow module, parquet-arrow module), we can bridge the Arrow batch directly to Iceberg Parquet files.
Solution
Make IcebergLakeWriter implement SupportsRecordBatchWrite
Create an Arrow batch writer that takes ArrowBatchData and writes to Iceberg Parquet files
Only enabled for append-only tables (same restriction as Paimon)
Preserve Iceberg file metrics tracking for proper table maintenance
The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors and producing GenericRecord objects. This avoids binary log deserialization but still materializes row objects because Iceberg and parquet-java lack columnar write APIs.
Upstream issues filed to enable true zero-copy Arrow-to-Parquet writes:
apache/parquet-java#3733 — Add ParquetWriter<VectorSchemaRoot> to the parquet-arrow module. This is the foundational primitive.
apache/iceberg#17748 — Add FileAppender<VectorSchemaRoot> to Iceberg's arrow module. Depends on (or works alongside) the parquet-java change.
Once these land, Fluss can replace the IcebergArrowBatchHelper (GenericRecord bridge) with a direct call to Iceberg's vectorized appender.
Anything else?
All necessary primitives exist in upstream Iceberg (iceberg-arrow 1.10.1) and Apache Parquet (parquet-arrow). No upstream changes needed for the initial implementation; upstream changes enable the optimized path.
The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors directly and producing Iceberg GenericRecord objects. This avoids binary log deserialization but still materializes row objects because parquet-java lacks a batch/columnar write API.
For true zero-copy Arrow-to-Parquet writing (matching what Paimon achieves with paimon-arrow), the gap is in the underlying Parquet Java library:
apache/parquet-java#3733 Filed to request batch write APIs + Arrow VectorSchemaRoot writer (Java parity with C++/Python write_table())
Once parquet-java exposes batch write methods on ValuesWriter, the Iceberg and Fluss layers can be updated to pass Arrow column buffers directly without row materialization.
Activity
qzyu999 commentedon Aug 20, 2026
Upstream Context
The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors directly and producing Iceberg GenericRecord objects. This avoids binary log deserialization but still materializes row objects because parquet-java lacks a batch/columnar write API.
For true zero-copy Arrow-to-Parquet writing (matching what Paimon achieves with paimon-arrow), the gap is in the underlying Parquet Java library:
Once parquet-java exposes batch write methods on ValuesWriter, the Iceberg and Fluss layers can be updated to pass Arrow column buffers directly without row materialization.