You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Arrow: Add FileAppender<VectorSchemaRoot> for vectorized Parquet writes #17748
Iceberg's write path (FileAppender<Record>, TaskWriter<Record>) accepts only row objects. Projects with Arrow-columnar data (Fluss, Flink, PyIceberg) must materialize GenericRecord instances to write Iceberg tables, even when the source data is already in columnar form.
The vectorized read path exists (iceberg-arrow module) but there is no write-side equivalent.
Proposal
Add a FileAppender<VectorSchemaRoot> implementation (likely in iceberg-arrow or a new iceberg-arrow-writer module) that:
Accepts Arrow VectorSchemaRoot batches
Writes to Parquet using Iceberg's existing ParquetWriter infrastructure (or parquet-java's Arrow writer once parquet-java#3733 lands)
Collects Iceberg Metrics (column sizes, null counts, min/max bounds) from the Arrow vectors
Integrates with existing OutputFileFactory, TaskWriter, and file rolling logic
Motivation
Apache Fluss — Arrow-native streaming storage that tiers to Iceberg. Currently materializes GenericRecord per row (fluss#4047).
Apache Flink — Flink's Iceberg connector serializes internal rows to Record objects at the write boundary.
Can start today using Iceberg's internal ColumnWriteStore access (Iceberg's ParquetWriter already manages this directly, not through parquet-java's public ParquetWriter.Builder)
Full zero-copy encoding depends on parquet-java#3733, but an initial implementation iterating Arrow vectors per-value is still a significant improvement over row-object construction
Scope
New module or extension in iceberg-arrow
Existing FileAppender<Record> and TaskWriter<Record> APIs unchanged
Activity
iskenderunlu commentedon Sep 2, 2026
I will work on this.
qzyu999 commentedon Sep 2, 2026
Hi, still waiting on the Parquet Java PR, if you have time please review. apache/parquet-java#3734