Skip to content

Arrow: Add FileAppender<VectorSchemaRoot> for vectorized Parquet writes #17748

Open

Description

@qzyu999

Problem

Iceberg's write path (FileAppender<Record>, TaskWriter<Record>) accepts only row objects. Projects with Arrow-columnar data (Fluss, Flink, PyIceberg) must materialize GenericRecord instances to write Iceberg tables, even when the source data is already in columnar form.

The vectorized read path exists (iceberg-arrow module) but there is no write-side equivalent.

Proposal

Add a FileAppender<VectorSchemaRoot> implementation (likely in iceberg-arrow or a new iceberg-arrow-writer module) that:

  1. Accepts Arrow VectorSchemaRoot batches
  2. Writes to Parquet using Iceberg's existing ParquetWriter infrastructure (or parquet-java's Arrow writer once parquet-java#3733 lands)
  3. Collects Iceberg Metrics (column sizes, null counts, min/max bounds) from the Arrow vectors
  4. Integrates with existing OutputFileFactory, TaskWriter, and file rolling logic

Motivation

  • Apache Fluss — Arrow-native streaming storage that tiers to Iceberg. Currently materializes GenericRecord per row (fluss#4047).
  • Apache Flink — Flink's Iceberg connector serializes internal rows to Record objects at the write boundary.
  • PyIceberg / iceberg-rust — Arrow-native clients (Polars, DataFusion, DuckDB) writing to Iceberg.

Dependencies

  • Can start today using Iceberg's internal ColumnWriteStore access (Iceberg's ParquetWriter already manages this directly, not through parquet-java's public ParquetWriter.Builder)
  • Full zero-copy encoding depends on parquet-java#3733, but an initial implementation iterating Arrow vectors per-value is still a significant improvement over row-object construction

Scope

  • New module or extension in iceberg-arrow
  • Existing FileAppender<Record> and TaskWriter<Record> APIs unchanged
  • Arrow dependency remains optional

Activity

  1. iskenderunlu commented on Sep 2, 2026

    @iskenderunlu

    I will work on this.

  2. qzyu999 commented on Sep 2, 2026

    @qzyu999
    Author

    I will work on this.

    Hi, still waiting on the Parquet Java PR, if you have time please review. apache/parquet-java#3734

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions