Repository navigation
Description
Implementation progress
Implemented in #386. Supported contracts accelerate by default. Optimized full 2M streaming Nexmark: 120 cells / 480 pairs pass, with no native row transposes/operator fallback. README-selection median geomeans: 1.22× / 1.81× memory and 2.80× / 2.94× RocksDB (mini-batching off/on). All ranges, previous-native comparison, candidate outcomes and audit retain regressions, including a 10% previous-native slowdown in RocksDB/on. Validation: 303 pass, 13 batch exclusions, six Rust units and 126 native fixtures. V3 sources remain stock on released 0.10.1. Dynamic #387 and state/checkpoint #391 remain open.
Problem and intended behavior
Accelerate Iceberg streaming Parquet reads/writes while retaining released Java table and checkpoint semantics. Java may traverse borrowed Arrow views; data payloads remain Arrow through Rust encoding, with no materialized Java rows or per-row JNI writes. Measure columnar gathers/casts honestly. Extends #13.
Streaming-only scope
All checkpoints concern streaming reads/writes; Arrow batches are the transport, not Flink batch execution. Batch reads/writes, overwrite/partition replacement and maintenance (compaction, expiration, orphan cleanup, delete conversion) are excluded. Existing Java commits, recovery and delete-file encoding remain in scope.
Initial sink deployment scope
- Target released Flink 2.2 and Iceberg Java 1.12.0 artifacts first. Begin with the normal single-table sink;
DynamicIcebergSinkis a later checkpoint below. - First deployable milestone: unpartitioned tables, INSERT-only input, Parquet, distribution
none, and explicitly verified schemas/settings. Disable upserts and require no equality-delete writer configuration, even if the input happens to contain only inserts. - Add partitioned append as a subsequent milestone under the same architecture, retaining Java's partition transforms and fanout writers.
- During initial append admission, keep updating/retracting input and identifier/equality-field tables on the stock connector; the keyed follow-on checkpoint below expands these explicitly. Dynamic destinations remain stock until the dynamic checkpoint. Unverified types/options, encryption, and unimplemented distribution/runtime modes remain stock fallback. Preserve the host's streaming overwrite rejection; batch overwrite is outside scope.
- Distinguish the normal legacy sink from the normal Sink V2 implementation. Iceberg 1.12.0's Flink 2.2 SQL connector defaults
table.exec.iceberg.use-v2-sinkto false. Initially accelerate only the runtime API actually wired and tested; preserve the user's selected API and fall back for the other. Do not silently switch it.
Architecture and smallest hook
The encoding boundary is Iceberg's FileAppender, supplied through FileWriterFactory.newDataWriter(...) and wrapped by the existing Java DataWriter.
StreamFusion ArrowBatch
-> columnar sink adapter
-> Java task writer visits Arrow-backed row cursors
-> Java rolling writer / DataWriter
-> StreamFusion FileAppender records accepted Arrow positions
-> parquet-rs single-file AsyncArrowWriter
-> custom Storage / FileWrite adapter
-> bounded JNI byte bridge
-> Java Iceberg-owned output stream
On file close:
native file statistics / split offsets
-> Java Iceberg metrics policy and DataFile
-> existing task completion / WriteResult
-> normal Flink Iceberg checkpoint and commit topology
Java retains catalog/table loading, schema validation, partition transforms, output locations and names, target-size rolling, file lifecycle, manifests, snapshot/branch properties, commits, and recovery. Rust encodes one Java-selected file at a time. Do not instantiate iceberg-rust's rolling or table writer in this path.
Released sink wiring must be proved first
Neither normal Flink sink currently exposes a public appender/factory setter. Both construct RowDataTaskWriterFactory, which constructs its own FlinkFileWriterFactory. The format registry is not an override mechanism: (PARQUET, RowData.class) is already registered, duplicates are rejected, and lookup is by the exact class.
The implementation therefore needs a small StreamFusion-owned adapter around sink writer creation and an append task-writer factory that supplies our FileWriterFactory. Reuse Java's public unpartitioned/fanout task writers and DataWriter where possible. Delegate the selected normal sink's downstream commit machinery rather than reimplementing it.
Demonstrate it against the published connector with a fake appender before building the native encoder. Any required package/classloader-sensitive access must be explicit, version-gated, and tested. Do not expand this into a broad copy of the connector or a fork/snapshot/local build of Iceberg; if the narrow approach cannot work on released APIs, record the concrete limitation before changing architecture.
Keeping rolling semantics
Keep one Java task-writer write() per actual row using an Arrow-backed cursor. The appender records only the batch positions that have reached that file; it must never encode future positions merely because they share an input batch.
Iceberg's BaseTaskWriter increments its file row count once per write and checks length() every 1,000 rows. Preserve that exact control flow:
- Buffer references to accepted Arrow ranges/selections and retain their ownership.
- Encode pending accepted positions before returning the appender's size estimate to a rolling check. Also drain at bounded batching/memory thresholds and before close; draining must not force a Parquet row-group flush.
- Use parquet-rs
bytes_written() + in_progress_size()while open (identical to the released Iceberg size estimate), then the final exact file size after close. - When Java rolls, close/finalize that native encoder and open another for Java's next output file. An input batch may span files.
- Drain and release all pending references before checkpoint completion; never serialize native handles or Arrow pointers into checkpoint state.
This preserves Java's decision cadence and target-size policy. It does not promise identical file boundaries or byte sizes: parquet-rs and parquet-mr use different buffering/compression estimates. Tests must distinguish preserved rolling control flow from encoder-dependent layout.
For partitioned append, each file tracks only positions actually routed to it. Test interleaved partitions, sparse selections, null partition values, and rolling within one shared input batch. Reuse Java's partition transforms, partition-key copies, escaping, and file naming.
Single-file native encoder and released Iceberg reuse
Use released iceberg 0.10.1 (Arrow/Parquet 58, Rust 1.94) for schema conversion, FileIO adapters and source task decoding. Encoding now calls its underlying parquet-rs AsyncArrowWriter directly: the released Iceberg wrapper hides retained-memory reporting. Charge reported encoder retention, estimated row-group metadata and the bounded 1 MiB JNI output buffer per open file against the task off-heap budget. Update reservations after writes; release on close/abort; quota failure aborts pending appends. This does not claim a bound on transient codec or decoder allocations. Byte-for-byte tests against the released wrapper cover codecs/page versions, sparse selections, NaNs, rolling-size estimates and split offsets.
OutputFile::new(Arc<dyn Storage>, path) / Storage / FileWrite bridge the Java-selected output stream. Keep Java FileIO plugins, credentials and configuration for every scheme, bounded reusable byte transfer and synchronous backpressure. No whole-file staging or hacked in-memory filesystem is needed. Close/abort and JNI thread-attachment ownership remain explicit. Comet's direct Arrow/Parquet writer is the reference.
JNI contract and ownership
- On file creation, send the Iceberg schema (including nested IDs), effective encoding settings, and file identity/metadata needed for one encoder. A full catalog or live table object is unnecessary.
- Transfer payloads through the existing Arrow C Data ownership conventions; send batch/range/selection information, not row payload serialization. Retain buffers until encoding finishes, then release promptly.
- Return an explicitly defined result containing raw file statistics, row count, final size, and split offsets. Do not assume Rust and Java object serializers or complete
DataFileobjects are wire-compatible. - Preserve StreamFusion's internal timestamp representation until the file boundary, then convert column-wise to the Iceberg-required physical representation.
- Follow Comet's JNI/Arrow ownership and alignment patterns. Include under-aligned Decimal128 buffers and sliced nested arrays in verification; Arrow 58 imports must not assume JVM buffers have Rust's preferred alignment.
Configuration and metadata compatibility
Resolve settings with Iceberg Java's FlinkWriteConf and SinkUtil.writeProperties, including per-write/session/table precedence and defaults. Inspect Hadoop-side options where the stock writer consumes them. Admit only combinations we reproduce; do not assume similarly named Flink, Paimon, and Iceberg options have identical defaults.
Important released-version detail: iceberg-rust 0.10.1's from_table_properties translates content-defined chunking but leaves most other properties at parquet-rs defaults. Build explicit parquet-rs WriterProperties; do not rely on that convenience method for configuration parity.
| Behavior | Initial treatment |
|---|---|
| Compression and codec levels | Translate effective values for individually verified codecs/levels; reject unsupported behavior. |
| Row-group byte target, page bytes, dictionary bytes, dictionary enablement, page version | Translate and verify physical output. Remove unintended parquet-rs default row-count caps where the host has none. |
| Page row limit | Honor Iceberg's default 20,000 as well as admitted overrides; this is required even without an explicit property. |
| Custom row-group/page size-check controls | Initially require verified defaults; encoder heuristics and exact split points can differ. |
| Per-column dictionary/statistics controls, bloom filters, adaptive blooms, content-defined chunking, encryption | Fall back until each effective setting is explicitly mapped and tested. |
| Unknown writer-affecting options | Fall back with a concrete reason. Reader-only or host-owned options should not be rejected indiscriminately. |
| Metrics modes, bounds truncation, inferred-column limits, list/map bounds suppression | Preserve Java Iceberg's MetricsConfig and interpretation; Rust supplies raw statistics. |
Reuse Java's footer/metrics utilities where appropriate, supplementing them with native field metrics such as NaN counts. Footer-only extraction is insufficient for complete floating-point parity. Prove the conversion into Java metrics, including signed zero, NaNs, nulls, nested values, and logical bound encoding. Reject combinations whose metadata cannot be reproduced accurately.
Keep Java DataWriter responsible for constructing the committed DataFile, with its partition/spec and key/sort metadata. Verify split offsets are valid, sorted, and usable by stock readers.
The single-file writer does not automatically establish Java writer parity. Verify nested field IDs, required/optional fields, list/map layout, decimal physical types, TIME units, timestamp timezone annotations/ranges, binary/fixed/UUID handling, and footer metadata. Unsupported types remain outside admission. Document harmless differences such as created_by, root schema naming, dictionary labels, page checksums, or footer metadata when present; never confuse those with incorrect data, field IDs, statistics, or transactional behavior.
Partitioned append extension and optional native partition calculation
Reuse Java's PartitionedFanoutWriter through a small subclass computing PartitionKey from reusable Arrow-backed views, as in RowDataPartitionedFanoutWriter. Java owns transforms, stable key copies, escaped paths/names, partition/spec metadata, independent per-partition rolling, and commits. Each native appender retains its accepted Arrow positions; encode contiguous slices directly and gather interleaved selections column-wise. Retain shared batches until all referencing appenders finish, include partition source columns in the data schema, and account for aggregate open-writer/retained-batch memory. Drain selections without forcing row-group flushes or changing rolling cadence.
Start with distribution none and identity partitions; admit bucket/time/truncate transforms incrementally. Hash/range require the corresponding verified Arrow exchange. Add upstream SQL coverage alongside implementation: static/dynamic partition inserts, null/escaped values, interleaved partitions, batches spanning files, and checkpoint/recovery.
Optional optimization: iceberg-rust's PartitionValueCalculator computes Arrow partition columns by batch; RecordBatchPartitionSplitter is another reference. Verify availability in the selected published crate. Initially retain Java calculation. Only if profiling justifies it, compute native partition columns once per batch and expose Arrow-backed views from which Java constructs keys; retain Java paths/fanout/rolling/metadata. Avoid per-row JNI and assumptions about key-object layouts.
Require exact Java parity for transforms/types, nulls, negative/pre-epoch values, bucket hashing, truncation, composite keys, nested field-ID projection, slices, and timestamps. Require upstream SQL native witnesses and release measurements including partition computation, JNI/ownership, Java key construction, allocation/retention, and sustained throughput before enabling this optimization.
Next checkpoint: identifier fields / primary keys with native data files only
After append-only writing and partitioned append are established, extend admission to Iceberg identifier fields, Flink primary keys, and the associated updating/upsert inputs. The agreed scope is native data-file writing only: all delete-file encoding remains Java-owned. Native delete-file encoding is not a completion requirement or an intended follow-up under this plan. Everything outside data-file payload encoding should continue through the normal released Java connector wherever possible.
Reuse Java's change-handling writers
Preserve the connector's resolution of schema identifier fields and configured equality fields, plus its selection of append versus delta task writers. A nonempty equality-field set selects the stock delta task-writer path even with upsert disabled, so identifier-field tables must not simply bypass the initial append admission gate.
Supply the custom factory's native newDataWriter(...) while delegating newEqualityDeleteWriter(...) and newPositionDeleteWriter(...) to the released Java factory. Preserve its exact equality-delete schemas and field-ID configuration, which differ between upsert and non-upsert modes. Java continues to own generated delete records and their encoding, including position-delete bookkeeping for rows replaced within a writer/checkpoint.
Keep Java's RowKind dispatch, ignored update-before behavior in upsert mode, key projection/copying, duplicate-key maps, row-position accounting, partition routing, file rolling, completed-file aggregation, manifests, sequence-number interpretation, checkpoint commits, and recovery. Keys and generated delete records may be materialized as required by the stock connector; the data-file payload must remain Arrow-backed until iceberg-rust writes Parquet. Reuse the connector's existing distribution rule for the admitted mode; when that requires a keyed exchange, implement a batch-preserving equivalent with matching key/channel semantics before admitting it.
The data appender must recognize Arrow-backed rows through the stock projection/wrapper layers without reconstructing payload rows. Its buffering must preserve the accepted write order and logical row offsets exactly: Java's position-delete references must identify the same rows after native batches are drained or a file rolls. Do not perform native deduplication, reorder accepted rows, or omit a row merely because Java later emits a delete for it. Buffering must not move a row to a different file from the one recorded by Java.
Admission and progression
Start with verified format-v2 tables and a narrow supported key/type set. Prove unpartitioned identifier-field append behavior first, then upsert/change handling and partitioned keyed writes. Preserve the connector's distinction between a declared identifier/primary key and enabled upsert mode. For partitioned upserts, retain its validation that partition source columns are included in the equality fields; do not introduce global cross-partition key semantics that the host does not provide.
Keep format-v3 tables outside initial keyed admission. If admitted later, retain Java's existing deletion-vector/Puffin path as well; the native-data-only boundary still applies. Do not require every Iceberg sidecar to be Parquet.
Required evidence in the same checkpoint
- Expand the upstream Iceberg Flink SQL selection with the corresponding identifier-field/primary-key and changelog/upsert cases, preserving upstream SQL, inputs, RowKinds, assertions, and recovery checks. Include the relevant upstream task-writer tests for duplicate keys and delete generation.
- Require native data-file encoding witnesses and separate evidence that equality/position delete writers remain the released Java implementations. A stock-only passing test does not establish native coverage.
- Compare final logical results and required delete/manifest metadata against stock Flink for repeated keys within a batch, across native drains, across batches, across file rolls, and across checkpoints; include changes to non-key values, deletes, update-before/update-after handling, composite keys, and empty/delete-only intervals.
- Exercise recovery before/after file finalization and checkpoint commit, plus partitioned cases and any admitted keyed exchange. Verify delete references, row positions, file/spec identity, and commit semantics even where native and stock file layouts differ.
- Measure end-to-end release performance with the real Java key bookkeeping, delete encoding, exchange, JNI, and commit costs included. This checkpoint accelerates data-file encoding; it does not claim native upsert state or delete processing.
Next checkpoint: normal dynamic Flink sink with the same native data-file boundary
After the single-table append, partitioned, and keyed checkpoints, extend the same integration to Flink's DynamicIcebergSink. The scope is deliberately narrow: keep data-file payloads in Arrow, shuffle them in batches, and write data files from Arrow batches in Rust. Java per-row traversal and all other host work are acceptable. Prefer the smallest adapters that preserve those three properties; do not replace dynamic table management, routing decisions, schema policy, or commits merely to make them native.
Reuse Java's dynamic sink and writer factories
The released dynamic writer creates the same RowDataTaskWriterFactory per write target: table, branch, schema ID, partition-spec ID, upsert mode, and equality fields. Adapt that factory-creation boundary to supply the previously implemented native-data/Java-delete factory. Reuse the single-file parquet-rs Arrow encoder, Java rolling/task writers, and metadata bridge from the earlier checkpoints.
Keep the stock dynamic generator and processor, catalog/table discovery and creation, metadata caches, partition/equality-key calculation, destination and writer-channel decisions, schema/spec evolution policy, per-target writer lifecycle, write-result aggregation, and dynamic committers wherever the payload contract permits. All equality/position-delete encoding remains Java-owned. Preserve the existing forward and shuffled writer paths and their normal result aggregation; do not introduce cross-table transaction guarantees beyond the host's behavior.
Java may inspect Arrow-backed rows to resolve destinations and routing. Sidecar metadata can be ordinary Java objects and serialized with host machinery. Keep the actual data-file payload as Arrow buffers, with row positions/selections associated with the resolved target. Preserve the full target identity and row/key order, including RowKinds, through buffering, batching, and recovery.
Replace payload boundaries only where required
The stock dynamic record serializer serializes RowData payloads. Adapt the shuffled boundary to carry Arrow batches plus routing/target metadata, splitting or gathering rows by the connector's selected destination channel as needed. Preserve the selected distribution mode, per-target parallelism, and key/channel semantics; do not replace them with a new routing policy. Transport schema/spec metadata and any checkpointed metadata using the existing host representations where possible. Retain normal Flink backpressure and record/buffer ownership. A forward path can retain Arrow references within its task; payloads crossing a network boundary must use a batch representation, not task-local pointers or one serialized row per element.
The stock schema converter may allocate GenericRowData when it reorders fields, fills missing columns, or widens types. This is the other boundary to adapt when needed: use Arrow-backed projection/conversion views, defer a conversion to the native file boundary when routing does not need it, or use existing columnar kernels. Preserve Java's resolution, validation, defaults/null behavior, and evolution decisions. Do not build an independent native schema-evolution engine. When a generator or converter actually replaces/materializes the payload and cannot meet the Arrow contract, keep that combination outside native admission until a narrow adaptation exists.
Java per-row traversal is explicitly allowed, including routing, partition/key inspection, bookkeeping, and generated delete records. Per-row payload transposition and per-row payload serialization across the shuffle are not. Compute metadata in the host, then hand accepted Arrow selections to the existing native data appender in batches.
Keyed Arrow dynamic contracts require dynamic-sink.immediate-table-update=true; the released deferred-updater/direct-output union can reorder values during metadata refresh. Reject deferred keyed contracts; keep deferred append admission. Preserve Flink's aligned committable exchanges and capture Arrow payload channel state. Recovery fixtures keep the source live and verify changing final keyed values after captured-channel rescaling, with a non-draining savepoint before terminal drain.
Incremental admission and required tests
- Prove the released dynamic factory wiring and reuse of Java aggregation/commit behavior. Start with verified generators, destinations, and matching schemas; expose clear admission limits rather than assuming arbitrary runtime targets are supported.
- Enable multi-table/multi-branch append with Arrow payloads on both admitted forward and shuffled paths. Preserve independent settings, schemas/specs, file naming, rolling, and writer identity per target.
- Extend to partitioned and keyed/upsert targets using the earlier Java writer and delete-encoding paths. Preserve per-key order and Java's file/row-position references through batch grouping and shuffle.
- Expand supported schema/spec changes only where an Arrow-backed view or columnar conversion is necessary. Reuse the host's evolution machinery throughout; no extra native implementation is required for behavior Java can already perform while retaining the Arrow payload.
Use the existing upstream integration harness for the corresponding Iceberg dynamic sink, processor, writer, serializer, aggregation, and committer tests, plus SQL entry points where supported. Add cases as each capability is admitted. Preserve upstream queries, fixtures, assertions, and parameter coverage, and add explicit witnesses for Arrow batch transport, Rust data-file encoding, and Java delete writers. Exercise mixed/interleaved destinations, differing target settings, both distribution paths, schema/spec identities, checkpoint/recovery, empty/delete-only intervals, and failure during target creation or file finalization. Bound and measure aggregate open-writer memory and retained Arrow inputs across destinations.
Dynamic targets introduce an admission-design checkpoint: destination schemas/settings may not all be knowable at planning time. Establish the supported generator/destination contract or an explicit, tested host fallback boundary before broadening admission; do not silently switch a partially written native file to another writer. Unsupported dynamic configurations should retain the normal connector behavior.
Benchmark the production path with Java routing/conversion policy, batch splitting/exchange, metadata serialization, native encoding, Java delete writing, and commits included. Prefer less implementation work whenever it preserves the three required properties. Do not make native routing, native partition calculation, native delete encoding, or native commits prerequisites for this checkpoint.
Reference implementation: dynamic factory creation and target lifecycle, normal dynamic topology, row payload serializer, and schema converter.
Shared Arrow batch exchange for every distribution mode
Use one shared payload exchange for hash, range, keyed, and dynamic routing. Let the existing Java selector/partitioner determine each row's destination using Arrow-backed views; collect accepted row positions per destination, split/gather into Arrow batches, and send one batch at a time through Flink's network. Preserve record/RowKind order within each destination and retain normal ownership/backpressure behavior. Routing metadata and Java per-row traversal are allowed; per-row payload serialization and per-row JNI writes are not.
Range distribution needs an adapter for its Java statistics/coordinator/control-message path and its upstream tests, not a separate Arrow shuffle implementation. Preserve the ordering of routing-state changes and control records relative to the input rows and checkpoint barriers; apply the host-selected routing state to each row, then use the shared batch split/exchange. Likewise, dynamic routing contributes destination/target metadata and existing writer-channel decisions to the same mechanism. Do not implement native hash/range/partition decisions when Java can already compute them while leaving the payload in Arrow.
Further streaming checkpoint: range distribution
Iceberg Java 1.12.0 supports experimental RANGE in both normal streaming sink APIs, requiring a partitioned table or sort order; distribution does not sort rows within files. Preserve Java statistics, coordinator aggregation, range decisions, checkpoint updates, and configuration. Use the shared Arrow exchange while preserving control ordering, ownership, and backpressure. Test low/high cardinality, skew, changing checkpoint boundaries, and recovery with upstream cases, transport witnesses, and release benchmarks.
Preserve equality-field hash fallback in the normal sink and hash fallback for dynamic sinks, which lack true range distribution. Gate by the tested released API. References: streaming documentation, legacy sink, Sink V2.
Checkpoint: format-v3 streaming writes with native data files
Extend the verified streaming native-data-file boundary to existing format-v3 tables. Keep Java's deletion-vector/Puffin writing, other delete encoding, row-lineage/sequence metadata handling, file aggregation, and commits. Validate that native Parquet row counts, ordering, file identity, statistics, and field metadata integrate with the released host's v3 behavior; do not infer v3 compatibility solely from successful v2 readback.
For incoming new rows, v3 permits data files to omit _row_id and _last_updated_sequence_number; readers inherit them from the Java-committed file/manifest metadata. Equality-delete updates are modeled as removing an old row and adding a new row, so retain the Java connector's lineage behavior rather than implementing a native row-ID allocator. Accurate native record counts, stable physical row ordering, and Java-owned file identity are essential to both lineage inheritance and deletion-vector positions. Test lineage values/metadata and commit retries as well as final row values. This does not cover moving/re-writing existing rows with preserved lineage, which belongs to the deferred maintenance/rewrite scope. See the released row-lineage rules.
Start with previously admitted data types and encoding settings. V3-specific types or encoding features require separate explicit admission and tests rather than becoming supported automatically. Add relevant upstream streaming append/upsert, deletion-vector, and recovery tests incrementally, proving native data-file writes alongside Java sidecars. Preserve the host's limitations: the dynamic sink does not support upgrading a table from v2 to v3 while the job is running. A live format upgrade is not part of this checkpoint.
V3 implementation stages and completion criteria
- Append to existing v3 tables: admit previously verified schemas/settings and reuse the existing Rust data-file writer. Keep Java responsible for lineage assignment/inheritance metadata, manifests, sequence numbers, and commits. No Rust row-ID allocator or new native table writer is required.
- Keyed v3 streaming writes: retain Java equality-delete and deletion-vector/Puffin paths while encoding only data files natively. Verify that native buffering, selections, file rolls, and physical row order preserve every Java file/position reference.
- V3 across admitted sink paths: extend the same verified behavior to the supported normal sink APIs and dynamic destinations, preserving branch/spec identity and the host's format-upgrade restrictions.
- Published-artifact wiring preserves Java's v3 writer selection and sidecar/commit behavior while runtime witnesses prove Rust data-file encoding.
- Upstream streaming append/upsert/delete and recovery tests are included in the required integration suite as each stage lands, with unchanged upstream assertions and explicit native-versus-fallback evidence.
- Stock v3 readback verifies lineage columns and file/manifest sequence metadata as well as final logical rows, including repeated keys, mid-batch file rolls, checkpoint boundaries, retry/recovery, and admitted branches/destinations.
- Native results report correct row counts, statistics, split offsets, and file identity; no new row-ID allocator, delete encoder, or commit implementation is introduced.
- Existing supported types/settings are admitted first. New v3 types/default semantics stay gated until separately verified. Maintenance/rewrite jobs and live v2-to-v3 upgrades remain outside scope.
- Release-mode end-to-end comparisons retain Java key/sidecar processing, batch exchange, JNI, native encoding, and checkpoint/commit costs; document verified coverage and limitations before enabling acceleration.
Implementation milestones
- Prove the Java hook: small sink/factory adapter plus fake appender using released artifacts. Demonstrate reuse of rolling, task completion, streaming checkpoints, and the selected normal commit topology. Establish explicit admission and stock fallback. In parallel, wire the Iceberg upstream SQL suite into the existing harness and establish its released-stock baseline.
- Add single-file native encoding: optional Iceberg JNI API, custom output adapter, Arrow retention/selections, explicit writer properties, file result conversion, and an unpartitioned INSERT-only local deployment. Require the admitted upstream SQL append cases to pass with native encoding witnesses; expand the upstream selection with every later coverage milestone.
- Prove compatibility and lifecycle: stock readback, settings/metadata parity, size checks and mid-batch rolls, failure cleanup, checkpoint/restart, and a representative remote Java FileIO path.
- Extend to partitioned append: reuse Java transforms/fanout; preserve Arrow payloads and Java partition paths. Add admission incrementally for transforms, keys, and distribution modes that have end-to-end evidence.
- Measure before enabling: native Criterion and sustained end-to-end stock/native comparisons, then document the verified coverage and deployable configuration.
- Extend to identifier fields / primary keys: reuse the normal Java delta writers and all Java delete-file encoders, replacing only data-file encoding. Start with verified format-v2/key combinations; add matching upstream SQL/task-writer tests and preserve row positions, distribution, checkpoint, and recovery behavior. Benchmark and document this expanded coverage before enabling it.
- Extend to the dynamic Flink sink: reuse Java destination/table/schema/commit machinery and the same native-data/Java-delete factory. Adapt only the boundaries necessary to retain Arrow payloads, shuffle batches, and encode data files in Rust; allow Java per-row traversal. Add matching upstream dynamic-sink and available SQL coverage incrementally, then benchmark and document the admitted combinations.
- Preserve normal streaming range distribution: reuse Java statistics/coordinator decisions with Arrow batch exchange; preserve the normal equality-key hash fallback and dynamic sink hash fallback. Add the upstream range/recovery cases and measured batch-transport evidence.
- Admit format-v3 streaming writes: deliver the explicit v3 checkpoint above in stages: append to existing v3 tables, keyed writes with Java equality-delete/deletion-vector encoding, then admitted normal/dynamic sink paths. Reuse the Rust data-file writer, preserve Java lineage/manifest/commit handling, and require upstream streaming tests plus lineage/readback/recovery evidence. New v3 types/default semantics remain gated; live upgrades and maintenance remain outside scope.
- Add the streaming Parquet source: preserve Java snapshot/split discovery and checkpoints; translate planned file tasks to released iceberg-rust and emit Arrow directly. Prove sequential task/offset recovery first, then expand supported schema/filter and Parquet-delete semantics. Keep deletion vectors gated until a published crate supports them; require upstream streaming source tests, native witnesses, and release measurements.
- Benchmark Iceberg end to end: add the final Nexmark checkpoint below: match the README matrix with four Iceberg input/output partitions, actual-key upsert outputs, stock/native correctness, committed-checkpoint timing, and published release results.
Create an optional streamfusion-iceberg artifact with its Java integration and matching native library. Keep iceberg-rust dependencies out of connector-neutral core and the generic Parquet module. Reuse shared codec/byte-bridge abstractions where useful; declare only the optional module dependencies actually needed. Missing modules must yield planner fallback. Deploy alongside the published normal Flink Iceberg connector.
Upstream Iceberg Flink SQL suite: add coverage as implementation progresses
Integrate Iceberg's upstream Flink SQL tests into StreamFusion's existing upstream integration harness from the first implementation milestone. This is required work throughout implementation, not a final cleanup task or a replacement by connector-local unit tests.
- Add an
icebergmode tobin/flink-suite.sh, with a pinned released Iceberg source-test inventory, published production artifacts, and the existing supported Flink-line/classpath isolation. Add it to the suite matrix, required upstream CI gate, report/diagnostic collection, and suite documentation. Do not build a modified sibling checkout or unreleased Iceberg production code. - Start by inventorying and running the portable cases from upstream
TestFlinkTableSinkandTestFlinkTableSinkExtended, including the shared fixtures they require. Preserve upstream SQL, inputs, parameter coverage, expected results, assertions, and timeouts. Keep necessary harness adaptations explicit and separate from production behavior. - Establish the released-stock baseline while proving the Java appender hook. With the first native unpartitioned append path, require the corresponding upstream SQL insert tests to pass and show a runtime witness that native Iceberg Parquet encoding executed. A passing stock fallback is not evidence of native coverage.
- Expand the selected upstream cases in the same changes that add partitioned append, schema/type/settings coverage, branches, or another normal sink runtime API. Supplement SQL cases with upstream sink checkpoint/recovery tests as the lifecycle is implemented, plus targeted StreamFusion tests for Arrow ownership and the JNI boundary.
- Keep portable out-of-scope streaming SQL cases in the suite where practical to verify stock fallback, including unsupported formats and updating inputs before keyed admission. Record whether each case expects native execution, stock fallback, or the host's normal rejection; do not weaken assertions or silently drop failing parameter combinations to obtain a passing suite. Do not expand the selected suite into batch-only overwrite/partition-replacement or maintenance tests. Inventory the streaming cases and preserve their upstream runtime mode.
- Maintain an explicit inventory of selected cases/parameters, remaining upstream coverage, and justified exclusions such as external infrastructure. Retain exact Surefire reports and diagnostics. Grow the required CI selection alongside each admitted capability rather than postponing upstream integration until the sink is complete.
Validation and completion criteria
-
The dynamic follow-on checkpoint preserves Arrow data-file payloads end to end, shuffles them as batches, reuses Rust data-file encoding, and keeps Java delete encoders and host table/schema/commit responsibilities, with incremental upstream coverage.
-
The keyed follow-on checkpoint accelerates only data-file encoding, proves Java equality/position delete writers remain in use, and preserves Java row-position and file identity bookkeeping across native buffering and rolling.
-
Iceberg upstream Flink SQL tests run through
bin/flink-suite.sh icebergand the required integration CI matrix, using pinned upstream tests and released production dependencies. -
Every admitted sink capability adds the corresponding upstream SQL cases in the same implementation change, with unchanged upstream assertions and explicit native-versus-fallback witnesses.
-
Published-artifact wiring proves the narrow appender hook without a broad sink rewrite or sibling-repository modifications.
-
Plan/runtime witnesses establish Arrow payload handoff and iceberg-rust Parquet encoding; no payload row transpose or per-row JNI write.
-
Stock Iceberg reads native files and native jobs append to stock-created tables with matching logical results and required schema/metadata semantics.
-
Fake-appender tests pin the 1,000-row check cadence; production tests cover tiny targets, batches spanning files, empty input, partial batches, checkpoint close, exact final sizes, and buffer reclamation.
-
Compatibility tests cover default/override precedence, unsupported-option fallback, physical schema, statistics/NaNs, nested/sliced data, timestamps, and C Data ownership/alignment.
-
Streaming recovery tests exercise failure before/after file finalization and checkpoint completion, with no missing/duplicate committed rows. Preserve normal branch/snapshot configuration.
-
Partitioned milestone additionally proves stock partition paths/transforms and multi-file selection correctness.
-
Release-mode Criterion calls production encoding/bridge paths across representative batch sizes, nulls, nested widths, and selections; fixture setup is outside timing.
-
End-to-end comparisons use identical input, query, resources, checkpoint interval, and table settings against stock Flink and prior StreamFusion behavior, with warmup/repeats and all production conversion/JNI/commit costs retained. Include row-fed and columnar-fed cases, checkpoint latency, allocation/retention, and sustained throughput. Keep stock as default if acceleration is not demonstrated.
-
Update
docs/connectors/iceberg.md, deployment instructions, benchmark inventory/results, and any necessary divergence record in the shipping changes; document exact admission and fallback reasons.
Checkpoint: streaming Iceberg Parquet source using released iceberg-rust
Add a source that retains the released Java connector for table discovery and streaming coordination while handing its planned file tasks directly to iceberg-rust for native Parquet reads into Arrow. This is a streaming source, including its configured initial snapshot phase followed by incremental discovery; it does not add a separate batch-execution source or a general update/delete changelog.
Released baseline and initial admission
Use published Iceberg Java 1.12.0/Flink 2.2 artifacts and the published iceberg Rust crate 0.10.1 as the verified starting point. Its Arrow/Parquet 58 dependencies match StreamFusion. Recheck releases during implementation, but do not depend on Git revisions, release candidates, forks, local Maven artifacts, or sibling repository builds.
Accelerate Parquet data files only. Start with verified unencrypted v1/v2 append-stream scans, supported schemas/projections, and no delete files; then extend supported task semantics incrementally below. Keep Avro/ORC data files and metadata-table/data-task readers on the stock source. Positional/equality delete files must also use a format supported by the selected native reader; initially admit their Parquet representations only. A table's default write format is insufficient to prove this: historical files and individual tasks can have other formats.
Use whitelist-first source admission for the supported table/scan/schema/configuration contract, and validate task metadata before native emission. Future snapshots can introduce new file features, so explicitly define how that contract is maintained rather than assuming current table properties describe every future task. Unsupported combinations must retain stock behavior; never discard tasks, omit deletes, or silently switch decoding/offset semantics after native output has begun. Any optional stock-task-to-Arrow fallback must be separately documented and tested and must not count as native read coverage. The admitted native path performs no Java payload-row traversal or row-to-Arrow reconstruction.
Retain Java discovery and source lifecycle
Java retains catalog/table loading, branch and starting-strategy validation, snapshot polling, manifest/file pruning, split planning and assignment, enumerator state, split serializers, watermark policy, and checkpoint/recovery coordination. Preserve the connector's exact configuration precedence, monitor interval, split sizing/lookback/open-cost settings, planning limits/failure handling, and supported startup modes. An initial snapshot scan inside a streaming job is within scope.
The narrow released source hook is IcebergSource.Builder.readerFunction(ReaderFunction<ArrowBatch>). It is public but deprecated in 1.12.0, so version-gate and test it. Build with .build() and Flink env.fromSource(...) using Arrow type information: Iceberg's convenience .buildStream() explicitly rejects custom reader functions. Use this hook to retain the normal IcebergSource enumerator, split assignment, reader lifecycle, and serializers, rather than copying them. Integrate through StreamFusion's optional connector/source planner extension. If the user selects the legacy source instead of FLIP-27, retain stock fallback until that API has its own proven adapter.
Task handoff and native execution
Java continuous enumerator / split planner
-> IcebergSourceSplit containing Java-planned FileScanTasks
-> explicit task-metadata translation over JNI
-> iceberg-rust ArrowReader over host-backed FileIO
-> Arrow RecordBatches through the C Data Interface
-> Flink source batch emission + emitted-position checkpoint update
-> StreamFusion columnar pipeline
Translate Java tasks into Rust FileScanTask values explicitly, following Comet's approach. Carry the actual file path, full size, assigned byte range, schema and projected field IDs, applicable delete descriptors/equality IDs, partition spec/data, name mapping, case sensitivity, and supported residual predicates. Include hidden columns needed for filtering/deletes while preserving the requested output schema. Derive any required file/partition/sequence metadata from the Java-planned task, not by replanning the table independently in Rust. Java and Rust task serialization formats are not interchangeable; 0.10.1 explicitly lacks generic serde support for some partition/spec/name-mapping fields. Share/cache common schema/spec descriptions where useful without losing per-task identity.
Use ArrowReaderBuilder and ArrowReader::read(FileScanTaskStream) to perform supported file-range selection, field-ID projection, schema mapping, partition-constant reconstruction, residual filtering, and delete application in Rust. The result is Arrow batches, not Java rows. Adapt timestamp bundles, constant representations, nested layouts, and other engine-specific Arrow details column-wise at the boundary where necessary. Preserve row counts for zero-column projections.
Expose Java Iceberg InputFile / seekable reads through a Rust Storage / FileRead adapter. Reuse the bounded positional-I/O bridge pattern from the Paimon native reader so Java owns filesystem plugins, credential resolution/refresh, custom FileIO behavior, and access to data and delete files. Do not create a second native catalog or object-store credential stack as a prerequisite. Handle concurrent positional reads safely; do not race a shared seek cursor. Async Rust workers must attach correctly to the JVM and never retain a thread-local JNI environment. Keep cancellation, stream closure, and Arrow release callbacks deterministic.
Streaming offsets, ordering, and backpressure
Start by feeding one file task at a time to the native reader with data-file concurrency 1. Comet can combine unordered task streams for a batch scan, but the Flink reader's checkpoint is a task/file ordinal plus a logical record offset; an untagged interleaving of files would invalidate it. Preserve task order and do not let an emitted Arrow batch span a position boundary that cannot be represented by the existing split state. More concurrency is optional later and requires explicit task tagging/reordering or a separately proven state contract.
Each RecordAndPosition<ArrowBatch> must carry the position after the emitted batch in the stock connector's logical record units. Do not use the default row iterator/batcher in a way that counts one batch as one row. Advance split state only when Flink emits the batch, never on native prefetch. Recovery reopens the same task and skips the already-emitted logical prefix, slicing the next Arrow batch as needed; changed batch sizes must not change restored results. Preserve the stock filtering stage/offset space, including delete filtering and source-level residuals. Absolute physical file row positions used for deletes are distinct from these logical recovery offsets.
Retain the existing bounded fetch handover and source mailbox behavior, split completion/events, watermark ordering/alignment, idle behavior, and wakeup/cancellation. Account for live Arrow batches, native decoder row groups, delete indexes, caches, and queued batches; a batch row count alone is not a memory bound. Ensure queued/cancelled/unconsumed batches and JNI handles release correctly. Adjust metrics to count logical rows/bytes rather than misreporting batch count as row count.
Configuration coverage and staged semantics
Most discovery/configuration can be inherited unchanged. Parquet encoding properties such as codec, dictionary/page layout, and row-group size are read from file metadata rather than translated from the table's writer options. Native support still depends on actual supported codecs/types/features in the selected released reader. Neither a self-describing file nor successful decoding proves all Iceberg/Flink semantics.
- Native append scan: prove the custom reader hook, Java FileIO bridge, correct task byte ranges, Arrow output, and checkpoint-compatible sequential task execution. Include the initial snapshot and later appended snapshots.
- Projection/schema/filter coverage: add verified nested/zero-column projections, field-ID/name mapping, partition constants, historical schema evolution/default values, physical type/timestamp conversions, and residuals. Preserve exact predicates that the source considers enforced; row-group pruning alone is not row filtering. Retain downstream native filters where appropriate, and fall back when the complete source semantics/offset contract cannot be preserved. Source limits and watermarks require explicit parity tests.
- Parquet position/equality deletes: use the released native reader's delete loading/filtering, including projected-away key columns, mixed deletes, file ranges, and physical-versus-logical offsets. Java supplies the authoritative applicable delete set; Rust does not replan applicability. This especially matters to a streaming source's initial snapshot. Preserve the Java incremental planner's append-snapshot behavior rather than claiming CDC or a continuously materialized view of arbitrary table updates.
- Future v3/deletion-vector reads: admit only after a published crate provides the necessary APIs and parity is proven. Current upstream support is real: iceberg-rust #3035 reads/applies Puffin deletion vectors and Comet #5853 wires them into its task handoff. It is in main/0.11 release candidates, not published crate 0.10.1, so it stays gated in the released-only implementation. When available, carry format, referenced data file, blob offset/length, and record count; distinguish vectors sharing one Puffin path by their full blob identity. Puffin sidecars needed for Parquet correctness are compatible with the Parquet-data-only scope. Sink-side v3 support does not imply released native-source v3 support.
Source tests and completion criteria
Add the corresponding upstream Iceberg streaming SQL/source/enumerator/split-reader tests to the existing required integration harness as each stage lands. Inventory the streaming cases from TestFlinkSourceSql, TestIcebergSourceSql, TestStreamScanSql, and the source-reader/enumerator tests against the pinned release, preserving SQL, fixtures, runtime mode, expected rows, assertions, and timeouts. Keep supported-versus-fallback coverage explicit and require native read witnesses.
- Stock Java discovery, split/enumerator serializers, and supported source options remain in use; no native table discovery/catalog implementation is introduced.
- Released dependencies only; native admission is Parquet-only and does not claim deletion-vector support on 0.10.1.
- Plan/runtime evidence proves direct native task reading into Arrow, with no Java payload-row traversal or row-to-Arrow transpose on admitted native reads.
- Stock/native results match for initial snapshot plus live appends, supported startup positions/branches, split ranges within files, multiple tasks sharing a file, empty files/splits, partitioned tables, projection/schema mapping, filters, limits, watermarks, and admitted delete combinations.
- Checkpoint/restart tests cover partial task consumption, changed batch sizes, post-filter logical offsets, split reassignment/rescaling, prefetched-but-unemitted batches, and cancellation. No missing/duplicated emitted rows or physical/logical position confusion.
- Host-backed FileIO tests include remote/range access and injected read failures, plus bounded memory/ownership checks for decoder buffers, delete indexes, Arrow batches, and C Data handles.
- Add release Criterion coverage for the production native read/task/Arrow-handoff boundary, and sustained stock-versus-native streaming comparisons with all JNI/I/O/filter/delete/checkpoint costs retained. Include columnar downstream execution and verify intended native operators execute.
- Update connector/deployment/source coverage and benchmark documentation in shipping changes; publish exact admitted settings and task semantics rather than declaring all table settings supported.
References: Comet task serialization, Comet native task execution, released Rust reader, released Rust task contract, released Flink source hook, Paimon host-backed native reader, upstream Rust DV support, and Comet DV handoff/tests. Comet's Git-pinned dependency is inspiration only; keep StreamFusion on published crates.
Final checkpoint: Iceberg-to-Iceberg Nexmark versus stock Flink
Add a separate end-to-end streaming Iceberg input → query → Iceberg output benchmark against the published stock Flink Iceberg connector. Reuse the headline README workload/settings and shared Nexmark harness; leave the existing Kafka headline benchmark intact. Also reuse the Fluss benchmark pattern for shared queries/keys and paired endpoints, preserving Iceberg streaming semantics.
- Match the headline matrix: the same deterministic 2M events, parallelism 4, all 23 executable queries (q0–q23 excluding q6), schemas, SQL, watermarks, UTC, source reuse, resource limits, and existing explicitly labeled expression variants. Compare stock heap versus native memory and stock RocksDB versus native RocksDB, each with mini-batch off/on; on means
allow-latency=2s,size=50000on both engines. Use one-second checkpoints, release builds with the headline allocator/settings, andstreamfusion.exchange.zeroCopyLocal=falseso normal Arrow IPC exchange costs remain included. Use identical hardware per pair and record it; compare ratios only within a host. - Four Iceberg partitions at both ends: every input and result table uses a deterministic partition spec with four partition destinations, identical between engines. Table partitions are distinct from tasks/files. Choose routing from existing input/result keys (for example a four-bucket transform); preserve SQL, grouping, and actual result-key semantics. For low-cardinality/scalar outputs, report occupied partitions rather than inventing rows to fill four destinations. For partitioned upserts, include partition source columns in equality fields and prove the routing is stable for each logical key; do not add a changing routing field that changes the result's identity.
- Updating outputs use identifier-field / PRIMARY KEY upserts: use the actual result keys already defined by the harness, with the normal Iceberg upsert sink and Java delete encoding. Append-only outputs use append sinks. Identifier fields differ from identity partition transforms; preserve streaming changelogs. Start with admitted format-v2 tables and the released native reader's supported delete formats; v3/deletion-vector measurements follow only when released dependencies support them.
- Fair storage and timing: build one equivalent partitioned input corpus outside timing and reuse identical snapshots, files, and layout for each pair; use fresh equivalent outputs per run. Pin the same catalog/FileIO/storage, partition/distribution settings, Parquet codec/level/page/row-group/dictionary properties, target file size, scan polling/splits, and resources. Kafka client knobs are inapplicable; document the Iceberg replacements. Time source discovery/read/decode, query and batch shuffle, JNI, data/delete file writes, checkpoints, and committed outputs. Define and prove a common finite-corpus drain/watermark protocol through the final committed sink checkpoint for the continuous source; cancellation or task completion alone is not successful completion. Do not switch the benchmark source into batch mode to obtain termination.
- Validate and publish: read committed outputs with stock Iceberg outside timing and compare final keyed state/append results using existing assertions and documented expression variants. Witness native source decoding, operators/exchange, and Rust data-file writing; label fallbacks/unsupported combinations explicitly. Reproduce headline sampling for comparable summary cells (memory: one measured run; disk: one warmup and best of two), retaining additional paired repeats and all observations to report variability. Publish per-query/backend/mode timings, throughput, speedups/geomeans, configurations/commands, and scan/checkpoint/commit/file/delete diagnostics. Include all production costs and distinguish startup from sustained behavior. Deliver benchmark code and documentation after partitioned source and keyed-sink admission, alongside upstream SQL coverage.
Related source-side direction: prefer paimon-rust for Paimon task processing
Use released paimon-rust readers/merges for native Paimon processing. Preserve Java discovery, split planning/assignment, FileIO, lifecycle, and checkpoint/recovery; pass task/schema/delete/merge metadata to Rust and emit Arrow.
Coverage is partial today: StreamFusion adapts paimon-rust Arrow comparison/selection for supported snapshot merges, while deletion vectors and historical schemas still use Java-reader-to-Arrow fallback. Admission excludes deletion files and mismatched file-schema IDs. Migrate these fallbacks where supported.
Verify published-API parity for deletion vectors, schema evolution, merge engines, changelog/RowKind, projection/filter, timestamps, FileIO, and ownership. Checkpoint emitted logical offsets only; preserve resume across batch sizes and prevent prefetch advancing state. Keep unsupported cases on the host path with documented coverage. Record release gaps and agree on scope before upstream/separate implementation work; no fork, Git dependency, or sibling build.
Track a separate follow-up with unchanged upstream streaming tests, native witnesses, merge/delete/schema/restart cases, and release benchmarks retaining discovery/JNI/merge/delete/checkpoint costs. This is not an Iceberg prerequisite; its agreed boundaries remain unchanged.
Sources of inspiration and reference implementations
- StreamFusion filesystem Parquet: encoding-only boundary and native encoder/byte bridge. Keep host-owned file lifecycle and bounded byte transfer.
- StreamFusion Paimon: format SPI and Arrow bundles. Arrow-backed row traversal can preserve host bookkeeping without transposing payloads.
- StreamFusion Delta: existing integration. Batch-preserving handoff, Java-owned output, and host interpretation of file metadata.
- Iceberg Java 1.12.0: DataWriter/FileAppender boundary, rolling policy, hardcoded task-writer factory, and format registry constraints. These define the compatibility and injection contracts.
- iceberg-rust 0.10.1: released crate, single-file Parquet writer, and OutputFile/FileWrite adapters. Reuse schema/IO/read APIs; retain byte-parity tests against its writer. Direct parquet-rs encoding exposes retention accounting.
- DataFusion Comet: local-file and buffered Parquet output, Arrow ownership/alignment, and Iceberg write/commit separation and configuration admission. The inspected Comet Iceberg path still writes through iceberg-java; it is architectural inspiration, not proof of a completed iceberg-rust JNI sink. Its Git-pinned dependencies and native rolling/path differences are not dependencies or behavior we need to adopt.
Final benchmark results are published: complete Iceberg input/output matrix, fresh Kafka→Iceberg, and matched Kafka→Paimon comparison. PR #386 contains the implementation, SQL/upstream recovery coverage and all measured limits; current supported/fallback contracts are recorded in the connector documentation. Routing and state/checkpoint optimization follow-ups remain in #387 and #391. Merged in #386 after all required GitHub checks passed.
Activity
[-]Add an Arrow-preserving append-only Iceberg sink with iceberg-rust Parquet encoding[/-][+]Add Arrow-preserving Iceberg streaming sinks and a native Parquet source[/+]Keep dynamic Iceberg payloads columnar around Java sink control
Account for retained Iceberg Parquet encoder memory
Measure Iceberg streaming completion with equal memory and watermark …
Keep Iceberg streaming commits in Java around Arrow file encoding