You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bin-pack data file compaction, the default strategy of iceberg-java's rewrite_data_files. It selects data files outside the target size range or with too many deletes, packs them into groups, rewrites each group into files of the target size, and commits a replace snapshot. The goal is parity with iceberg-java's BinPackRewriteFilePlanner and its rewrite commit.
Bin-pack planner (no ticket yet): select files and pack them into groups, with iceberg-java's options and defaults
Rewrite runner (no ticket yet): read each group with deletes applied, write files at the target size, and commit in one commit or in batches with partial progress
We experimented with binpack in apache/datafusion-comet#7282 which gives very optimistic insight.
The idea is to implement binpack using iceberg-rs building block, scan and writer and binpack implemented in Comet. Comet also used to trigger.
┌────────────┬──────────────────────────────────────────────────┬────────────────────────────────────────┐
│ │ spark-0868413a… │ spark-9b825fe9… │
├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
│ Case │ comet-iceberg │ spark (no Comet plugin) │
├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
│ Time (PST) │ 11:53 to 12:07 │ 12:34 to 14:13 │
├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
│ App wall │ 809 s │ 5,912 s │
├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
│ Executors │ 4 x 12 slots, 4-CPU limit, 8g heap, 32g off-heap │ same, no off-heap │
└────────────┴──────────────────────────────────────────────────┴────────────────────────────────────────┘
Memory (peaks sampled at heartbeats)
┌────────────────────────┬─────────────────────────┬──────────────────┐
│ │ Comet executors │ Spark executors │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ Heap used │ 4.6 to 4.9 GiB of 8 GiB │ 5.3 to 5.4 GiB │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ RSS │ 6.65 to 6.73 GiB │ 6.56 to 6.65 GiB │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ Off-heap execution │ 27 to 39 MiB of 32 GiB │ none │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ Comet native allocated │ 82 to 118 MiB │ n/a │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ Executor GC, whole app │ 0.16 to 0.27 s │ 2.0 to 3.7 s │
├────────────────────────┼─────────────────────────┼──────────────────┤
│ Pod memory │ 49.2 GiB │ 17.2 GiB │
└────────────────────────┴─────────────────────────┴──────────────────┘
IIUC, I think datafusion-comet uses iceberg-rust for data read/write task? It still relies on java library for tx actions?
yeah, we built the PoC to be as close to java implementation as possible, just replacing IO heavy parts to iceberg-rs, let me add the diagram that should clarify things.
Spark vs comet-iceberg
Both runs use the same iceberg-java driver code: planning, bin-pack, a job per group, and the commit. Comet only replaces the scan and the write with iceberg-rust (highlighted). Numbers are from the 2026-11-16 runs, averaged over the timed CALLs.
flowchart TB
plan["Driver, iceberg-java<br/>bin-pack 24,000 files into 150 groups<br/>1 job per group, 5 at a time"]
subgraph SPARK ["Spark task, ~45 s per group"]
s1["Iceberg Parquet reader, JVM<br/>1 file at a time"] --> s2["ColumnarToRow"] --> s3["Iceberg Parquet writer, JVM"]
end
subgraph COMET ["Comet task, ~3 s per group"]
c1["CometIcebergNativeScan<br/>iceberg-rust, 4 files at a time"] --> c2["CometIcebergWrite<br/>iceberg-rust Parquet writer"]
end
commit["Driver, iceberg-java<br/>RewriteFiles commit after all groups<br/>remove 24,000 files, add 150 of ~13.7 MiB"]
plan --> s1 & c1
s3 & c2 --> commit
classDef rust fill:#ffe9b3,stroke:#d97706,color:#000
class c1,c2 rust
Loading
One file group:
sequenceDiagram
participant D as Driver
participant T as Executor task
participant S as S3
D->>T: 1 group, 160 files, ~35 MiB
alt Spark, ~45 s, 13% CPU
loop 160 files, 1 at a time
T->>S: read via S3A, ~250 ms off-CPU each
end
T->>T: ColumnarToRow, JVM Parquet writer
else Comet, ~3 s
loop 160 files, 4 at a time
T->>S: read via iceberg-rust
end
T->>T: Arrow batches to iceberg-rust writer
end
T->>S: write 1 file, ~13.7 MiB
T-->>D: new data file
Note over D,S: 150 groups per CALL, Spark ~1,381 s, Comet ~97 s
TL;DR: excited about this! Before diving into PR reviews, could we align on the end goal with a short alignment doc and a Rust PoC? Details and questions below.
praise: thanks for driving this, @comphead. The diagrams are clear and 809s vs. 5,912s is compelling. Happy to help review.
suggestion: could we start with a short alignment doc? Ideally it has use cases, the end-to-end flow and the PR order, plus a full Rust PoC with benchmark data. That would help reviewers and committers see the end goal and commit to the work. Spark's SPIP template could be a useful starting point.
question: how do this epic and the Comet PoC (Iceberg bin pack compaction test datafusion-comet#7282) relate? The PoC runs planning, bin-packing and the commit in Scala/Java and uses iceberg-rust only for the existing scan + write, so none of this epic's Rust PRs run in it yet.
3a. What's the goal, and what needs to change in iceberg-rust to get there?
3b. Grouping the issues into one epic helps, but I don't see how the pieces fit into a working Rust compaction path. Would a PoC or the alignment doc above help show that? The SPIP questions could be a good guide.
3c. Can non-Comet engines take advantage of this, and is that in scope?
question: which PRs are ready for review now?
suggestion: would you be able to join an upcoming Rust community sync and present this? A walkthrough would be a great way to get more eyes on it.
What I'd really like to see is an incremental approach/proposal here.
In particular: if we already have scans and writes in Rust, then what are the missing bits and do they all need to be implemented to benefit or only bits? What is the expected improvement of each bit?
Compaction is one of those things where people implement it in different ways and the Spark-based compaction procedure is just one implementation. Are we wanting to replicate that same algorithm or do something different? Does implementing the same algorithm actually get us much improvement over what can be done with today's Comet plugin? (I'm pulling a bit on what's mentioned in Nevin's and Renjie's responses here I think.)
This is a bit of a dump of my initial thoughts, I'd love to come back to this and dig a bit deeper. In terms of roadmap, I think answers to above will help formulate where that sits among the community.
Hi, @comphead What I'm curious about is why datafusion-comet requires the features mentioned in this issue? Your diagram shows that it only uses iceberg-rust for read/write
Hi, @comphead What I'm curious about is why datafusion-comet requires the features mentioned in this issue? Your diagram shows that it only uses iceberg-rust for read/write
The diagram shows how the PoC works right now using Comet but relying mostly on iceberg-rust blocks, as there is minor actual Comet work in PoC rather delegate parts to iceberg-rust which does the heavy work.
The idea of PoC was to tie existing pieces together, implement missing parts on Comet side to have E2E run and collect metrics. The idea of the current epic is to implement those missing parts in iceberg-rust so Comet and other consumers would reuse compaction fully from this repo
Activity
comphead commentedon Nov 17, 2026
We experimented with binpack in apache/datafusion-comet#7282 which gives very optimistic insight.
The idea is to implement binpack using
iceberg-rsbuilding block, scan and writer and binpack implemented in Comet. Comet also used to trigger.Memory (peaks sampled at heartbeats)
blackmwk commentedon Nov 17, 2026
IIUC, I think datafusion-comet uses iceberg-rust for data read/write task? It still relies on java library for tx actions?
comphead commentedon Nov 17, 2026
yeah, we built the PoC to be as close to java implementation as possible, just replacing IO heavy parts to
iceberg-rs, let me add the diagram that should clarify things.Spark vs comet-iceberg
Both runs use the same iceberg-java driver code: planning, bin-pack, a job per group, and the commit. Comet only replaces the scan and the write with iceberg-rust (highlighted). Numbers are from the 2026-11-16 runs, averaged over the timed CALLs.
flowchart TB plan["Driver, iceberg-java<br/>bin-pack 24,000 files into 150 groups<br/>1 job per group, 5 at a time"] subgraph SPARK ["Spark task, ~45 s per group"] s1["Iceberg Parquet reader, JVM<br/>1 file at a time"] --> s2["ColumnarToRow"] --> s3["Iceberg Parquet writer, JVM"] end subgraph COMET ["Comet task, ~3 s per group"] c1["CometIcebergNativeScan<br/>iceberg-rust, 4 files at a time"] --> c2["CometIcebergWrite<br/>iceberg-rust Parquet writer"] end commit["Driver, iceberg-java<br/>RewriteFiles commit after all groups<br/>remove 24,000 files, add 150 of ~13.7 MiB"] plan --> s1 & c1 s3 & c2 --> commit classDef rust fill:#ffe9b3,stroke:#d97706,color:#000 class c1,c2 rustOne file group:
sequenceDiagram participant D as Driver participant T as Executor task participant S as S3 D->>T: 1 group, 160 files, ~35 MiB alt Spark, ~45 s, 13% CPU loop 160 files, 1 at a time T->>S: read via S3A, ~250 ms off-CPU each end T->>T: ColumnarToRow, JVM Parquet writer else Comet, ~3 s loop 160 files, 4 at a time T->>S: read via iceberg-rust end T->>T: Arrow batches to iceberg-rust writer end T->>S: write 1 file, ~13.7 MiB T-->>D: new data file Note over D,S: 150 groups per CALL, Spark ~1,381 s, Comet ~97 snevzheng commentedon Nov 17, 2026
TL;DR: excited about this! Before diving into PR reviews, could we align on the end goal with a short alignment doc and a Rust PoC? Details and questions below.
praise: thanks for driving this, @comphead. The diagrams are clear and 809s vs. 5,912s is compelling. Happy to help review.
suggestion: could we start with a short alignment doc? Ideally it has use cases, the end-to-end flow and the PR order, plus a full Rust PoC with benchmark data. That would help reviewers and committers see the end goal and commit to the work. Spark's SPIP template could be a useful starting point.
question: how do this epic and the Comet PoC (Iceberg bin pack compaction test datafusion-comet#7282) relate? The PoC runs planning, bin-packing and the commit in Scala/Java and uses iceberg-rust only for the existing scan + write, so none of this epic's Rust PRs run in it yet.
question: which PRs are ready for review now?
suggestion: would you be able to join an upcoming Rust community sync and present this? A walkthrough would be a great way to get more eyes on it.
thought: this could fit into the Q4 roadmap ([DISCUSSION] 2026 Q4 Roadmap #3680), wdyt? cc @mbutrovich @liurenjie1024 @dannycjones
Warmly,
Nevin (🤖 claude assisted)
dannycjones commentedon Nov 17, 2026
What I'd really like to see is an incremental approach/proposal here.
In particular: if we already have scans and writes in Rust, then what are the missing bits and do they all need to be implemented to benefit or only bits? What is the expected improvement of each bit?
Compaction is one of those things where people implement it in different ways and the Spark-based compaction procedure is just one implementation. Are we wanting to replicate that same algorithm or do something different? Does implementing the same algorithm actually get us much improvement over what can be done with today's Comet plugin? (I'm pulling a bit on what's mentioned in Nevin's and Renjie's responses here I think.)
This is a bit of a dump of my initial thoughts, I'd love to come back to this and dig a bit deeper. In terms of roadmap, I think answers to above will help formulate where that sits among the community.
blackmwk commentedon Nov 19, 2026
Hi, @comphead What I'm curious about is why datafusion-comet requires the features mentioned in this issue? Your diagram shows that it only uses iceberg-rust for read/write
comphead commentedon Nov 19, 2026
The diagram shows how the PoC works right now using Comet but relying mostly on
iceberg-rustblocks, as there is minor actual Comet work in PoC rather delegate parts toiceberg-rustwhich does the heavy work.The idea of PoC was to tie existing pieces together, implement missing parts on Comet side to have E2E run and collect metrics. The idea of the current epic is to implement those missing parts in
iceberg-rustso Comet and other consumers would reuse compaction fully from this repo