Skip to content

[EPIC] Bin-pack compaction support #3702

Open

Description

@comphead

What's the feature are you trying to implement?

Bin-pack data file compaction, the default strategy of iceberg-java's rewrite_data_files. It selects data files outside the target size range or with too many deletes, packs them into groups, rewrites each group into files of the target size, and commits a replace snapshot. The goal is parity with iceberg-java's BinPackRewriteFilePlanner and its rewrite commit.

Tasks

Related

Willingness to contribute

I can contribute to this feature independently

Activity

  1. self-assigned this
    on Nov 16, 2026
  2. comphead commented on Nov 17, 2026

    @comphead
    CollaboratorAuthor

    We experimented with binpack in apache/datafusion-comet#7282 which gives very optimistic insight.
    The idea is to implement binpack using iceberg-rs building block, scan and writer and binpack implemented in Comet. Comet also used to trigger.

    ┌────────────┬──────────────────────────────────────────────────┬────────────────────────────────────────┐
    │            │                 spark-0868413a…                  │            spark-9b825fe9…             │
    ├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
    │ Case       │ comet-iceberg                                    │ spark (no Comet plugin)                │
    ├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
    │ Time (PST) │ 11:53 to 12:07                                   │ 12:34 to 14:13                         │
    ├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
    │ App wall   │ 809 s                                            │ 5,912 s                                │
    ├────────────┼──────────────────────────────────────────────────┼────────────────────────────────────────┤
    │ Executors  │ 4 x 12 slots, 4-CPU limit, 8g heap, 32g off-heap │ same, no off-heap                      │
    └────────────┴──────────────────────────────────────────────────┴────────────────────────────────────────┘
    

    Memory (peaks sampled at heartbeats)

    
    ┌────────────────────────┬─────────────────────────┬──────────────────┐
    │                        │     Comet executors     │ Spark executors  │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ Heap used              │ 4.6 to 4.9 GiB of 8 GiB │ 5.3 to 5.4 GiB   │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ RSS                    │ 6.65 to 6.73 GiB        │ 6.56 to 6.65 GiB │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ Off-heap execution     │ 27 to 39 MiB of 32 GiB  │ none             │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ Comet native allocated │ 82 to 118 MiB           │ n/a              │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ Executor GC, whole app │ 0.16 to 0.27 s          │ 2.0 to 3.7 s     │
    ├────────────────────────┼─────────────────────────┼──────────────────┤
    │ Pod memory             │ 49.2 GiB                │ 17.2 GiB         │
    └────────────────────────┴─────────────────────────┴──────────────────┘
    
  3. blackmwk commented on Nov 17, 2026

    @blackmwk
    Contributor

    IIUC, I think datafusion-comet uses iceberg-rust for data read/write task? It still relies on java library for tx actions?

  4. comphead commented on Nov 17, 2026

    @comphead
    CollaboratorAuthor

    IIUC, I think datafusion-comet uses iceberg-rust for data read/write task? It still relies on java library for tx actions?

    yeah, we built the PoC to be as close to java implementation as possible, just replacing IO heavy parts to iceberg-rs, let me add the diagram that should clarify things.

    Spark vs comet-iceberg

    Both runs use the same iceberg-java driver code: planning, bin-pack, a job per group, and the commit. Comet only replaces the scan and the write with iceberg-rust (highlighted). Numbers are from the 2026-11-16 runs, averaged over the timed CALLs.

    flowchart TB
        plan["Driver, iceberg-java<br/>bin-pack 24,000 files into 150 groups<br/>1 job per group, 5 at a time"]
    
        subgraph SPARK ["Spark task, ~45 s per group"]
            s1["Iceberg Parquet reader, JVM<br/>1 file at a time"] --> s2["ColumnarToRow"] --> s3["Iceberg Parquet writer, JVM"]
        end
    
        subgraph COMET ["Comet task, ~3 s per group"]
            c1["CometIcebergNativeScan<br/>iceberg-rust, 4 files at a time"] --> c2["CometIcebergWrite<br/>iceberg-rust Parquet writer"]
        end
    
        commit["Driver, iceberg-java<br/>RewriteFiles commit after all groups<br/>remove 24,000 files, add 150 of ~13.7 MiB"]
    
        plan --> s1 & c1
        s3 & c2 --> commit
    
        classDef rust fill:#ffe9b3,stroke:#d97706,color:#000
        class c1,c2 rust
    
    Loading

    One file group:

    sequenceDiagram
        participant D as Driver
        participant T as Executor task
        participant S as S3
        D->>T: 1 group, 160 files, ~35 MiB
        alt Spark, ~45 s, 13% CPU
            loop 160 files, 1 at a time
                T->>S: read via S3A, ~250 ms off-CPU each
            end
            T->>T: ColumnarToRow, JVM Parquet writer
        else Comet, ~3 s
            loop 160 files, 4 at a time
                T->>S: read via iceberg-rust
            end
            T->>T: Arrow batches to iceberg-rust writer
        end
        T->>S: write 1 file, ~13.7 MiB
        T-->>D: new data file
        Note over D,S: 150 groups per CALL, Spark ~1,381 s, Comet ~97 s
    
    Loading
  5. nevzheng commented on Nov 17, 2026

    @nevzheng

    TL;DR: excited about this! Before diving into PR reviews, could we align on the end goal with a short alignment doc and a Rust PoC? Details and questions below.

    1. praise: thanks for driving this, @comphead. The diagrams are clear and 809s vs. 5,912s is compelling. Happy to help review.

    2. suggestion: could we start with a short alignment doc? Ideally it has use cases, the end-to-end flow and the PR order, plus a full Rust PoC with benchmark data. That would help reviewers and committers see the end goal and commit to the work. Spark's SPIP template could be a useful starting point.

    3. question: how do this epic and the Comet PoC (Iceberg bin pack compaction test datafusion-comet#7282) relate? The PoC runs planning, bin-packing and the commit in Scala/Java and uses iceberg-rust only for the existing scan + write, so none of this epic's Rust PRs run in it yet.

      • 3a. What's the goal, and what needs to change in iceberg-rust to get there?
      • 3b. Grouping the issues into one epic helps, but I don't see how the pieces fit into a working Rust compaction path. Would a PoC or the alignment doc above help show that? The SPIP questions could be a good guide.
      • 3c. Can non-Comet engines take advantage of this, and is that in scope?
    4. question: which PRs are ready for review now?

    5. suggestion: would you be able to join an upcoming Rust community sync and present this? A walkthrough would be a great way to get more eyes on it.

    6. thought: this could fit into the Q4 roadmap ([DISCUSSION] 2026 Q4 Roadmap #3680), wdyt? cc @mbutrovich @liurenjie1024 @dannycjones

    Warmly,
    Nevin (🤖 claude assisted)

  6. dannycjones commented on Nov 17, 2026

    @dannycjones
    Contributor

    What I'd really like to see is an incremental approach/proposal here.

    In particular: if we already have scans and writes in Rust, then what are the missing bits and do they all need to be implemented to benefit or only bits? What is the expected improvement of each bit?

    Compaction is one of those things where people implement it in different ways and the Spark-based compaction procedure is just one implementation. Are we wanting to replicate that same algorithm or do something different? Does implementing the same algorithm actually get us much improvement over what can be done with today's Comet plugin? (I'm pulling a bit on what's mentioned in Nevin's and Renjie's responses here I think.)

    This is a bit of a dump of my initial thoughts, I'd love to come back to this and dig a bit deeper. In terms of roadmap, I think answers to above will help formulate where that sits among the community.

  7. blackmwk commented on Nov 19, 2026

    @blackmwk
    Contributor

    Hi, @comphead What I'm curious about is why datafusion-comet requires the features mentioned in this issue? Your diagram shows that it only uses iceberg-rust for read/write

  8. comphead commented on Nov 19, 2026

    @comphead
    CollaboratorAuthor

    Hi, @comphead What I'm curious about is why datafusion-comet requires the features mentioned in this issue? Your diagram shows that it only uses iceberg-rust for read/write

    The diagram shows how the PoC works right now using Comet but relying mostly on iceberg-rust blocks, as there is minor actual Comet work in PoC rather delegate parts to iceberg-rust which does the heavy work.

    The idea of PoC was to tie existing pieces together, implement missing parts on Comet side to have E2E run and collect metrics. The idea of the current epic is to implement those missing parts in iceberg-rust so Comet and other consumers would reuse compaction fully from this repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions