Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,11 @@ jobs:
org.apache.comet.CometIcebergResidualPushdownSuite
org.apache.comet.iceberg.IcebergReflectionSuite
org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite
org.apache.comet.CometIcebergCompactionSuite
org.apache.comet.iceberg.compaction.BinPackPlannerSuite
org.apache.comet.iceberg.compaction.CompactionSourceSuite
org.apache.comet.iceberg.compaction.CompactionScanIcebergSuite
org.apache.comet.iceberg.compaction.CompactionOptionsSuite
org.apache.comet.csv.CometCsvNativeReadSuite
org.apache.comet.text.CometTextNativeReadSuite
org.apache.comet.CometFuzzTestSuite
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/pr_build_macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,11 @@ jobs:
org.apache.comet.CometIcebergResidualPushdownSuite
org.apache.comet.iceberg.IcebergReflectionSuite
org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite
org.apache.comet.CometIcebergCompactionSuite
org.apache.comet.iceberg.compaction.BinPackPlannerSuite
org.apache.comet.iceberg.compaction.CompactionSourceSuite
org.apache.comet.iceberg.compaction.CompactionScanIcebergSuite
org.apache.comet.iceberg.compaction.CompactionOptionsSuite
org.apache.comet.csv.CometCsvNativeReadSuite
org.apache.comet.text.CometTextNativeReadSuite
org.apache.comet.CometFuzzTestSuite
Expand Down
213 changes: 213 additions & 0 deletions docs/source/contributor-guide/iceberg-compaction.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/source/contributor-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ JVM Shuffle <jvm_shuffle>
Native Shuffle <native_shuffle>
Memory Management <memory_management>
Iceberg Writes <iceberg-writes>
Iceberg Compaction <iceberg-compaction>
ANSI Error Propagation <sql_error_propagation>
S3 Credential Provider Design <s3-credential-provider-design>
```
Expand Down
95 changes: 94 additions & 1 deletion docs/source/user-guide/latest/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,11 @@ Comet accelerates Iceberg tables whose data files are Parquet, in two ways:
files in Rust. It is enabled by default.
- **[Writing](#writes)**: with `spark.comet.write.iceberg.enabled=true`, Comet writes the data
files of eligible writes natively. It is disabled by default.
- **[Compaction](#compaction)**: with `spark.comet.iceberg.compaction.enabled=true`, Comet runs
the `rewrite_data_files` procedure's bin-pack rewrite itself, on Spark 4.0 and later. It is
disabled by default.

This page covers both, followed by the [known issues](#known-issues). Behavior that applies to
This page covers all three, followed by the [known issues](#known-issues). Behavior that applies to
Parquet files in general is documented on the [Parquet](parquet.md) page.

## Native Reader
Expand Down Expand Up @@ -888,6 +891,96 @@ must match iceberg-java exactly, or the write falls back. The open bugs in which
still differs are listed in
[Known Issues](#writes-that-differ-from-spark).

## Compaction

On Spark 4.0 and later, Comet can run Iceberg's `rewrite_data_files` procedure with the `binpack`
strategy itself. It is disabled by default. Set `spark.comet.iceberg.compaction.enabled=true` to
turn it on, and `spark.comet.write.iceberg.enabled=true` as well to have the new files written by
the [native writer](#writes).

```sql
CALL catalog.system.rewrite_data_files(
table => 'db.events',
where => 'event_date = "2026-10-27"',
options => map('min-input-files', '10', 'partial-progress.enabled', 'true'))
```

The call, its arguments, its options and its result row are Iceberg's. Comet follows the steps of
Iceberg's own `RewriteDataFilesSparkAction`, so a call plans the same file groups with either
engine.

### How Comet runs a rewrite

1. **Plan.** Comet lists the data files of the table's current snapshot that match `where`, and
groups them the way Iceberg's `BinPackRewriteFilePlanner` does. A file is a candidate when its
size is outside `min-file-size-bytes` and `max-file-size-bytes`, or when it has at least
`delete-file-threshold` delete files, or when `delete-ratio-threshold` of its rows are deleted.
Candidates are packed within each partition into groups of up to `max-file-group-size-bytes`.
A group is kept when it has several files and at least `min-input-files` of them, when it has
several files and more than the target size, when it holds more than `max-file-size-bytes`, or
when one of its files qualifies for its deletes.
2. **Rewrite.** Comet reads the files of a group with the [native Iceberg reader](#native-reader),
in splits sized so that each becomes about one new file, and writes them back with the
[native writer](#writes) when it is eligible and Iceberg's writer otherwise. Deletes are
applied while reading, so the new files do not contain deleted rows. Iceberg rewrites each
group in a Spark job of its own, `max-concurrent-file-group-rewrites` jobs at a time, so a table
with many partitions has many small jobs and most of the cluster is idle. Comet rewrites all the
groups of a commit in one job, in which every Spark task reads one split and writes about one
file. A job covers at most 10,000 splits. A group is rewritten in a job of its own, as Iceberg
does it, when the native reader cannot read it, when its files are not in the output
partition spec, when the table is format version 3, or when
`spark.comet.iceberg.compaction.singleJob.enabled` is `false`.
3. **Commit.** The groups are committed as one `replace` snapshot that is validated against the
snapshot the plan read, so a concurrent change to a rewritten file fails the commit instead of
losing it. With `partial-progress.enabled`, groups are committed in up to
`partial-progress.max-commits` snapshots as they finish, and a job covers the groups of one
commit. If a commit fails, the files it wrote are deleted.

### Supported calls

- `strategy` is `binpack` or not given. `sort` and `zorder` run Iceberg's procedure.
- `sort_order` is not supported. A call that has one runs Iceberg's procedure.
- `where` is supported and converted by Iceberg.
- These options are supported, with Iceberg's defaults: `target-file-size-bytes`,
`min-file-size-bytes`, `max-file-size-bytes`, `min-input-files`, `rewrite-all`,
`max-file-group-size-bytes`, `delete-file-threshold`, `delete-ratio-threshold`,
`rewrite-job-order`, `max-files-to-rewrite`, `max-concurrent-file-group-rewrites`,
`partial-progress.enabled`, `partial-progress.max-commits`,
`partial-progress.max-failed-commits` and `use-starting-sequence-number`.
- `output-spec-id` is supported for the table's current spec only.
- `remove-dangling-deletes` is supported when it is `false`.

### When Comet leaves a call to Iceberg

Comet runs Iceberg's own procedure, with the call unchanged, when the call has something Comet does
not reproduce. That covers the strategies and options listed above, an option Comet does not know,
a value Iceberg would reject, a `where` that does not convert, and a table that is not an Iceberg
table. Running Iceberg's procedure is also how an invalid call gets Iceberg's error message. The
check happens before anything is written, so a call that Comet declines is never half done. The
driver logs the reason at `INFO`.

A failure after the rewrite has started is the call's failure, as it is with Iceberg.

### Telling which engine ran

The jobs of a rewrite that Comet ran have the job group `COMET-REWRITE-DATA-FILES-<n>` in the
Spark UI, and Iceberg's have `REWRITE-DATA-FILES-<n>`.

### Differences from Iceberg

- A Spark job covers several file groups, so a task failure fails all of them. Without partial
progress the rewrite fails as it does in Iceberg when any group fails. With partial progress, a
job covers the groups of one commit, so a failure costs that commit. The files the job wrote are
deleted, and the rest of the rewrite goes on.
- `max-concurrent-file-group-rewrites` counts Spark jobs, which cover several groups each.
- Groups are started in the order the table's scan returned their partitions when
`rewrite-job-order` is `none`. Iceberg's order depends on how it hashes partitions. The other job
orders sort the same way.
- Comet commits one batch of groups at a time in `partial-progress` mode. Iceberg commits from
the thread that finished the group, so batches can commit at the same time.
- Spark 3.x is not supported yet. There, Iceberg's SQL extensions plan their own `CALL` node, which
Comet does not intercept.

## Known Issues

This section lists open issues in which Comet's Iceberg reads and writes give a different result
Expand Down
28 changes: 28 additions & 0 deletions spark/src/main/scala/org/apache/comet/CometConf.scala
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,34 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)

val COMET_ICEBERG_COMPACTION_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.iceberg.compaction.enabled")
.category(CATEGORY_EXEC)
.doc(
"Whether Comet runs the Iceberg `rewrite_data_files` procedure with the `binpack` " +
"strategy itself. Comet plans the file groups the way Iceberg does, rewrites each " +
"group through its native Iceberg scan and, when `spark.comet.write.iceberg.enabled` " +
"is also true, its native Iceberg writer, and commits the result with the same " +
"options and defaults as Iceberg. A call Comet cannot reproduce, such as a sort or " +
"z-order strategy, an unsupported option or a non-Iceberg table, runs Iceberg's own " +
"procedure instead. Requires Spark 4.0 or later.")
.booleanConf
.createWithDefault(false)

val COMET_ICEBERG_COMPACTION_SINGLE_JOB_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.iceberg.compaction.singleJob.enabled")
.category(CATEGORY_EXEC)
.doc(
"Whether a compaction run by `spark.comet.iceberg.compaction.enabled` rewrites the file " +
"groups of each commit in one Spark job, in which every Spark task reads and rewrites " +
"about one output file of data. Iceberg runs one job per file group, " +
"`max-concurrent-file-group-rewrites` of them at a time, so a table with many small " +
"groups leaves most of the cluster idle. When false, or when Comet's native Iceberg " +
"scan cannot read a group, each group is rewritten in a job of its own as Iceberg " +
"does it.")
.booleanConf
.createWithDefault(true)

val COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT: ConfigEntry[Int] =
conf("spark.comet.scan.icebergNative.dataFileConcurrencyLimit")
.category(CATEGORY_SCAN)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import org.apache.spark.sql.internal.SQLConf

import org.apache.comet.CometConf._
import org.apache.comet.iceberg.IcebergWriteStrategy
import org.apache.comet.iceberg.compaction.CometIcebergProcedureRule
import org.apache.comet.rules.{CometCoalesceShufflePartitions, CometPlanAdaptiveDynamicPruningFilters, CometReuseSubquery, CometRule, CometSpark34AqeDppFallbackRule}
import org.apache.comet.shims.ShimCometSparkSessionExtensions

Expand Down Expand Up @@ -116,6 +117,9 @@ class CometSparkSessionExtensions
injectQueryStageOptimizerRuleShim(extensions, CometReuseSubquery)
injectQueryStageOptimizerRuleShim(extensions, CometCoalesceShufflePartitions)
extensions.injectPlannerStrategy { session => IcebergWriteStrategy(session) }
// Spark 4 invokes a CALL's procedure while it analyzes the statement, after optimizing it, so
// an optimizer rule is where Comet's rewrite_data_files replaces Iceberg's.
extensions.injectOptimizerRule { session => CometIcebergProcedureRule(session) }
}

case class CometColumnar(session: SparkSession) extends ColumnarRule {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ object IcebergReflection extends Logging {
val UNBOUND_PREDICATE = "org.apache.iceberg.expressions.UnboundPredicate"
val SPARK_BATCH_QUERY_SCAN = "org.apache.iceberg.spark.source.SparkBatchQueryScan"
val SPARK_STAGED_SCAN = "org.apache.iceberg.spark.source.SparkStagedScan"

// Comet's own scan of a compaction job's bins, which exposes the accessors Comet reads off
// an Iceberg scan.
val COMPACTION_SCAN = "org.apache.comet.iceberg.compaction.CompactionScan"
val SPARK_SCHEMA_UTIL = "org.apache.iceberg.spark.SparkSchemaUtil"
val TABLE = "org.apache.iceberg.Table"
val DEFAULT_LOCATION_PROVIDER =
Expand Down Expand Up @@ -85,9 +89,15 @@ object IcebergReflection extends Logging {
* `SparkStagedScan` also backs reads against Iceberg metadata tables (e.g. `POSITION_DELETES`),
* but the gate for that lives in `getMetadataLocation`, which returns None for metadata-table
* instances.
*
* `CompactionScan` is Comet's own scan of the bins of a compaction job. It is not an Iceberg
* class, but it exposes the accessors Comet reads off an Iceberg scan.
*/
val ICEBERG_SCAN_CLASSES: Set[String] =
Set(ClassNames.SPARK_BATCH_QUERY_SCAN, ClassNames.SPARK_STAGED_SCAN)
Set(
ClassNames.SPARK_BATCH_QUERY_SCAN,
ClassNames.SPARK_STAGED_SCAN,
ClassNames.COMPACTION_SCAN)

def isIcebergScanClass(name: String): Boolean = ICEBERG_SCAN_CLASSES.contains(name)

Expand Down
Loading