Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .ai/skills/review-comet-iceberg-write-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ admits. The two need different evidence.
writes it previously refused. "The tests pass" is not evidence here, because the existing tests
were written against the narrower gate. Ask for a test that writes the newly admitted
configuration through both writers and compares the result, and for the user guide's eligibility
table to change in the same PR.
table and its "Plans for each fallback" lists to change in the same PR.

For every gate change:

Expand Down Expand Up @@ -245,7 +245,7 @@ Ask specifically:
steps after them, the cleanup ownership table, the test table, and the Iceberg version per Spark
profile
- **`user-guide/latest/iceberg.md`, Writes section**: the eligibility table, the "falls back" lists, the
failure-handling section, and the accepted-divergences list. A PR that lifts or adds a rule, or
"Plans for each fallback" lists, the failure-handling section, and the accepted-divergences list. A PR that lifts or adds a rule, or
changes a divergence, must update these. A divergence the code has since fixed must be removed.
- **`user-guide/latest/iceberg.md`, Native Reader section**: its unsupported-features list mentions writes. The
Known Issues section lists the open write issues
Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributor-guide/iceberg-writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,8 @@ The gate has these properties, and a new rule has to keep them:
Most rules are pinned by `CometIcebergWriteDetectionSuite`, and the row lineage rule by
`CometIcebergWriteActionSuite`. Add a case to the detection suite for any new rule, for
both the accepted and the declined side. Update the eligibility table in the user guide in the same
change. Whether each existing restriction is permanent is tracked in
[#5643](https://github.com/apache/datafusion-comet/issues/5643).
change, and the user guide's "Plans for each fallback" lists, which record whether each rule is
planned to be lifted, waiting on another project, or not planned.

## What Crosses the Boundary

Expand Down
75 changes: 75 additions & 0 deletions docs/source/user-guide/latest/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -694,6 +694,81 @@ or GCS each read is a GET request, so the task waits two request round trips per
small next to uploading a large file, but a write that produces many small files, such as a fanout
write over many partitions, pays it for every one of them.

#### Plans for each fallback

Each eligibility rule above is planned to be lifted, waiting on work in another project, or not
planned. A rule is lifted only once the native writer matches iceberg-java's output for the
setting it guards.

Planned to be lifted:

- `write.object-storage.enabled=true`
([#6956](https://github.com/apache/datafusion-comet/issues/6956))
- S3 credential providers other than an explicit S3A provider list, among them anonymous
providers and S3A delegation tokens; S3A server-side encryption settings when a Hadoop
credential store is configured; and ACL, storage class, requester-pays and `dsse-kms` settings
([#6957](https://github.com/apache/datafusion-comet/issues/6957))
- Merge-on-read `DELETE`, `UPDATE` and `MERGE` on a format version 2 table
([#6240](https://github.com/apache/datafusion-comet/issues/6240))
- Bloom filters (`write.parquet.bloom-filter-*`)
([PR #5724](https://github.com/apache/datafusion-comet/pull/5724))
- Copy-on-write `DELETE`, `UPDATE` and `MERGE`, and `rewrite_data_files`, on a format version 3
table with Iceberg 1.10 or newer
([#6669](https://github.com/apache/datafusion-comet/issues/6669))
- Azure (`abfs`, `abfss`, `wasb`, `wasbs`) and `oss` data locations, S3-compatible scheme
aliases, and a `gs` data location opened through `HadoopFileIO`
([#6958](https://github.com/apache/datafusion-comet/issues/6958))
- `parquet.enable.dictionary`, `parquet.columnindex.truncate.length`,
`write.parquet.page-version=v2`, `write.parquet.stats-enabled.column.*`, non-default
`row-group-check-*` counts, and a zstd level of `0`
([#6954](https://github.com/apache/datafusion-comet/issues/6954))
- `uuid` and `unknown` columns
([#6959](https://github.com/apache/datafusion-comet/issues/6959))
- `hdfs` data locations on a cluster without Kerberos. iceberg-rust's `hdfs-native` backend
([apache/iceberg-rust#3111](https://github.com/apache/iceberg-rust/pull/3111)) already serves
the native scan ([#5894](https://github.com/apache/datafusion-comet/issues/5894)), so the work is
on Comet's side ([#7637](https://github.com/apache/datafusion-comet/issues/7637)). On a cluster
with Kerberos the write waits on the same delegation token support as the native scan.

Waiting on work in another project:

- Merge-on-read on a format version 3 table, which writes deletion vectors, waits on a deletion
vector writer in iceberg-rust
([apache/iceberg-rust#2580](https://github.com/apache/iceberg-rust/issues/2580)).
- Table encryption (`encryption.*` properties, an `EncryptingFileIO`, or Iceberg's standard
encryption manager) waits on iceberg-rust's table encryption support
([apache/iceberg-rust#2034](https://github.com/apache/iceberg-rust/issues/2034)).
- `variant` columns wait on native `VariantType` support in Comet
([#5433](https://github.com/apache/datafusion-comet/issues/5433), part of [#5438](https://github.com/apache/datafusion-comet/issues/5438)).
- A negative zstd compression level waits on parquet-rs, which accepts zstd levels 1 to 22
([apache/arrow-rs#11545](https://github.com/apache/arrow-rs/issues/11545)).
- Format version 4 waits on a final version 4 table specification.
- `timestamp_ns`, `geometry` and `geography` columns wait on Iceberg's Spark integration, which has
no Spark type for them, so Spark cannot write such a table today.

Not planned:

- Avro and ORC data files. The native writer writes Parquet only, and iceberg-rust has no Avro or
ORC data file writer.
- Any other data location scheme, which no iceberg-rust storage backend serves.
- A custom location provider (`write.location-provider.impl`, or one a custom `TableOperations`
supplies), a `FileIO` outside the recognized classes, or a custom encryption manager. Each is
user code that runs in the JVM, which the native writer cannot call.
- S3 object tags, access points, transfer acceleration, dual-stack and cross-region endpoints,
remote signing, and custom S3A signers. Comet's native S3 client has no equivalent for them.
- Any other `write.parquet.*` key, such as one a newer Iceberg version adds. The allowlist fails
closed so that no key is silently ignored, and Comet vets each new key when it adds support for
that Iceberg version.
- A size or limit that is not a positive Java int, a format version 1 spec that mixes a live field
with a `void` field whose source column was dropped, and an `s3`, `s3a` or `gs` location with no
bucket in its authority. iceberg-java rejects these writes, or resolves the location through
Hadoop, so falling back keeps its behavior and its error.
- A gzip level outside 0 to 9 or a brotli level outside 0 to 11. parquet-rs rejects them, and
translating each one is not planned.
Comment on lines +766 to +767

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#6996 added a rule after this branch's base: brotli falls back unless Hadoop's BrotliCodec class loads (requireLoadableBrotliCodec). It is not in any of the three lists yet. I think it belongs under "Not planned" next to the other writes iceberg-java fails.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added under "Not planned" in fa64c57, linked to #6960, next to the gzip and brotli level entry.

- `brotli` compression when Hadoop's `BrotliCodec` class cannot be loaded. iceberg-java fails the
write and no JVM reader could read the files, so falling back keeps iceberg-java's error
([#6960](https://github.com/apache/datafusion-comet/issues/6960)).

### Failure handling

Eligibility is decided entirely at plan time. That includes the reflection surface: every
Expand Down