Skip to content

fix: native Iceberg writes on Iceberg 1.12, and run Iceberg's suites on 1.12.0 / Spark 4.2 - #7520

Draft
andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:iceberg-1-12-support
Draft

andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:iceberg-1-12-support

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #7240.

Part of #7493. The first item there (pointing the spark-4.2 profile at iceberg-spark-runtime-4.2_2.13:1.12.0) is not done, see below.

Rationale for this change

Iceberg 1.12 removed the public two-argument ManifestFiles.read(ManifestFile, FileIO), so every native Iceberg write silently fell back to iceberg-java. No CI job covered 1.12, so nothing caught it.

The Spark 4.2 runtime that #7493 wants the profile to use does not exist: Iceberg 1.12.0 has a spark/v4.2 module in source, but dev/stage-binaries.sh at the release tag publishes SPARK_VERSIONS=3.5,4.0,4.1 only, and iceberg-spark-runtime-4.2_2.13:1.12.0 does not resolve from Maven Central. The spark-4.2 profile therefore stays on the 4.0 runtime, and the !isSpark42Plus guards and the #4969 test stay as they are, until a release publishes it.

What changes are included in this PR?

One commit per part:

  1. ManifestFiles.read(manifest, io, specsById), public on every supported version (1.5.2 through 1.12), replaces the two-argument form in the executor probe and in the manifest decode. The decode passes the table's PartitionSpec for the stamped spec id. DetectionDelegatingFileIO moves to Java because HadoopConfigurable.getConf is abstract on some supported versions and a default method on 1.12, which Scala cannot spell for both.
  2. dev/diffs/iceberg/1.12.0.diff, ported from the 1.11.0 diff and regenerated against the apache-iceberg-1.12.0 tag (v4.1 becomes v4.2; the session builder in the compaction benchmark moved to IcebergCompactionBenchmark), and an iceberg_1_12 job running Iceberg 1.12.0 on Spark 4.2.0. It is nightly plus the run-iceberg-tests label, because Spark 4.2 support is still experimental. ci.yml, compute-changes.py, check-ci-config.py and the workflows README are updated.
  3. Docs: iceberg-spark-tests.md, iceberg-writes.md, ci.md, the Iceberg user guide, and a spark/pom.xml comment recording why the 4.2 runtime is not used.

How are these changes tested?

  • CometIcebergRewriteActionSuite, CometIcebergWriteActionSuite and CometIcebergWriteDetectionSuite pass on the default Iceberg 1.11 (206 of 206).
  • The same suites against the Iceberg 1.12.0 jar on Spark 4.1 (a temporary local pom edit): 204 of 206 pass, with no "reflection did not resolve" warnings. The two failures are in Comet's tests, not the fix: one needs GenericAppenderFactory, which the 1.12 runtime jar does not bundle, and fall-back: format-version=4 now needs an absolute table location. RESTCatalogAdapter.java also does not compile against 1.12 (RESTUtil.NAMESPACE_SPLITTER, CatalogHandlers.loadTable and RESTUtil.decodeNamespace changed). These need fixing when a profile moves to 1.12.
  • The new diff applies cleanly to the apache-iceberg-1.12.0 tag. With Comet installed for Spark 4.2, TestFilterPushDown (42 tests) and TestRewriteDataFilesAction (195 tests, 8 skipped) pass on Iceberg 1.12.0 and Spark 4.2.0 with no failures. The full iceberg_1_12 job has not run yet.
  • python3 dev/ci/check-ci-config.py passes.

…writes work on 1.12

Iceberg 1.12 removed the public ManifestFiles.read(ManifestFile, FileIO)
overload, so the executor-side reflection probe failed and every native
Iceberg write fell back to iceberg-java. The three-argument overload
(manifest, io, specsById) is public on every supported version (1.5.2
through 1.12), so resolve that and pass the table's PartitionSpec for the
stamped spec id.

DetectionDelegatingFileIO moves to Java: HadoopConfigurable.getConf is
abstract on some supported versions and a default method on 1.12, and Scala
has no spelling that compiles against both.
Add dev/diffs/iceberg/1.12.0.diff, ported from the 1.11.0 diff and
regenerated against the apache-iceberg-1.12.0 tag, and an iceberg_1_12 job
in the nightly tier and behind the run-iceberg-tests label. Spark 4.2
support is still experimental, so it does not join the merge queue.
…ays on the 4.0 runtime

Iceberg 1.12.0 has a Spark 4.2 module in source, but its release script only
publishes the 3.5, 4.0 and 4.1 runtimes, so iceberg-spark-runtime-4.2 does
not exist on Maven Central.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reflection fix reads correctly. ManifestFiles.read(ManifestFile, FileIO, Map) is public in 1.5.2, 1.8.1, 1.10.0, 1.11.0 and 1.12.0, and ManifestReader only calls specsById.get(manifest.partitionSpecId()), so the single-entry map is enough. The branch is based on 780a860 and needs a rebase. It conflicts with main in .github/workflows/README.md, dev/ci/check-ci-config.py, dev/ci/compute-changes.py, ci.md, iceberg-spark-tests.md and iceberg-writes.md (mainly #7516, which moved Iceberg 1.11 to the PR tier, and #7521, which removed Spark 3.4 and Iceberg 1.8). The tier text, ICEBERG_OPT_IN, the "all five" comment and the new "1.5.2 through 1.12" comments all need updating to the post-3.4 world. I have not built or run anything.

Comment thread .github/workflows/ci.yml
spark-full: '4.1.3'
java: 17

iceberg_1_12:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale is that no CI job covered 1.12, so nothing caught the silent fallback. Would this job have caught it? dev/ci/summarize-iceberg-writes.py never fails a job, and the upstream Iceberg tests do not assert which writer ran. I expect that reverting part 1 leaves iceberg_1_12 green, with the native share only visible in the step summary. The Comet suites that would fail still run on 1.11 only, since the spark-4.2 profile uses the 4.0 runtime. Could the job (or a Comet test) assert that at least one write ran natively, or that the executor reflection probe resolves against the 1.12 core jar? Otherwise the fix has no regression guard.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, it would not have caught it. Fixed in 6e991c5. summarize-iceberg-writes.py has a new --require-native flag that exits non-zero, after printing the summary, when none of the recorded writes ran on the native writer. That includes the case where no writes were recorded at all. The cross-shard summary in iceberg-spark-shard-coverage and the extensions job pass it, so the guard covers every Iceberg version, not just 1.12. Before relying on it, I ran the summarizer on last night's shard-1 reports. 1.9.1 had 316 native writes (97.5%) and 1.10.0 had 340 (97.7%). The 1.10.0 extensions run had 2927 (37.5%). So no version is near zero. A shard's own summary still never fails, because a single shard can legitimately record no writes. Unit tests for the flag are in test-summarize-iceberg-writes.py.

* that is public on every supported Iceberg release (1.5.2 through 1.12; 1.12 dropped the
* two-argument one).
*/
private def manifestReadMethodOf(fileIOClass: Class[_]): java.lang.reflect.Method =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves the class and the Method on every call, once per task in decodeManifestToDataFiles and once per plan in the probe. The same object already has a cached getMethod(clazz, name, paramTypes*) backed by methodCache. Would using it here keep this lookup consistent with the rest of the file and avoid the repeated Class.getMethod scan? The probe and the executor already share the helper, which is good, so only the lookup itself would change. The comment also says "1.5.2 through 1.12", but main no longer supports the 1.5.2 (Spark 3.4) profile after #7521.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6e991c5. manifestReadMethodOf now goes through the cached getMethod(clazz, name, paramTypes*), and the comment says 1.8.1 through 1.12.

delegate.setConf(conf);
}

public Configuration getConf() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The port keeps every override the Scala class had, and nothing else references the deleted Scala class. Is Java needed though? Scala only requires override when the inherited member is concrete (the 1.12 default getConf), and allows it when the member is abstract (1.8.1 through 1.11, via Hadoop's Configurable). I have not compiled it, but override def getConf: Configuration = delegate.getConf looks like it would satisfy both shapes and keep the class next to its only user. If the Java file is kept, the class comment's claim that Scala "rejects it for an abstract one" should be checked.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, the Java wasn't needed. In 6e991c5 the class is back in CometIcebergWriteDetectionSuite.scala with override def getConf. I compiled that class on its own with scalac 2.12.18 and 2.13.18 against iceberg-core 1.8.1, 1.10.0, 1.11.0 and 1.12.0, and every combination compiles. Without override, 1.12.0 fails with "needs override' modifier" against HadoopConfigurable's default getConf. The suite also test-compiles under the default, spark-3.5andspark-4.2` profiles.


The example below uses Spark's package downloader to retrieve Comet $COMET_VERSION and Iceberg
1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. The native Iceberg
1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, 1.11, and 1.12. The native Iceberg

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Comet has been tested with Iceberg ... 1.12" is stronger than the evidence in the PR. Comet's own Iceberg suites do not run on 1.12 in CI (they ran locally against a temporary pom edit, 204 of 206, with RESTCatalogAdapter.java not compiling on 1.12). What does run is Iceberg's own Spark 4.2 suite from source, nightly. Could the sentence say that, or list 1.12 separately?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in 6e991c5. The paragraph now says that Comet's own suites run against Iceberg 1.8, 1.10 and 1.11, and that Iceberg's Spark suites run with Comet against 1.9, 1.10, 1.11 and 1.12. It also says 1.12 is tested only through Iceberg's suites, with Spark 4.2. The sentence used to list 1.5 as well, which went stale with #7521, so I dropped it.

Comment thread dev/ci/compute-changes.py Outdated
".mvn/**",
"mvnw",
],
"iceberg_1_12": [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a fourth copy of the same 24-entry path list (iceberg_1_9, iceberg_1_10, iceberg_1_11, now iceberg_1_12). Would it be worth defining the list once and referencing it, as spark_4_1_hive is populated after the dict, so a new path cannot be added to three of the four?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6e991c5. iceberg_1_9 holds the list, and iceberg_1_10, iceberg_1_11 and iceberg_1_12 are filled in after the dict, the same way as spark_4_1_hive. I dumped FILTERS and POLICY before and after the change, and the JSON is byte-identical. check-ci-config.py passes.

@comphead

Copy link
Copy Markdown
Contributor

Correction to my review above: it was anchored to an earlier push of this branch (0db2646e2d), so its inline comments show as outdated. The current head 439104f9f4 is already rebased past #7516 and #7521 and merges cleanly with main, so please ignore the rebase and stale-tier paragraph in the summary.

The five inline points still apply at 439104f9f4:

  • ci.yml iceberg_1_12 (line 623): nothing in the job fails if the native writer silently falls back again, since summarize-iceberg-writes.py never fails the job.
  • IcebergReflection.scala line 1975: manifestReadMethodOf calls Class.getMethod directly instead of the cached getMethod helper, and line 1972 still says "1.5.2 through 1.12".
  • DetectionDelegatingFileIO.java line 71: is Java needed, given that Scala accepts override on an abstract member too?
  • iceberg.md line 39: "tested with ... 1.12" goes further than what CI runs.
  • compute-changes.py line 388: a fourth copy of the Iceberg path list.

- Fail the Iceberg shard coverage and extensions jobs when no write ran on
  the native writer (summarize-iceberg-writes.py --require-native), so a
  release that makes the gate decline every write no longer passes silently.
- Resolve ManifestFiles.read through the cached getMethod helper, and drop
  1.5.2 from its supported-version range.
- Move DetectionDelegatingFileIO back to Scala: override def getConf compiles
  against Iceberg 1.8.1 through 1.12.0.
- Say which suites test which Iceberg versions in the user guide.
- Define the Iceberg path filter once in compute-changes.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:writer Native Parquet writer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Iceberg write silently falls back on Iceberg 1.12: ManifestFiles.read(ManifestFile, FileIO) was removed

2 participants