Repository navigation
Conversation
…writes work on 1.12 Iceberg 1.12 removed the public ManifestFiles.read(ManifestFile, FileIO) overload, so the executor-side reflection probe failed and every native Iceberg write fell back to iceberg-java. The three-argument overload (manifest, io, specsById) is public on every supported version (1.5.2 through 1.12), so resolve that and pass the table's PartitionSpec for the stamped spec id. DetectionDelegatingFileIO moves to Java: HadoopConfigurable.getConf is abstract on some supported versions and a default method on 1.12, and Scala has no spelling that compiles against both.
Add dev/diffs/iceberg/1.12.0.diff, ported from the 1.11.0 diff and regenerated against the apache-iceberg-1.12.0 tag, and an iceberg_1_12 job in the nightly tier and behind the run-iceberg-tests label. Spark 4.2 support is still experimental, so it does not join the merge queue.
…ays on the 4.0 runtime Iceberg 1.12.0 has a Spark 4.2 module in source, but its release script only publishes the 3.5, 4.0 and 4.1 runtimes, so iceberg-spark-runtime-4.2 does not exist on Maven Central.
0db2646 to
439104f
Compare
comphead
left a comment
There was a problem hiding this comment.
The reflection fix reads correctly. ManifestFiles.read(ManifestFile, FileIO, Map) is public in 1.5.2, 1.8.1, 1.10.0, 1.11.0 and 1.12.0, and ManifestReader only calls specsById.get(manifest.partitionSpecId()), so the single-entry map is enough. The branch is based on 780a860 and needs a rebase. It conflicts with main in .github/workflows/README.md, dev/ci/check-ci-config.py, dev/ci/compute-changes.py, ci.md, iceberg-spark-tests.md and iceberg-writes.md (mainly #7516, which moved Iceberg 1.11 to the PR tier, and #7521, which removed Spark 3.4 and Iceberg 1.8). The tier text, ICEBERG_OPT_IN, the "all five" comment and the new "1.5.2 through 1.12" comments all need updating to the post-3.4 world. I have not built or run anything.
| spark-full: '4.1.3' | ||
| java: 17 | ||
|
|
||
| iceberg_1_12: |
There was a problem hiding this comment.
The rationale is that no CI job covered 1.12, so nothing caught the silent fallback. Would this job have caught it? dev/ci/summarize-iceberg-writes.py never fails a job, and the upstream Iceberg tests do not assert which writer ran. I expect that reverting part 1 leaves iceberg_1_12 green, with the native share only visible in the step summary. The Comet suites that would fail still run on 1.11 only, since the spark-4.2 profile uses the 4.0 runtime. Could the job (or a Comet test) assert that at least one write ran natively, or that the executor reflection probe resolves against the 1.12 core jar? Otherwise the fix has no regression guard.
There was a problem hiding this comment.
You're right, it would not have caught it. Fixed in 6e991c5. summarize-iceberg-writes.py has a new --require-native flag that exits non-zero, after printing the summary, when none of the recorded writes ran on the native writer. That includes the case where no writes were recorded at all. The cross-shard summary in iceberg-spark-shard-coverage and the extensions job pass it, so the guard covers every Iceberg version, not just 1.12. Before relying on it, I ran the summarizer on last night's shard-1 reports. 1.9.1 had 316 native writes (97.5%) and 1.10.0 had 340 (97.7%). The 1.10.0 extensions run had 2927 (37.5%). So no version is near zero. A shard's own summary still never fails, because a single shard can legitimately record no writes. Unit tests for the flag are in test-summarize-iceberg-writes.py.
| * that is public on every supported Iceberg release (1.5.2 through 1.12; 1.12 dropped the | ||
| * two-argument one). | ||
| */ | ||
| private def manifestReadMethodOf(fileIOClass: Class[_]): java.lang.reflect.Method = |
There was a problem hiding this comment.
This resolves the class and the Method on every call, once per task in decodeManifestToDataFiles and once per plan in the probe. The same object already has a cached getMethod(clazz, name, paramTypes*) backed by methodCache. Would using it here keep this lookup consistent with the rest of the file and avoid the repeated Class.getMethod scan? The probe and the executor already share the helper, which is good, so only the lookup itself would change. The comment also says "1.5.2 through 1.12", but main no longer supports the 1.5.2 (Spark 3.4) profile after #7521.
There was a problem hiding this comment.
Done in 6e991c5. manifestReadMethodOf now goes through the cached getMethod(clazz, name, paramTypes*), and the comment says 1.8.1 through 1.12.
| delegate.setConf(conf); | ||
| } | ||
|
|
||
| public Configuration getConf() { |
There was a problem hiding this comment.
The port keeps every override the Scala class had, and nothing else references the deleted Scala class. Is Java needed though? Scala only requires override when the inherited member is concrete (the 1.12 default getConf), and allows it when the member is abstract (1.8.1 through 1.11, via Hadoop's Configurable). I have not compiled it, but override def getConf: Configuration = delegate.getConf looks like it would satisfy both shapes and keep the class next to its only user. If the Java file is kept, the class comment's claim that Scala "rejects it for an abstract one" should be checked.
There was a problem hiding this comment.
You're right, the Java wasn't needed. In 6e991c5 the class is back in CometIcebergWriteDetectionSuite.scala with override def getConf. I compiled that class on its own with scalac 2.12.18 and 2.13.18 against iceberg-core 1.8.1, 1.10.0, 1.11.0 and 1.12.0, and every combination compiles. Without override, 1.12.0 fails with "needs override' modifier" against HadoopConfigurable's default getConf. The suite also test-compiles under the default, spark-3.5andspark-4.2` profiles.
|
|
||
| The example below uses Spark's package downloader to retrieve Comet $COMET_VERSION and Iceberg | ||
| 1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. The native Iceberg | ||
| 1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, 1.11, and 1.12. The native Iceberg |
There was a problem hiding this comment.
"Comet has been tested with Iceberg ... 1.12" is stronger than the evidence in the PR. Comet's own Iceberg suites do not run on 1.12 in CI (they ran locally against a temporary pom edit, 204 of 206, with RESTCatalogAdapter.java not compiling on 1.12). What does run is Iceberg's own Spark 4.2 suite from source, nightly. Could the sentence say that, or list 1.12 separately?
There was a problem hiding this comment.
Reworded in 6e991c5. The paragraph now says that Comet's own suites run against Iceberg 1.8, 1.10 and 1.11, and that Iceberg's Spark suites run with Comet against 1.9, 1.10, 1.11 and 1.12. It also says 1.12 is tested only through Iceberg's suites, with Spark 4.2. The sentence used to list 1.5 as well, which went stale with #7521, so I dropped it.
| ".mvn/**", | ||
| "mvnw", | ||
| ], | ||
| "iceberg_1_12": [ |
There was a problem hiding this comment.
This is a fourth copy of the same 24-entry path list (iceberg_1_9, iceberg_1_10, iceberg_1_11, now iceberg_1_12). Would it be worth defining the list once and referencing it, as spark_4_1_hive is populated after the dict, so a new path cannot be added to three of the four?
There was a problem hiding this comment.
Done in 6e991c5. iceberg_1_9 holds the list, and iceberg_1_10, iceberg_1_11 and iceberg_1_12 are filled in after the dict, the same way as spark_4_1_hive. I dumped FILTERS and POLICY before and after the change, and the JSON is byte-identical. check-ci-config.py passes.
|
Correction to my review above: it was anchored to an earlier push of this branch ( The five inline points still apply at
|
- Fail the Iceberg shard coverage and extensions jobs when no write ran on the native writer (summarize-iceberg-writes.py --require-native), so a release that makes the gate decline every write no longer passes silently. - Resolve ManifestFiles.read through the cached getMethod helper, and drop 1.5.2 from its supported-version range. - Move DetectionDelegatingFileIO back to Scala: override def getConf compiles against Iceberg 1.8.1 through 1.12.0. - Say which suites test which Iceberg versions in the user guide. - Define the Iceberg path filter once in compute-changes.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Which issue does this PR close?
Closes #7240.
Part of #7493. The first item there (pointing the
spark-4.2profile aticeberg-spark-runtime-4.2_2.13:1.12.0) is not done, see below.Rationale for this change
Iceberg 1.12 removed the public two-argument
ManifestFiles.read(ManifestFile, FileIO), so every native Iceberg write silently fell back to iceberg-java. No CI job covered 1.12, so nothing caught it.The Spark 4.2 runtime that #7493 wants the profile to use does not exist: Iceberg 1.12.0 has a
spark/v4.2module in source, butdev/stage-binaries.shat the release tag publishesSPARK_VERSIONS=3.5,4.0,4.1only, andiceberg-spark-runtime-4.2_2.13:1.12.0does not resolve from Maven Central. Thespark-4.2profile therefore stays on the 4.0 runtime, and the!isSpark42Plusguards and the #4969 test stay as they are, until a release publishes it.What changes are included in this PR?
One commit per part:
ManifestFiles.read(manifest, io, specsById), public on every supported version (1.5.2 through 1.12), replaces the two-argument form in the executor probe and in the manifest decode. The decode passes the table'sPartitionSpecfor the stamped spec id.DetectionDelegatingFileIOmoves to Java becauseHadoopConfigurable.getConfis abstract on some supported versions and a default method on 1.12, which Scala cannot spell for both.dev/diffs/iceberg/1.12.0.diff, ported from the 1.11.0 diff and regenerated against theapache-iceberg-1.12.0tag (v4.1becomesv4.2; the session builder in the compaction benchmark moved toIcebergCompactionBenchmark), and aniceberg_1_12job running Iceberg 1.12.0 on Spark 4.2.0. It is nightly plus therun-iceberg-testslabel, because Spark 4.2 support is still experimental.ci.yml,compute-changes.py,check-ci-config.pyand the workflows README are updated.iceberg-spark-tests.md,iceberg-writes.md,ci.md, the Iceberg user guide, and aspark/pom.xmlcomment recording why the 4.2 runtime is not used.How are these changes tested?
CometIcebergRewriteActionSuite,CometIcebergWriteActionSuiteandCometIcebergWriteDetectionSuitepass on the default Iceberg 1.11 (206 of 206).GenericAppenderFactory, which the 1.12 runtime jar does not bundle, andfall-back: format-version=4now needs an absolute table location.RESTCatalogAdapter.javaalso does not compile against 1.12 (RESTUtil.NAMESPACE_SPLITTER,CatalogHandlers.loadTableandRESTUtil.decodeNamespacechanged). These need fixing when a profile moves to 1.12.apache-iceberg-1.12.0tag. With Comet installed for Spark 4.2,TestFilterPushDown(42 tests) andTestRewriteDataFilesAction(195 tests, 8 skipped) pass on Iceberg 1.12.0 and Spark 4.2.0 with no failures. The fulliceberg_1_12job has not run yet.python3 dev/ci/check-ci-config.pypasses.