Repository navigation
Data, Flink: Derive writer test formats from the classpath instead of hardcoding Vortex - #40
Conversation
…ortex The shared writer test bases in iceberg-data enumerate VORTEX in their @parameters, but the Flink modules inherit those tests with only the iceberg-data test jar on the classpath -- iceberg-vortex is not a Flink dependency and there is no Vortex format model for RowData -- so every Vortex case failed with "Format model is not registered for format VORTEX and type interface org.apache.flink.table.data.RowData". Flink 1.20, 2.1 and 2.2 worked around this by overriding parameters() in TestFlinkFileWriterFactory, TestFlinkPositionDeltaWriters and TestFlinkRollingFileWriters. Flink 2.3 arrived with the upstream merge without those overrides, so the default Flink build broke. Rather than replay the overrides once per Flink version, take the formats from FileFormatTestSupport, which already gates Vortex on iceberg-vortex being loadable and is what the format model TCK uses. Engines that ship Vortex (Spark, iceberg-data itself) keep their Vortex coverage unchanged; engines that do not simply stop enumerating it, so the next Flink version merged from upstream needs no fork-specific edit. The three per-version Flink overrides are dropped, leaving those files identical to upstream. Verified: the four sink writer test classes pass on Flink 1.20, 2.1, 2.2 and 2.3 (148 tests each, no Vortex cases), and TestGenericFileWriterFactory in iceberg-data still runs its 10 Vortex cases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VjsATyzReo8NYYWgwvcbSE
|
|
|
| Run | Event | Branch | Result |
|---|---|---|---|
| 35120283386 | push | main |
failure |
| 35101672261 | push | main |
failure |
| 34806337372 | schedule | main |
failure |
| 34083331744 | schedule | main |
failure |
| 33357319047 | schedule | main |
failure |
No fix to port. The conflict is between code scanning default setup being enabled on this fork and the advanced .github/workflows/codeql.yml inherited from upstream; GitHub refuses the SARIF upload when both are active. Resolving it is a repository-settings change — disable default setup under Settings → Code security, or disable/remove the advanced workflow on this fork — not something a commit on this PR can do, and not something I'd widen this PR to attempt. I also haven't re-run it: the error is a deterministic configuration rejection, and the base-branch history above already establishes it isn't this change.
I'm leaving this check red and continuing to drive the rest of CI — the Flink and Spark jobs are what this PR is about.
Generated by Claude Code
CVE Scan failures on the four
|
| CVE | Severity | Package | Installed | Fixed |
|---|---|---|---|---|
| GHSA-r7wm-3cxj-wff9 | HIGH | com.fasterxml.jackson.core:jackson-core |
2.21.0 |
2.18.8, 2.21.4 |
| CVE-2026-54512 | HIGH | com.fasterxml.jackson.core:jackson-databind |
2.21.0 |
2.18.8, 3.1.4, 2.21.4 |
| CVE-2026-54513 | HIGH | com.fasterxml.jackson.core:jackson-databind |
2.21.0 |
2.18.8, 2.21.4, 3.1.4 |
Why it isn't this PR's. This change touches only test sources (data/src/test, flink/*/flink/src/test) — it cannot alter the contents of a Spark runtime shadow jar, and it does not touch gradle/libs.versions.toml or any build.gradle. The same CVE Scan passed on main at the identical base commit 4e9441b ~90 minutes before this run (run 35120283351), and the Jackson pins have been unchanged since 89987a6 (Sep 11). What moved is Trivy's vulnerability database, not the code: these advisories landed between the two runs. The next push to main will fail the same way.
No fix to port, and the real fix is out of scope here. The version catalog already pins the fixed version — jackson221 = { strictly = "2.21.4" } — and spark/v4.2/build.gradle forces it across sparkProjects, which includes the runtime project:
force "com.fasterxml.jackson.core:jackson-databind:${libs.versions.jackson221.get()}"
force "com.fasterxml.jackson.core:jackson-core:${libs.versions.jackson221.get()}"Yet the scanned bundle still carries 2.21.0, so the forced version is not reaching the shaded jar. That's a build-system investigation — why the resolutionStrategy force does not apply to the shadow jar's contents — spanning spark/v3.5, v4.0, v4.1 and v4.2. Proposed patch, for a separate PR: work out which configuration actually feeds shadowJar and apply the force (or a strictly constraint) there, then confirm with ./gradlew :iceberg-spark:iceberg-spark-runtime-4.2_2.13:dependencies that jackson-core/jackson-databind resolve to 2.21.4. Note spark/v3.5 and v4.0 force jackson218 (2.18.8), which is also a fixed version per all three advisories, so they should land clean by the same route.
I have not added these CVEs to .github/trivyignore/ — suppressing a live HIGH advisory isn't something I'll do to turn a check green, and I'm not widening a test-only PR with a dependency-resolution change.
I've re-run the failed jobs once to confirm the finding is deterministic rather than a transient database state; I'll report the outcome here and won't re-run again.
Generated by Claude Code
Re-run outcome: reproduced identically, and a correction to the diagnosisAll four One correction to my previous comment. I wrote that
That makes the diagnosis simpler than I first described, not more complex: there is one root cause, not a per-version propagation gap. Something outside the and then constrain whichever configuration the shadow jar actually consumes. Fixing that one place should clear all four distributions at once. Nothing here changes this PR's scope: the diff is test-only and cannot affect bundle contents. Generated by Claude Code |
Flink tests discover available format from classpath instead of hardcoding