Skip to content

Data, Flink: Derive writer test formats from the classpath instead of hardcoding Vortex - #40

Merged
robert3005 merged 1 commit into
mainfrom
claude/charming-dirac-hb6avf
Sep 16, 2026
Merged

robert3005 merged 1 commit into
mainfrom
claude/charming-dirac-hb6avf

Conversation

@robert3005

@robert3005 robert3005 commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Flink tests discover available format from classpath instead of hardcoding

…ortex

The shared writer test bases in iceberg-data enumerate VORTEX in their
@parameters, but the Flink modules inherit those tests with only the
iceberg-data test jar on the classpath -- iceberg-vortex is not a Flink
dependency and there is no Vortex format model for RowData -- so every
Vortex case failed with "Format model is not registered for format VORTEX
and type interface org.apache.flink.table.data.RowData".

Flink 1.20, 2.1 and 2.2 worked around this by overriding parameters() in
TestFlinkFileWriterFactory, TestFlinkPositionDeltaWriters and
TestFlinkRollingFileWriters. Flink 2.3 arrived with the upstream merge
without those overrides, so the default Flink build broke.

Rather than replay the overrides once per Flink version, take the formats
from FileFormatTestSupport, which already gates Vortex on iceberg-vortex
being loadable and is what the format model TCK uses. Engines that ship
Vortex (Spark, iceberg-data itself) keep their Vortex coverage unchanged;
engines that do not simply stop enumerating it, so the next Flink version
merged from upstream needs no fork-specific edit. The three per-version
Flink overrides are dropped, leaving those files identical to upstream.

Verified: the four sink writer test classes pass on Flink 1.20, 2.1, 2.2
and 2.3 (148 tests each, no Vortex cases), and TestGenericFileWriterFactory
in iceberg-data still runs its 10 Vortex cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjsATyzReo8NYYWgwvcbSE
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Member Author

Analyze Actions (CodeQL) failure — not from this PR

Job 104907632995 failed with a repository configuration error, not a code finding:

##[error]Code Scanning could not process the submitted SARIF file:
CodeQL analyses from advanced configurations cannot be processed when the default setup is enabled
CodeQL job status was configuration error.

Why it isn't this PR's: the diff touches only test sources under data/src/test and flink/*/flink/src/test — no workflow or Actions files, which is all the actions CodeQL language analyzes. The same check is red on the base branch and has been on every run for weeks:

Run Event Branch Result
35120283386 push main failure
35101672261 push main failure
34806337372 schedule main failure
34083331744 schedule main failure
33357319047 schedule main failure

No fix to port. The conflict is between code scanning default setup being enabled on this fork and the advanced .github/workflows/codeql.yml inherited from upstream; GitHub refuses the SARIF upload when both are active. Resolving it is a repository-settings change — disable default setup under Settings → Code security, or disable/remove the advanced workflow on this fork — not something a commit on this PR can do, and not something I'd widen this PR to attempt. I also haven't re-run it: the error is a deterministic configuration rejection, and the base-branch history above already establishes it isn't this change.

I'm leaving this check red and continuing to drive the rest of CI — the Flink and Spark jobs are what this PR is about.


Generated by Claude Code

Copy link
Copy Markdown
Member Author

CVE Scan failures on the four spark-runtime-* jobs — not from this PR

CVE Scan run 35129783933: spark-runtime-3.5_2.12, spark-runtime-4.0_2.13, spark-runtime-4.1_2.13 and spark-runtime-4.2_2.13 failed. Every other distribution in the same matrix passed — flink-runtime-1.20/2.1/2.2/2.3, aws-bundle, azure-bundle, gcp-bundle, kafka-connect-runtime, open-api-test-fixtures-runtime.

All four failed at the same step, Report Trivy scan results, with three HIGH findings in the bundled Jackson:

## Trivy CVE Scan Results — spark-runtime-4.2_2.13
Found 3 HIGH/CRITICAL vulnerabilities.
CVE Severity Package Installed Fixed
GHSA-r7wm-3cxj-wff9 HIGH com.fasterxml.jackson.core:jackson-core 2.21.0 2.18.8, 2.21.4
CVE-2026-54512 HIGH com.fasterxml.jackson.core:jackson-databind 2.21.0 2.18.8, 3.1.4, 2.21.4
CVE-2026-54513 HIGH com.fasterxml.jackson.core:jackson-databind 2.21.0 2.18.8, 2.21.4, 3.1.4

Why it isn't this PR's. This change touches only test sources (data/src/test, flink/*/flink/src/test) — it cannot alter the contents of a Spark runtime shadow jar, and it does not touch gradle/libs.versions.toml or any build.gradle. The same CVE Scan passed on main at the identical base commit 4e9441b ~90 minutes before this run (run 35120283351), and the Jackson pins have been unchanged since 89987a6 (Sep 11). What moved is Trivy's vulnerability database, not the code: these advisories landed between the two runs. The next push to main will fail the same way.

No fix to port, and the real fix is out of scope here. The version catalog already pins the fixed version — jackson221 = { strictly = "2.21.4" } — and spark/v4.2/build.gradle forces it across sparkProjects, which includes the runtime project:

force "com.fasterxml.jackson.core:jackson-databind:${libs.versions.jackson221.get()}"
force "com.fasterxml.jackson.core:jackson-core:${libs.versions.jackson221.get()}"

Yet the scanned bundle still carries 2.21.0, so the forced version is not reaching the shaded jar. That's a build-system investigation — why the resolutionStrategy force does not apply to the shadow jar's contents — spanning spark/v3.5, v4.0, v4.1 and v4.2. Proposed patch, for a separate PR: work out which configuration actually feeds shadowJar and apply the force (or a strictly constraint) there, then confirm with ./gradlew :iceberg-spark:iceberg-spark-runtime-4.2_2.13:dependencies that jackson-core/jackson-databind resolve to 2.21.4. Note spark/v3.5 and v4.0 force jackson218 (2.18.8), which is also a fixed version per all three advisories, so they should land clean by the same route.

I have not added these CVEs to .github/trivyignore/ — suppressing a live HIGH advisory isn't something I'll do to turn a check green, and I'm not widening a test-only PR with a dependency-resolution change.

I've re-run the failed jobs once to confirm the finding is deterministic rather than a transient database state; I'll report the outcome here and won't re-run again.


Generated by Claude Code

Copy link
Copy Markdown
Member Author

Re-run outcome: reproduced identically, and a correction to the diagnosis

All four spark-runtime-* jobs failed again on re-run with the same three findings — deterministic, not a transient Trivy database state. As said, I won't re-run again.

One correction to my previous comment. I wrote that spark/v3.5 and v4.0 force jackson218 (2.18.8), which is a fixed version per all three advisories, and so "should land clean by the same route". That's wrong: spark-runtime-3.5_2.12 reports the same installed version as 4.2 —

| GHSA-r7wm-3cxj-wff9 | HIGH | com.fasterxml.jackson.core:jackson-core     | 2.21.0 | 2.18.8, 2.21.4 |
| CVE-2026-54512      | HIGH | com.fasterxml.jackson.core:jackson-databind | 2.21.0 | 2.18.8, 3.1.4, 2.21.4 |
| CVE-2026-54513      | HIGH | com.fasterxml.jackson.core:jackson-databind | 2.21.0 | 2.18.8, 2.21.4, 3.1.4 |

2.21.0 is neither the version v3.5 forces (2.18.8) nor the one v4.2 forces (2.21.4), and it is not the catalog's jackson-bom (2.22.2) either. So the bundles carry a Jackson that no declared pin selects, identically across all four Spark versions.

That makes the diagnosis simpler than I first described, not more complex: there is one root cause, not a per-version propagation gap. Something outside the resolutionStrategy.force — most likely Spark's own Jackson arriving through a configuration that feeds shadowJar but isn't covered by configurations.all — is what lands in the jar. The investigation for the follow-up PR should therefore start by identifying where 2.21.0 enters, e.g.

./gradlew :iceberg-spark:iceberg-spark-runtime-4.2_2.13:dependencyInsight \
    --dependency jackson-databind --configuration runtimeClasspath

and then constrain whichever configuration the shadow jar actually consumes. Fixing that one place should clear all four distributions at once.

Nothing here changes this PR's scope: the diff is test-only and cannot affect bundle contents.


Generated by Claude Code

@robert3005
robert3005 merged commit 7349836 into main Sep 16, 2026
51 of 61 checks passed
@robert3005
robert3005 deleted the claude/charming-dirac-hb6avf branch September 16, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants