Repository navigation
Upgrade to parquet 1.17.1 + iceberg 1.11.0 on Java 17 (5.0.0) - #1160
sfc-gh-miqbal wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1160 +/- ##
==========================================
+ Coverage 55.82% 60.50% +4.67%
==========================================
Files 214 214
Lines 14616 14701 +85
Branches 1606 1612 +6
==========================================
+ Hits 8160 8895 +735
+ Misses 5859 5123 -736
- Partials 597 683 +86 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3c640a1 to
1aaac3c
Compare
b316476 to
e2196a8
Compare
sfc-gh-alhuang
left a comment
There was a problem hiding this comment.
LGTM overall, question on the minimum java version, PTAL.
| fail-fast: false | ||
| matrix: | ||
| java: [ 8 ] | ||
| java: [ 17 ] |
There was a problem hiding this comment.
I thought we discussed that parquet 1.17 requires java 11+. Can we use the minimum java for merge gate check?
There was a problem hiding this comment.
Iceberg 1.11, and then later upgrade to 1.12 which we are planning, will require java 17
There was a problem hiding this comment.
Sounds good, let's bump to java 17 then. cc @sfc-gh-hmadan @sfc-gh-ggeng
There was a problem hiding this comment.
can we get @sfc-gh-hmadan 's approve on this. I though previously we only want to bump to java 11. I remember we do have quite some customers on java 11
There was a problem hiding this comment.
Update: ATM SSv2 minimum support version is java 11+, discussing on this. https://snowflake.slack.com/archives/C08E56WB06N/p1782510903007879
e2196a8 to
e9034e2
Compare
Bumps the streaming ingest SDK to parquet 1.17.1 and iceberg 1.11.0. iceberg 1.11 ships Java 17 bytecode, so the language level moves to Java 17 and the artifact version to 5.0.0. - parquet 1.14.4 -> 1.17.1, iceberg 1.6.1 -> 1.11.0 - Java 8 -> 17: compiler source/target, CI build matrix, README; drop consumer JDK 8/11 from the e2e-jar matrix (the artifact is now Java 17) - converge transitive deps for the enforcer: avro 1.12.1, jackson 2.21.3, slf4j 2.0.17, snappy 1.1.10.7, commons-compress 1.28.0, commons-codec 1.19.0, commons-io 2.20.0 - shade-relocate org.locationtech.jts (pulled in by iceberg) and register its EDL 1.0 license; accept the bare "MIT" license name (slf4j 2.0) - run tests on the classpath (useModulePath=false) so the SDK's org.apache.parquet.* helper classes resolve under the Java 17 module system - add the --add-opens required by PowerMock/JDBC/JWT reflection on Java 17 - align maven-failsafe-plugin to 3.2.5 (matches surefire) - resolve CI dependencies through the Google-hosted Maven Central mirror to avoid repo.maven.apache.org rate-limiting on the runners - version 4.4.3 -> 5.0.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e9034e2 to
1ca910e
Compare
Per review (the 429 issue is mitigated by reruns), remove the Google-hosted Maven Central mirror workaround: - delete mvn_settings_ci.xml - revert .mvn/maven.config to the baseline retry settings - revert scripts/run_gh_actions.sh (drop --settings wiring) - run_e2e_jar_test.sh: drop the ~/.m2/settings.xml copy (keeps the JDK 17 build) - End2EndTest.yml: drop --settings on the Windows mvn commands (keeps Java 17) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Upgrades the streaming ingest SDK to parquet 1.17.1 and iceberg 1.11.0, which moves the language level to Java 17 (iceberg 1.11 ships Java 17 bytecode). Bumps the artifact version to 5.0.0.
Changes
e2e-jar-testmatrix (the artifact is now Java 17)RequireUpperBoundDeps): avro 1.12.1, jackson 2.21.3, slf4j 2.0.17, snappy 1.1.10.7, commons-compress 1.28.0, commons-codec 1.19.0, commons-io 2.20.0org.locationtech.jts(pulled in by iceberg) into the shaded namespace and register its EDL 1.0 license; accept the bareMITlicense name (slf4j 2.0)useModulePath=false) so the SDK'sorg.apache.parquet.*helper classes resolve under the module system; add the--add-opensflags required by PowerMock / JDBC / JWT reflectionmaven-failsafe-pluginto 3.2.5 (matches surefire)repo.maven.apache.orgrate-limiting on the runnersCompatibility
Breaking: the SDK is now a Java 17 artifact — consumers on Java 8/11 must upgrade.
Notes
Had to route to GCS because Github checks kept getting rate limited when downloading artifacts, if everything looks good I can remove that but checks may fail without that
The few failing checks I think are flaky, e.g. even this job from 4.4.3 release PR has a jacoco error that is similar to the reason one of the checks is failing https://github.com/snowflakedb/snowflake-ingest-java/actions/runs/25461822030/job/74705663103?pr=1152
🤖 Generated with Claude Code