Skip to content

Upgrade to parquet 1.17.1 + iceberg 1.11.0 on Java 17 (5.0.0) - #1160

Open
sfc-gh-miqbal wants to merge 2 commits into
masterfrom
release/5.0.0
Open

sfc-gh-miqbal wants to merge 2 commits into
masterfrom
release/5.0.0

Conversation

@sfc-gh-miqbal

@sfc-gh-miqbal sfc-gh-miqbal commented Jun 25, 2026 •

Copy link
Copy Markdown

Summary

Upgrades the streaming ingest SDK to parquet 1.17.1 and iceberg 1.11.0, which moves the language level to Java 17 (iceberg 1.11 ships Java 17 bytecode). Bumps the artifact version to 5.0.0.

Changes

  • Versions: parquet 1.14.4 → 1.17.1, iceberg 1.6.1 → 1.11.0, SDK 4.4.3 → 5.0.0
  • Java 17: compiler source/target, CI build matrix, and README updated; consumer JDK 8/11 dropped from the e2e-jar-test matrix (the artifact is now Java 17)
  • Dependency convergence (enforcer RequireUpperBoundDeps): avro 1.12.1, jackson 2.21.3, slf4j 2.0.17, snappy 1.1.10.7, commons-compress 1.28.0, commons-codec 1.19.0, commons-io 2.20.0
  • Shading / licensing: relocate org.locationtech.jts (pulled in by iceberg) into the shaded namespace and register its EDL 1.0 license; accept the bare MIT license name (slf4j 2.0)
  • Tests on Java 17: run on the classpath (useModulePath=false) so the SDK's org.apache.parquet.* helper classes resolve under the module system; add the --add-opens flags required by PowerMock / JDBC / JWT reflection
  • Build plugins: align maven-failsafe-plugin to 3.2.5 (matches surefire)
  • CI dependency resolution: route through the Google-hosted Maven Central mirror to avoid repo.maven.apache.org rate-limiting on the runners

Compatibility

Breaking: the SDK is now a Java 17 artifact — consumers on Java 8/11 must upgrade.

Notes

Had to route to GCS because Github checks kept getting rate limited when downloading artifacts, if everything looks good I can remove that but checks may fail without that

The few failing checks I think are flaky, e.g. even this job from 4.4.3 release PR has a jacoco error that is similar to the reason one of the checks is failing https://github.com/snowflakedb/snowflake-ingest-java/actions/runs/25461822030/job/74705663103?pr=1152

🤖 Generated with Claude Code

@codecov-commenter

codecov-commenter commented Jun 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.50%. Comparing base (38a6d6b) to head (a9343d6).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1160      +/-   ##
==========================================
+ Coverage   55.82%   60.50%   +4.67%     
==========================================
  Files         214      214              
  Lines       14616    14701      +85     
  Branches     1606     1612       +6     
==========================================
+ Hits         8160     8895     +735     
+ Misses       5859     5123     -736     
- Partials      597      683      +86     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sfc-gh-miqbal sfc-gh-miqbal changed the title Bump to parquet 1.17.1 + iceberg 1.11.0; require Java 17 (5.0.0) Upgrade to parquet 1.17.1 + iceberg 1.11.0 on Java 17 (5.0.0) Jun 25, 2026
@sfc-gh-miqbal
sfc-gh-miqbal force-pushed the release/5.0.0 branch 3 times, most recently from b316476 to e2196a8 Compare June 26, 2026 00:27
@sfc-gh-miqbal
sfc-gh-miqbal marked this pull request as ready for review June 26, 2026 05:26
@sfc-gh-miqbal
sfc-gh-miqbal requested review from a team as code owners June 26, 2026 05:26

@sfc-gh-alhuang sfc-gh-alhuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, question on the minimum java version, PTAL.

fail-fast: false
matrix:
java: [ 8 ]
java: [ 17 ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we discussed that parquet 1.17 requires java 11+. Can we use the minimum java for merge gate check?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iceberg 1.11, and then later upgrade to 1.12 which we are planning, will require java 17

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, let's bump to java 17 then. cc @sfc-gh-hmadan @sfc-gh-ggeng

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we get @sfc-gh-hmadan 's approve on this. I though previously we only want to bump to java 11. I remember we do have quite some customers on java 11

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: ATM SSv2 minimum support version is java 11+, discussing on this. https://snowflake.slack.com/archives/C08E56WB06N/p1782510903007879

Comment thread e2e-jar-test/run_e2e_jar_test.sh Resolved
Comment thread e2e-jar-test/run_e2e_jar_test.sh Resolved
Comment thread pom.xml Resolved
Bumps the streaming ingest SDK to parquet 1.17.1 and iceberg 1.11.0. iceberg
1.11 ships Java 17 bytecode, so the language level moves to Java 17 and the
artifact version to 5.0.0.

- parquet 1.14.4 -> 1.17.1, iceberg 1.6.1 -> 1.11.0
- Java 8 -> 17: compiler source/target, CI build matrix, README; drop consumer
  JDK 8/11 from the e2e-jar matrix (the artifact is now Java 17)
- converge transitive deps for the enforcer: avro 1.12.1, jackson 2.21.3,
  slf4j 2.0.17, snappy 1.1.10.7, commons-compress 1.28.0, commons-codec 1.19.0,
  commons-io 2.20.0
- shade-relocate org.locationtech.jts (pulled in by iceberg) and register its
  EDL 1.0 license; accept the bare "MIT" license name (slf4j 2.0)
- run tests on the classpath (useModulePath=false) so the SDK's
  org.apache.parquet.* helper classes resolve under the Java 17 module system
- add the --add-opens required by PowerMock/JDBC/JWT reflection on Java 17
- align maven-failsafe-plugin to 3.2.5 (matches surefire)
- resolve CI dependencies through the Google-hosted Maven Central mirror to
  avoid repo.maven.apache.org rate-limiting on the runners
- version 4.4.3 -> 5.0.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review (the 429 issue is mitigated by reruns), remove the Google-hosted
Maven Central mirror workaround:
- delete mvn_settings_ci.xml
- revert .mvn/maven.config to the baseline retry settings
- revert scripts/run_gh_actions.sh (drop --settings wiring)
- run_e2e_jar_test.sh: drop the ~/.m2/settings.xml copy (keeps the JDK 17 build)
- End2EndTest.yml: drop --settings on the Windows mvn commands (keeps Java 17)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants