Skip to content

[SNOW-3837078] Include FileInputFormat in the shaded SDK jar - #1166

Merged
sfc-gh-yelgabra merged 7 commits into
masterfrom
yelgabra-SNOW-3837078-shade-fileinputformat
Aug 27, 2026
Merged

sfc-gh-yelgabra merged 7 commits into
masterfrom
yelgabra-SNOW-3837078-shade-fileinputformat

Conversation

@sfc-gh-yelgabra

@sfc-gh-yelgabra sfc-gh-yelgabra commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Parquet readback loads ParquetReadOptions.Builder, which references Hadoop FileInputFormat. The shade filter only kept mapreduce/*.class, so that class was missing from the fat jar when enable_parquet_readback_verification is on.

Shaded e2e ingest now turns the flag on (once, not on/off) so a fat jar missing FileInputFormat fails in the same environment.

https://snowflakecomputing.atlassian.net/browse/SNOW-3837078?focusedCommentId=14141925

Parquet readback loads ParquetReadOptions.Builder, which references FileInputFormat. The shade filter only kept mapreduce/*.class, so that class was missing from the fat jar when the flag is enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@sfc-gh-yelgabra
sfc-gh-yelgabra enabled auto-merge (squash) August 27, 2026 00:37

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

Unit tests compile against unshaded parquet, so they never load relocated
Hadoop types. Copy the verification cases into e2e-jar-test/shaded-internal
so the FileInputFormat shade gap fails CI when that path is exercised.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sfc-gh-yelgabra sfc-gh-yelgabra changed the title [SNOW-3837078] Include Hadoop FileInputFormat in the shaded SDK jar [SNOW-3837078] Include FileInputFormat in the shaded SDK jar Aug 27, 2026

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

Drop the extra e2e module. Run the RowBuffer verification cases from
StandardIngestE2ETest against whatever jar the existing e2e project
installs, and turn the flag on for basic ingest so the fat-jar Hadoop
gap fails in the same environment.

Co-authored-by: Cursor <cursoragent@cursor.com>

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

The RowBuffer cases already live in unit tests. Shaded e2e only needs the
flag on so basicTest loads FileInputFormat from the fat jar.

Co-authored-by: Cursor <cursoragent@cursor.com>

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

Keep the default e2e path unchanged. Parameterize StandardIngestE2ETest so
basic ingest runs twice and the shaded jar is exercised with the flag on.

Co-authored-by: Cursor <cursoragent@cursor.com>

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@RunWith(Parameterized.class)
public class StandardIngestE2ETest {

@Parameterized.Parameters(name = "enableParquetReadbackVerification={0}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long would each time run? would this more likely to lead to timeout issue?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also why we need to test the disable pass?

Co-authored-by: Cursor <cursoragent@cursor.com>

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@sfc-gh-yelgabra
sfc-gh-yelgabra merged commit 5f05240 into master Aug 27, 2026
46 of 51 checks passed
@sfc-gh-yelgabra
sfc-gh-yelgabra deleted the yelgabra-SNOW-3837078-shade-fileinputformat branch August 27, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants