Repository navigation
[SNOW-3837078] Include FileInputFormat in the shaded SDK jar - #1166
Merged
sfc-gh-yelgabra merged 7 commits intoAug 27, 2026
Merged
Conversation
Parquet readback loads ParquetReadOptions.Builder, which references FileInputFormat. The shade filter only kept mapreduce/*.class, so that class was missing from the fat jar when the flag is enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
sfc-gh-ggeng
approved these changes
Aug 27, 2026
sfc-gh-yelgabra
enabled auto-merge (squash)
August 27, 2026 00:37
Unit tests compile against unshaded parquet, so they never load relocated Hadoop types. Copy the verification cases into e2e-jar-test/shaded-internal so the FileInputFormat shade gap fails CI when that path is exercised. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the extra e2e module. Run the RowBuffer verification cases from StandardIngestE2ETest against whatever jar the existing e2e project installs, and turn the flag on for basic ingest so the fat-jar Hadoop gap fails in the same environment. Co-authored-by: Cursor <cursoragent@cursor.com>
The RowBuffer cases already live in unit tests. Shaded e2e only needs the flag on so basicTest loads FileInputFormat from the fat jar. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the default e2e path unchanged. Parameterize StandardIngestE2ETest so basic ingest runs twice and the shaded jar is exercised with the flag on. Co-authored-by: Cursor <cursoragent@cursor.com>
| @RunWith(Parameterized.class) | ||
| public class StandardIngestE2ETest { | ||
|
|
||
| @Parameterized.Parameters(name = "enableParquetReadbackVerification={0}") |
Collaborator
There was a problem hiding this comment.
how long would each time run? would this more likely to lead to timeout issue?
Collaborator
There was a problem hiding this comment.
also why we need to test the disable pass?
Co-authored-by: Cursor <cursoragent@cursor.com>
sfc-gh-yelgabra
deleted the
yelgabra-SNOW-3837078-shade-fileinputformat
branch
August 27, 2026 19:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parquet readback loads
ParquetReadOptions.Builder, which references HadoopFileInputFormat. The shade filter only keptmapreduce/*.class, so that class was missing from the fat jar whenenable_parquet_readback_verificationis on.Shaded e2e ingest now turns the flag on (once, not on/off) so a fat jar missing
FileInputFormatfails in the same environment.https://snowflakecomputing.atlassian.net/browse/SNOW-3837078?focusedCommentId=14141925