Repository navigation
[SNOW-3883414] parquet compression corruption check - #1162
sfc-gh-yelgabra merged 33 commits into
Conversation
df692fc to
63b0d20
Compare
ParquetReadOptions.builder() statically imports ParquetInputFormat.getFilter, which forces loading of ParquetInputFormat (extends FileInputFormat from hadoop-mapreduce-client-core). Our new verifyReadBack() calls this on the production flush path, causing ClassNotFoundException in e2e jar tests where only the production jar (without test-scoped deps) is on the classpath. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This reverts commit f102da0.
…pile scope" This reverts commit 162bc9c.
…k verification BdecParquetReader loads InternalParquetRecordReader, which has static imports from ParquetInputFormat (extends hadoop-mapreduce FileInputFormat). This ClassNotFoundException breaks e2e jar tests where hadoop-mapreduce-client-core is not on the runtime classpath. New approach: parse the parquet footer with Util.readFileMetaData() and walk each column chunk's pages with Util.readPageHeader() (both from parquet-format-structures, already compile-scope). For GZIP pages, wrap the compressed values in GZIPInputStream and exhaust it — this triggers the CRC32 check that catches VM-level compression corruption. DataPageV2 rep/def level bytes are uncompressed and skipped before the GZIPInputStream. No pom.xml changes needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move verification logic into BdecParquetReader.verify() static method which uses the original while(reader.read()!=null) loop. Catch NoClassDefFoundError to skip silently when hadoop-mapreduce-client-core is absent (e.g. e2e-jar-test); full verification runs in production. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Counts rows actually read back and throws if the count doesn't match the expected row count. This catches num_values=0 page corruption (e.g. SNOW-3899515) in addition to the existing GZIP CRC32 check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…preduce-client-core Rewrite BdecParquetReader.verify() to walk the parquet structure directly using parquet-format-structures (compile-scoped) and decompress pages via CodecFactory, removing the dependency on hadoop-mapreduce-client-core that was causing NoClassDefFoundError → channel invalidation in environments without the full Hadoop stack (e2e-jar-test, Kafka Connect). Catches both known VM-corruption classes: - Compressed data corruption (GZIP CRC32 failures): CodecFactory.decompress() throws on invalid DEFLATE blocks, mirroring C++ ParquetCompressedBuffer::decompress() - Page header corruption (num_values=0): row count accumulated from first column's page headers and compared to expectedRowCount at the end Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dependency Used directly in BdecParquetReader.verify() via Util, FileMetaData, PageHeader, RowGroup, ColumnChunk, and ColumnMetaData. Previously only a transitive dependency pulled in by parquet-hadoop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tures Same exclusion already applied to parquet-common — CDDL + GPLv2 with classpath exception license is not on the approved list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1162 +/- ##
==========================================
+ Coverage 55.82% 59.99% +4.16%
==========================================
Files 214 214
Lines 14616 14647 +31
Branches 1606 1611 +5
==========================================
+ Hits 8160 8788 +628
+ Misses 5859 5193 -666
- Partials 597 666 +69 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ote hadoop dep - Replace Thrift-based verify() with simpler BdecParquetReader.read() loop - Promote hadoop-mapreduce-client-core from test to compile scope (required because InternalParquetRecordReader extends hadoop's RecordReader) - Exclude guice-servlet, netty-all, hadoop-yarn-client transitive deps that fail the license checker - Remove parquet-format-structures (no longer needed) - Default enableParquetInternalReadbackVerification to false (opt-in) - Add testParquetReadBackVerificationCatchesRowCountMismatch test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
analyze-only rejects compile-scope deps that aren't directly imported in source. The dep is only needed at runtime (JVM class loading of InternalParquetRecordReader's superclass RecordReader) so runtime scope is correct and exempt from the unused-declared check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed JAR The shade plugin bundles all runtime deps. hadoop-mapreduce-client-core includes mapred-default.xml and other resources at non-snowflake paths which fail the check_content.sh namespace check. Add per-artifact shade filters: include only org/apache/hadoop/mapreduce/*.class (RecordReader and peer types) and exclude hadoop-hdfs-client entirely (an unneeded transitive). This avoids bundling resource files or HDFS classes while keeping the classes needed for InternalParquetRecordReader's class loading. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add testFlushRows benchmark to measure serialize+verify cost with enableReadbackVerification=false vs true - Move fillBuffer into auxiliary FlushFixture @State so it only runs before testFlushRows, not testInsertRow - Promote enableIcebergStreaming to JMH @PARAM so JMH forks cover both FDN and Iceberg (previously the JUnit @parameterized value was lost in the forked JVM) - Register channel in ChannelCache so setNeedFlush works when the Iceberg buffer exceeds the max chunk size threshold Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sfc-gh-psaha
left a comment
There was a problem hiding this comment.
Let's do a more robust benchmark with a wider schema. Also, the numbers look scary when looking at just flush in an isolated fashion. Can you do a benchmark against preprod6 for the full ingestion and measure differences in client side throughput (try to push as fast as possible and see how much we can support) and cpu usage with flag on and off? You can use a cloud workspace that's not doing anything else. Single client, single channel, use LexisNexis table schema.
| public static final String MAX_REGISTRATION_QUEUE_SIZE = | ||
| "MAX_REGISTRATION_QUEUE_SIZE".toLowerCase(); | ||
|
|
||
| public static final String ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION = |
There was a problem hiding this comment.
What's the "Internal" referring to?
There was a problem hiding this comment.
it was meant to refer to "compressed", but i removed it for simplicity
| * Verifies that all data in a parquet blob can be decompressed and has the expected row count. | ||
| * Catches compressed-data corruption (e.g. GZIP CRC32 failures) and row count mismatches. | ||
| */ | ||
| public static void verify(byte[] data, long expectedRowCount) throws IOException { |
There was a problem hiding this comment.
Would this catch the issue in https://snowflakecomputing.atlassian.net/browse/SNOW-3903306 as well? Have you verified?
There was a problem hiding this comment.
It depends on the root cause of the ticket.
If it's a file that originally had rows but was corrupted, then yes, the row counter assertion will catch a header corruption. I just added a test for verification.
If it was a valid 0 row write then no, this PR does not deal with it.
| try { | ||
| BdecParquetReader.verify(mergedData.toByteArray(), expectedRowCount); | ||
| } catch (IOException e) { | ||
| throw new SFException( |
There was a problem hiding this comment.
What is the high level behavior on this exception? Is the blob rebuilt or does the client just fail and give up? We'd ideally want the first behavior.
There was a problem hiding this comment.
It used to just give up, I just added retry logic
…OW-3903306 test - Rename ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION → ENABLE_PARQUET_READBACK_VERIFICATION (drop ambiguous "Internal") - verifyReadBack() now throws IOException directly instead of wrapping in SFException, so FlushService takes the correct invalidateAllChannelsInBlob path on failure - Wrap write+verify in a 3-attempt retry loop: transient memory corruption may produce clean output on re-serialization from the in-memory rows snapshot - Add testParquetPageHeaderNumValuesCorruptionCaughtByRowCountCheck: proves verify() catches SNOW-3903306-style bit-flip corruption in the Thrift page header num_values field (GZIP body left intact)
The new testParquetPageHeaderNumValuesCorruptionCaughtByRowCountCheck test
uses org.apache.parquet.format.{PageHeader,Util} from parquet-format-structures.
The dependency analysis plugin requires all used deps to be explicitly declared.
…ed JAR Declaring it as scope=test excluded it from the maven-shade-plugin output, causing ClassNotFoundException at runtime in the shaded e2e tests. Changed to compile scope (no scope tag) so it gets shaded in. Added to ignoredDependencies to suppress the analyze-only warning about a compile dep used only in test code.
sfc-gh-psaha
left a comment
There was a problem hiding this comment.
Can you add a test for the retry behavior?
| this.verifyRowCounts(parquetWriter, rowCount, channelsDataPerTable, rows.size()); | ||
| if (enableParquetInternalReadbackVerification) { | ||
| verifyReadBack(mergedData, rowCount); | ||
| IOException lastVerifyException = null; |
There was a problem hiding this comment.
nit: it'd be clearer to read if you branch off earlier
if (enableVerification) {
// loop and verify
} else {
write and move on.
}
Split the write+verify block in ParquetFlusher.serializeFromJavaObjects() into explicit if/else branches instead of a unified loop with a ternary maxAttempts. The if(enableParquetReadbackVerification) branch retries up to 3 times on IOException; the else branch writes once with no retry overhead. Add testParquetReadBackVerificationRetrySucceeds to RowBufferTest: subclasses ParquetFlusher with enableParquetReadbackVerification=true and overrides verifyReadBack() to throw on the first call, confirming that the retry path recovers and returns a valid SerializationResult.
| } | ||
| } | ||
| if (actualRowCount != expectedRowCount) { | ||
| throw new IOException( |
There was a problem hiding this comment.
Why IOException instead of SF exception? can you check if we catch and re-throw later?
| col.setLength(11); | ||
| col.setScale(0); | ||
| buffer.setupSchema(Collections.singletonList(col)); | ||
| loadData(buffer, Collections.singletonMap("C1", "hello")); |
There was a problem hiding this comment.
let's add a test with more than 1 row of data
…ust IOException The else branch previously threw SFException (e.g. from verifyRowCounts) without calling invalidateAllChannelsInBlob. The future would complete exceptionally, RegisterService would swallow it, and the affected channels would stay alive with their rows silently lost. The error message already said "all channels will be invalidated" but only the IOException path actually did it. Collapse IOException and the catch-all into the same invalidation path: any unexpected exception during blob building invalidates the channels and returns null, so the customer gets SFException(INVALID_CHANNEL) on their next insertRow. Only the crypto exceptions (MD5, encryption) still propagate, as those indicate unrecoverable configuration failures.
… with verifyRowCounts
Summary
JIRA: SNOW-3883414
Add SSv1 SDK Parquet compression read-back verification
Corrupted BDEC files cause XP workers to crash with an unrecoverable FATAL assert at scan time. This adds BdecParquetReader.verify(), called on the serialize path when ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION=true, which fully decompresses and reads back the serialized bytes to surface compression corruption before the file is committed.
Performance
Benchmarked the flag against preprod6 using a single client and channel, two schema configurations, with a 30s warmup and 60s measurement window per pass. Showing 10-run averages.
Narrow (17 columns)
Wide (1151 columns)