Skip to content

[SNOW-3883414] parquet compression corruption check - #1162

Merged
sfc-gh-yelgabra merged 33 commits into
masterfrom
yelgabra-SNOW-3883414-parquet-compression-corruption-check
Aug 19, 2026
Merged

sfc-gh-yelgabra merged 33 commits into
masterfrom
yelgabra-SNOW-3883414-parquet-compression-corruption-check

Conversation

@sfc-gh-yelgabra

@sfc-gh-yelgabra sfc-gh-yelgabra commented Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

JIRA: SNOW-3883414

Add SSv1 SDK Parquet compression read-back verification

Corrupted BDEC files cause XP workers to crash with an unrecoverable FATAL assert at scan time. This adds BdecParquetReader.verify(), called on the serialize path when ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION=true, which fully decompresses and reads back the serialized bytes to surface compression corruption before the file is committed.

Performance

Benchmarked the flag against preprod6 using a single client and channel, two schema configurations, with a 30s warmup and 60s measurement window per pass. Showing 10-run averages.

Narrow (17 columns)

Metric No verify Verify Delta
BlobBuild latency 1,125 ms 1,327 ms +17.9%
Flush drain (total latency) 2,529 ms 2,699 ms +6.7%
CPU 163,099 ms 173,303 ms +6.3%

Wide (1151 columns)

Metric No verify Verify Delta
BlobBuild latency 1,348 ms 1,661 ms +23.2%
Flush drain (total latency) 2,656 ms 2,850 ms +7.3%
CPU 196,994 ms 214,868 ms +9.1%


[SNOW-3883414]: https://snowflakecomputing.atlassian.net/browse/SNOW-3883414?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

@sfc-gh-yelgabra
sfc-gh-yelgabra force-pushed the yelgabra-SNOW-3883414-parquet-compression-corruption-check branch from df692fc to 63b0d20 Compare August 7, 2026 19:27
sfc-gh-yelgabra and others added 7 commits August 7, 2026 13:45
ParquetReadOptions.builder() statically imports ParquetInputFormat.getFilter,
which forces loading of ParquetInputFormat (extends FileInputFormat from
hadoop-mapreduce-client-core). Our new verifyReadBack() calls this on the
production flush path, causing ClassNotFoundException in e2e jar tests where
only the production jar (without test-scoped deps) is on the classpath.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…k verification

BdecParquetReader loads InternalParquetRecordReader, which has static imports
from ParquetInputFormat (extends hadoop-mapreduce FileInputFormat). This
ClassNotFoundException breaks e2e jar tests where hadoop-mapreduce-client-core
is not on the runtime classpath.

New approach: parse the parquet footer with Util.readFileMetaData() and walk
each column chunk's pages with Util.readPageHeader() (both from
parquet-format-structures, already compile-scope). For GZIP pages, wrap the
compressed values in GZIPInputStream and exhaust it — this triggers the CRC32
check that catches VM-level compression corruption. DataPageV2 rep/def level
bytes are uncompressed and skipped before the GZIPInputStream.

No pom.xml changes needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move verification logic into BdecParquetReader.verify() static method
which uses the original while(reader.read()!=null) loop. Catch
NoClassDefFoundError to skip silently when hadoop-mapreduce-client-core
is absent (e.g. e2e-jar-test); full verification runs in production.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sfc-gh-yelgabra sfc-gh-yelgabra changed the title Yelgabra snow 3883414 parquet compression corruption check [SNOW-3883414] parquet compression corruption check Aug 7, 2026
sfc-gh-yelgabra and others added 6 commits August 7, 2026 16:13
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Counts rows actually read back and throws if the count doesn't match
the expected row count. This catches num_values=0 page corruption
(e.g. SNOW-3899515) in addition to the existing GZIP CRC32 check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…preduce-client-core

Rewrite BdecParquetReader.verify() to walk the parquet structure directly
using parquet-format-structures (compile-scoped) and decompress pages via
CodecFactory, removing the dependency on hadoop-mapreduce-client-core that
was causing NoClassDefFoundError → channel invalidation in environments
without the full Hadoop stack (e2e-jar-test, Kafka Connect).

Catches both known VM-corruption classes:
- Compressed data corruption (GZIP CRC32 failures): CodecFactory.decompress()
  throws on invalid DEFLATE blocks, mirroring C++ ParquetCompressedBuffer::decompress()
- Page header corruption (num_values=0): row count accumulated from first
  column's page headers and compared to expectedRowCount at the end

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dependency

Used directly in BdecParquetReader.verify() via Util, FileMetaData, PageHeader,
RowGroup, ColumnChunk, and ColumnMetaData. Previously only a transitive dependency
pulled in by parquet-hadoop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tures

Same exclusion already applied to parquet-common — CDDL + GPLv2 with
classpath exception license is not on the approved list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Aug 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.99%. Comparing base (38a6d6b) to head (265e3cf).

Files with missing lines Patch % Lines
...est/streaming/internal/ClientBufferParameters.java 80.00% 0 Missing and 1 partial ⚠️
.../net/snowflake/ingest/utils/ParameterProvider.java 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1162      +/-   ##
==========================================
+ Coverage   55.82%   59.99%   +4.16%     
==========================================
  Files         214      214              
  Lines       14616    14647      +31     
  Branches     1606     1611       +5     
==========================================
+ Hits         8160     8788     +628     
+ Misses       5859     5193     -666     
- Partials      597      666      +69     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sfc-gh-yelgabra and others added 6 commits August 11, 2026 16:38
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ote hadoop dep

- Replace Thrift-based verify() with simpler BdecParquetReader.read() loop
- Promote hadoop-mapreduce-client-core from test to compile scope (required
  because InternalParquetRecordReader extends hadoop's RecordReader)
- Exclude guice-servlet, netty-all, hadoop-yarn-client transitive deps that
  fail the license checker
- Remove parquet-format-structures (no longer needed)
- Default enableParquetInternalReadbackVerification to false (opt-in)
- Add testParquetReadBackVerificationCatchesRowCountMismatch test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
analyze-only rejects compile-scope deps that aren't directly imported in
source. The dep is only needed at runtime (JVM class loading of
InternalParquetRecordReader's superclass RecordReader) so runtime scope
is correct and exempt from the unused-declared check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed JAR

The shade plugin bundles all runtime deps. hadoop-mapreduce-client-core
includes mapred-default.xml and other resources at non-snowflake paths
which fail the check_content.sh namespace check.

Add per-artifact shade filters: include only org/apache/hadoop/mapreduce/*.class
(RecordReader and peer types) and exclude hadoop-hdfs-client entirely
(an unneeded transitive). This avoids bundling resource files or HDFS
classes while keeping the classes needed for InternalParquetRecordReader's
class loading.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add testFlushRows benchmark to measure serialize+verify cost with
  enableReadbackVerification=false vs true
- Move fillBuffer into auxiliary FlushFixture @State so it only runs
  before testFlushRows, not testInsertRow
- Promote enableIcebergStreaming to JMH @PARAM so JMH forks cover both
  FDN and Iceberg (previously the JUnit @parameterized value was lost
  in the forked JVM)
- Register channel in ChannelCache so setNeedFlush works when the
  Iceberg buffer exceeds the max chunk size threshold

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sfc-gh-yelgabra
sfc-gh-yelgabra marked this pull request as ready for review August 12, 2026 23:47
@sfc-gh-yelgabra
sfc-gh-yelgabra requested review from a team as code owners August 12, 2026 23:47
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@sfc-gh-psaha sfc-gh-psaha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do a more robust benchmark with a wider schema. Also, the numbers look scary when looking at just flush in an isolated fashion. Can you do a benchmark against preprod6 for the full ingestion and measure differences in client side throughput (try to push as fast as possible and see how much we can support) and cpu usage with flag on and off? You can use a cloud workspace that's not doing anything else. Single client, single channel, use LexisNexis table schema.

public static final String MAX_REGISTRATION_QUEUE_SIZE =
"MAX_REGISTRATION_QUEUE_SIZE".toLowerCase();

public static final String ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the "Internal" referring to?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was meant to refer to "compressed", but i removed it for simplicity

* Verifies that all data in a parquet blob can be decompressed and has the expected row count.
* Catches compressed-data corruption (e.g. GZIP CRC32 failures) and row count mismatches.
*/
public static void verify(byte[] data, long expectedRowCount) throws IOException {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this catch the issue in https://snowflakecomputing.atlassian.net/browse/SNOW-3903306 as well? Have you verified?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on the root cause of the ticket.
If it's a file that originally had rows but was corrupted, then yes, the row counter assertion will catch a header corruption. I just added a test for verification.
If it was a valid 0 row write then no, this PR does not deal with it.

try {
BdecParquetReader.verify(mergedData.toByteArray(), expectedRowCount);
} catch (IOException e) {
throw new SFException(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the high level behavior on this exception? Is the blob rebuilt or does the client just fail and give up? We'd ideally want the first behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to just give up, I just added retry logic

…OW-3903306 test

- Rename ENABLE_PARQUET_INTERNAL_READBACK_VERIFICATION →
  ENABLE_PARQUET_READBACK_VERIFICATION (drop ambiguous "Internal")
- verifyReadBack() now throws IOException directly instead of wrapping in
  SFException, so FlushService takes the correct invalidateAllChannelsInBlob
  path on failure
- Wrap write+verify in a 3-attempt retry loop: transient memory corruption
  may produce clean output on re-serialization from the in-memory rows snapshot
- Add testParquetPageHeaderNumValuesCorruptionCaughtByRowCountCheck: proves
  verify() catches SNOW-3903306-style bit-flip corruption in the Thrift page
  header num_values field (GZIP body left intact)
The new testParquetPageHeaderNumValuesCorruptionCaughtByRowCountCheck test
uses org.apache.parquet.format.{PageHeader,Util} from parquet-format-structures.
The dependency analysis plugin requires all used deps to be explicitly declared.
…ed JAR

Declaring it as scope=test excluded it from the maven-shade-plugin output,
causing ClassNotFoundException at runtime in the shaded e2e tests.
Changed to compile scope (no scope tag) so it gets shaded in.
Added to ignoredDependencies to suppress the analyze-only warning about
a compile dep used only in test code.

@sfc-gh-psaha sfc-gh-psaha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test for the retry behavior?

this.verifyRowCounts(parquetWriter, rowCount, channelsDataPerTable, rows.size());
if (enableParquetInternalReadbackVerification) {
verifyReadBack(mergedData, rowCount);
IOException lastVerifyException = null;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it'd be clearer to read if you branch off earlier

if (enableVerification) {
// loop and verify
} else {
write and move on.
}

Split the write+verify block in ParquetFlusher.serializeFromJavaObjects() into
explicit if/else branches instead of a unified loop with a ternary maxAttempts.
The if(enableParquetReadbackVerification) branch retries up to 3 times on
IOException; the else branch writes once with no retry overhead.

Add testParquetReadBackVerificationRetrySucceeds to RowBufferTest: subclasses
ParquetFlusher with enableParquetReadbackVerification=true and overrides
verifyReadBack() to throw on the first call, confirming that the retry path
recovers and returns a valid SerializationResult.
}
}
if (actualRowCount != expectedRowCount) {
throw new IOException(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why IOException instead of SF exception? can you check if we catch and re-throw later?

col.setLength(11);
col.setScale(0);
buffer.setupSchema(Collections.singletonList(col));
loadData(buffer, Collections.singletonMap("C1", "hello"));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add a test with more than 1 row of data

@sfc-gh-ggeng sfc-gh-ggeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

…ust IOException

The else branch previously threw SFException (e.g. from verifyRowCounts) without
calling invalidateAllChannelsInBlob. The future would complete exceptionally,
RegisterService would swallow it, and the affected channels would stay alive with
their rows silently lost. The error message already said "all channels will be
invalidated" but only the IOException path actually did it.

Collapse IOException and the catch-all into the same invalidation path: any
unexpected exception during blob building invalidates the channels and returns
null, so the customer gets SFException(INVALID_CHANNEL) on their next insertRow.
Only the crypto exceptions (MD5, encryption) still propagate, as those indicate
unrecoverable configuration failures.
@sfc-gh-yelgabra
sfc-gh-yelgabra merged commit 1a5dbc2 into master Aug 19, 2026
46 of 51 checks passed
@sfc-gh-yelgabra
sfc-gh-yelgabra deleted the yelgabra-SNOW-3883414-parquet-compression-corruption-check branch August 19, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants