Skip to content

GH-3404: fix CI TestByteBitPacking512VectorLE.unpackValuesUsingVector randomly oom - #3954

Merged
Fokko merged 1 commit into
apache:parquet-1.17.xfrom
adoroszlai:gh-3404-1.17.x
Nov 15, 2026
Merged

Fokko merged 1 commit into
apache:parquet-1.17.xfrom
adoroszlai:gh-3404-1.17.x

Conversation

@adoroszlai

Copy link
Copy Markdown

Rationale for this change

Backport #3405 (fix for flaky TestByteBitPacking512VectorLE.unpackValuesUsingVector) to 1.17.x.

What changes are included in this PR?

git cherry-pick -x efe71c218c9f6ba567f97c7476e7037a53b76333

Are these changes tested?

https://github.com/adoroszlai/parquet-java/actions/runs/43578780509
https://github.com/adoroszlai/parquet-java/actions/runs/43578780517

Are there any user-facing changes?

No.

@adoroszlai

Copy link
Copy Markdown
Author

@HuaHuaY @wgtmac please take a look

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai

@Fokko
Fokko merged commit ac81696 into apache:parquet-1.17.x Nov 15, 2026
5 checks passed
@adoroszlai
adoroszlai deleted the gh-3404-1.17.x branch November 15, 2026 07:31
@adoroszlai

Copy link
Copy Markdown
Author

Thanks @Fokko for reviewing and merging this.

@adoroszlai

Copy link
Copy Markdown
Author

Looks like #3405 was incomplete, the test is still failing in 12/100 runs on parquet-1.17.x. #3476 made further changes (73204ee) to fix the OOME as part of a bigger improvement:

    * Fix OOM in TestByteBitPacking512VectorLE on CI
    
    The 512-bit vector tests were allocating arrays of 268 million elements
    (~1 GB each) and iterating through the entire value space (up to 524K
    chunks for bitWidth=32). On the JDK 21 CI runner — which has AVX-512
    support so the tests are not skipped — the heap was exceeded, causing
    OutOfMemoryError in the DirectByteBuffer and ReadOnlyByteBuffer tests.
    
    Reduce itemMax from 268435456 to 8192 (still 512+ vector iterations per
    chunk) and only test the first and last chunks via a stream filter.
    This covers both low-range and high-range boundary values while keeping
    memory trivial (~33 KB per array) and runtime under 1 second.
    
    Assisted-by: GitHub Copilot:claude-opus-4.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants