Skip to content

Fix int32 overflow in decompress() for frames > 256 MiB#21

Open
rasmusfaber wants to merge 1 commit into
101arrowz:masterfrom
rasmusfaber:fix_large_frames
Open

Fix int32 overflow in decompress() for frames > 256 MiB#21
rasmusfaber wants to merge 1 commit into
101arrowz:masterfrom
rasmusfaber:fix_large_frames

Conversation

@rasmusfaber
Copy link
Copy Markdown

fzstd.decompress() throws invalid zstd data (or silently corrupts output) on any zstd frame whose compressed size exceeds 256 MiB.

Root cause: bit positions are computed as bt << 3, where bt is a byte offset that grows across blocks within a frame. << coerces to int32, so at bt = 2**28 it wraps:

(1 << 28) << 3  →  -2147483648    // int32 wrap
(1 << 28) * 8   →   2147483648    // correct

The paired >> 3 then reads the wrong bytes and decoding derails. Four sites in src/index.ts use this pattern; the streaming Decompress class is unaffected because it resets the offset per chunk.

Fix

Replace x << 3 with x * 8 so the bit position is a float64 that doesn't wrap, and replace the paired x >> 3 with Math.floor(x / 8) (14 sites). x & 7 stays as-is, int32 coercion preserves the low three bits.

Math.floor(x / 8) is chosen over x / 8 | 0 because | 0 would have wrapped negative once the quotient reached 2³¹ (compressed byte offset >= 2 GiB), introducing a secondary overflow within the 4 GiB Uint8Array ceiling. Math.floor is correct for any frame size.

Performance

No measurable regression. On a 200 MiB match-heavy frame, 21 alternated runs per build: +0.44% (inside noise). V8 lowers Math.floor(x / 8) to the same integer shift as >> 3 on SMI operands, and the surrounding inner loop is memory-bound.

End-to-end check on a real 1.38 GiB JSON payload (1.03 GiB compressed): pre-fix throws after 2.6 s; post-fix decompresses in ~11 s, sha256 matches the original.

Test

tests/large_frame_overflow_test.ts round-trips a 500 MiB match-heavy corpus through a single frame > 256 MiB,:

deno test --no-check --allow-run=zstd \
  --v8-flags=--max-old-space-size=6144 tests/large_frame_overflow_test.ts

Fails before this change (invalid zstd data), passes after. Runs in ~7 s.
Requires zstd on PATH.

I am happy to remove the test again, but I thought it useful to demonstrate the problem and verify the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant