Valid CHDv5 images with RLE maps rejected by `totalhunks > file_size * 8` check

[seanbrar]

In header_read() (src/libchdr_chd.c), commit 5cc52fd introduced an allocation guard:

/* totalhunks is used to size the map allocation; a malformed header
 * can otherwise request multi-GB allocations for map[] even when the
 * file itself is tiny. Every hunk map entry consumes at least one bit
 * in the compressed on-disk map, so totalhunks cannot legitimately
 * exceed file_size * 8. */
if ((uint64_t)header->totalhunks > chd->file_size * 8)
    return CHDERR_INVALID_DATA;

While the intent to guard against malformed headers driving multi-GB allocations during fuzzing is completely sound, the assumption that "every hunk map entry consumes at least one bit in the compressed on-disk map" doesn't hold for CHDv5.

Because CHDv5 uses run-length encoding (RLE) and Huffman compression for map entries (specifically for repeated, uncompressed, or self-referential hunks), long runs of identical hunks can be represented in far fewer bits than 1 bit per hunk. As a result, valid, highly compressed or sparse CHDv5 images can legitimately have totalhunks > file_size * 8 and are incorrectly rejected with CHDERR_INVALID_DATA by chd_open().

Steps to Reproduce

This can be reproduced without any proprietary data using chdman (tested on 0.242+) and a zero-filled raw image:

# 1. Create a 64 MiB zero-filled image
python3 -c 'with open("input.raw", "wb") as f: f.truncate(64 * 1024 * 1024)'

# 2. Compress to CHDv5 (4096-byte hunks, 2048-byte units, zlib)
chdman createraw -i input.raw -o output.chd -hs 4096 -us 2048 -c zlib

# 3. Verify with chdman (succeeds)
chdman verify -i output.chd

# 4. Open with libchdr
# Result: chd_open fails with CHDERR_INVALID_DATA

In this 64 MiB example:

• hunkbytes: 4096
• totalhunks: 16,384
• Output .chd size: 1,163 bytes
• file_size * 8: 9,304 bits

Because 16,384 > 9,304, header_read() rejects the valid file.

Real-World / Scale Impact

I ran into this while testing large disc image (Blu-ray scale) support in RPCS3. For a sparse 50 GB logical disc image, chdman generates ~12.2M hunks compressed into ~23 KB. Under the current check, any 50 GB image smaller than ~1.5 MB is rejected, even though the header, map, and payload are completely valid.

As a diagnostic check, appending trailing zero padding to the 23 KB file to bring it above 1.5 MB allows chd_open() to succeed and all data to be read and verified without error, confirming that the map decoding logic itself handles the RLE map cleanly.

Suggested Direction

The allocation guard is still important to prevent malicious/corrupt headers requesting tens of gigabytes of RAM during fuzzing. However, rather than checking totalhunks > file_size * 8, would it make sense to enforce an explicit reasonable upper bound on map allocation size or logical hunks, so that valid RLE-compressed maps aren't rejected?

[rtissera]

Good catch, on it.

[rtissera]

Fixed in #191: the guard now bounds what is actually allocated — an exact file-size check for v1-v4, an explicit cap on the materialized map for v5 — instead of assuming a minimum of one bit per hunk, which run-length and Huffman coded v5 maps do not respect.