OverflowError in evaluate_vectorized when masking uint8 arrays with 0xffff

[antonyef]

When running NIST statistical tests (specifically avalanche or correlation types) on ciphers that utilize the vectorized byte evaluator (e.g., PRESENT), an OverflowError occurs.

[peacker]

Hi @antonyef, thanks for creating the issue!
Unfortunately, the NIST Statistical Tests module has often caused trouble, as it uses the original NIST C implementation, which generates several files from which we extract the output. We are considering reimplementing the statistical tests in Python to better integrate them into the CLAASP framework.
You can check the branch https://github.com/Crypto-TII/claasp/tree/feature/python-nist-sts-implementation, where the tests are now in Python, and your issue should be fixed.

Below are more details on the fix:

Fixed

The OverflowError has been resolved in commit 490eeb8 on branch feature/python-nist-sts-implementation.

Root Cause:
The code used 0xffff (65535) as a mask for uint8 numpy arrays, which can only hold values 0-255. This caused an overflow when applying the mask with the &= operator.

Solution:
Changed the mask from 0xffff to 0xff (255) in generic_functions_vectorized_byte.py at line 154:

Before:
left_byte_mask = 0xffff # 65535 - causes overflow on uint8

After:
left_byte_mask = 0xff # 255 - correct for uint8

This ensures that when all 8 bits in a byte are used (number_of_output_bits % 8 == 0), the mask correctly preserves all bits without overflow.

Testing

All vectorized byte function tests pass: 16 passed in 0.46s

NIST statistical tests with PRESENT cipher (which uses a vectorized byte evaluator) now work correctly.

Files Changed

generic_functions_vectorized_byte.py

For the future, please copy-paste the full script and output instead of pasting a screenshot. This will make it easier to reproduce your code with a copy-paste rather than rewriting it by hand.