Fix potential integer overflow in rowBytes multiplications#3053
Open
rootvector2 wants to merge 2 commits intoAOMediaCodec:mainfrom
Open
Fix potential integer overflow in rowBytes multiplications#3053rootvector2 wants to merge 2 commits intoAOMediaCodec:mainfrom
rootvector2 wants to merge 2 commits intoAOMediaCodec:mainfrom
Conversation
Cast the first operand to (size_t) before multiplying two uint32_t values involving rowBytes, alphaRowBytes, or yuvRowBytes to prevent unsigned integer wrap-around on large images.
y-guyon
approved these changes
Feb 23, 2026
src/reformat.c
Outdated
| assert(src->format != AVIF_RGB_FORMAT_RGB_565 || src->depth == 8); | ||
|
|
||
| const uint8_t * const srcPixel = &src->pixels[y * src->rowBytes + x * info->pixelBytes]; | ||
| const uint8_t * const srcPixel = &src->pixels[(size_t)y * src->rowBytes + x * info->pixelBytes]; |
Contributor
There was a problem hiding this comment.
Could be (size_t)x too I guess
Contributor
Author
|
I've updated the arithmetic to ensure all intermediate multiplications are performed in size_t and added explicit overflow checks before allocation in codec_svt.c. In reformat.c, both x and y are now promoted to size_t before multiplication to avoid any intermediate 32-bit overflow. This keeps the calculations fully in the size_t domain and prevents potential wraparound prior to allocation or pointer offset computation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cast the first operand to (size_t) before multiplying two uint32_t values involving rowBytes, alphaRowBytes, or yuvRowBytes to prevent unsigned integer wrap-around on large images.