Repository navigation
GH-55606: [C++] Detect overflow when parsing decimals from strings - #55619
Open
csarushan1729 wants to merge 1 commit into
Open
csarushan1729 wants to merge 1 commit into
csarushan1729 wants to merge 1 commit into
Conversation
|
|
|
|
|
|
3 similar comments
|
|
|
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
When Arrow turns a string into a decimal (
Decimal32/64/128/256::FromString), it builds the number in a fixed-size integer. If the value is too big for that integer, it wraps around, like a car odometer going from 99999 back to 00000. The cast code then checks the precision on the wrapped value, and it often looks fine. Socast(string -> decimal)gives a wrong number with no error. The CSV reader and anything else that usesFromStringhas the same problem.Some examples of what came back before:
2**642**256 + 72**128 - 1"99e38""9e9"There are three places where it can wrap:
ShiftAndAddwas dropping the carry out of the top word."99e38"). The multiply was not checked.decimal_test.ccalready had these cases written, but commented out with a note saying overflows are not detected yet.What changes are included in this PR?
ShiftAndAddnow returnsfalseif there is a carry out of the top word. The multiply-and-add step is moved into a new function,MultiplyAndAdd.MultiplyByPowerOfTen), before the sign is applied, so it gets the same overflow check. The existing-parsed_scale > kMaxScalecheck now happens earlier. The error is the same as before.FitsInSignedWords) rejects values that don't fit the signed type. The smallest negative number (-2^(N-1)) is still allowed, becauseDecimal128Test.PrintMinValueneeds it.uint64_taccumulator.All overflow cases give the same error as before:
Invalid: The string '...' cannot be represented as decimalNNN.Are these changes tested?
Yes.
Decimal128Test.FromStringLimitsandDecimal256Test.FromStringLimits.Decimal32Test.FromStringOverflowandDecimal64Test.FromStringOverflow.Cast.StringToDecimalOverflow, which uses the examples from the issue.The new tests fail without the change in
decimal.ccand pass with it. I ran these on Linux (Ubuntu under WSL, GCC, Boost 1.90), and all passed:arrow-utility-test --gtest_filter=*Decimal*(327 tests)arrow-compute-scalar-cast-test --gtest_filter=*Decimal*(26 tests)Are there any user-facing changes?
Yes. If a string is too big for the decimal type, users now get an error instead of a wrong number.
This PR contains a "Critical Fix". Before this, casting or parsing such strings silently gave wrong data, and the user had no way to know.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: