Skip to content

GH-55606: [C++] Detect overflow when parsing decimals from strings - #55619

Open
csarushan1729 wants to merge 1 commit into
apache:mainfrom
csarushan1729:GH-55606
Open

csarushan1729 wants to merge 1 commit into
apache:mainfrom
csarushan1729:GH-55606

Conversation

@csarushan1729

@csarushan1729 csarushan1729 commented Nov 15, 2026 •

Copy link
Copy Markdown

Rationale for this change

When Arrow turns a string into a decimal (Decimal32/64/128/256::FromString), it builds the number in a fixed-size integer. If the value is too big for that integer, it wraps around, like a car odometer going from 99999 back to 00000. The cast code then checks the precision on the wrapped value, and it often looks fine. So cast(string -> decimal) gives a wrong number with no error. The CSV reader and anything else that uses FromString has the same problem.

Some examples of what came back before:

input target result before
39 nines decimal128(38, 0) -20847100762815390390123822295304634369
2**64 decimal64(18, 0) 0
2**256 + 7 decimal256(76, 0) 7
2**128 - 1 decimal128(38, 0) -1
"99e38" decimal128(38, 0) 31811359292784559562136384478721867776
"9e9" decimal32(9, 0) 410065408

There are three places where it can wrap:

  1. While adding up the digits. ShiftAndAdd was dropping the carry out of the top word.
  2. While scaling up for a positive exponent (like "99e38"). The multiply was not checked.
  3. When the number is between 2^(N-1) and 2^N for 128/256-bit decimals. It sets the sign bit, so the value comes out negative.

decimal_test.cc already had these cases written, but commented out with a note saying overflows are not detected yet.

What changes are included in this PR?

  • ShiftAndAdd now returns false if there is a carry out of the top word. The multiply-and-add step is moved into a new function, MultiplyAndAdd.
  • The scale-up for a positive exponent is now done on the unsigned value (MultiplyByPowerOfTen), before the sign is applied, so it gets the same overflow check. The existing -parsed_scale > kMaxScale check now happens earlier. The error is the same as before.
  • For Decimal128 and Decimal256, a new check (FitsInSignedWords) rejects values that don't fit the signed type. The smallest negative number (-2^(N-1)) is still allowed, because Decimal128Test.PrintMinValue needs it.
  • Decimal32 and Decimal64 keep their old "value is bigger than max" check. It now also catches values that would have wrapped the uint64_t accumulator.

All overflow cases give the same error as before: Invalid: The string '...' cannot be represented as decimalNNN.

Are these changes tested?

Yes.

  • I turned on the commented-out "XXX" overflow cases in Decimal128Test.FromStringLimits and Decimal256Test.FromStringLimits.
  • I added boundary checks. For 128-bit: 2^127 - 1 and -2^127 parse fine, while 2^127, -2^127 - 1 and 2^128 fail. For 256-bit: 2^255 - 1 parses, while 2^255 and 2^256 + 7 fail.
  • I added new tests Decimal32Test.FromStringOverflow and Decimal64Test.FromStringOverflow.
  • I added Cast.StringToDecimalOverflow, which uses the examples from the issue.

The new tests fail without the change in decimal.cc and pass with it. I ran these on Linux (Ubuntu under WSL, GCC, Boost 1.90), and all passed:

  • arrow-utility-test --gtest_filter=*Decimal* (327 tests)
  • arrow-compute-scalar-cast-test --gtest_filter=*Decimal* (26 tests)

Are there any user-facing changes?

Yes. If a string is too big for the decimal type, users now get an error instead of a wrong number.

This PR contains a "Critical Fix". Before this, casting or parsing such strings silently gave wrong data, and the user had no way to know.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has no components, please add labels for components.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has no components, please add labels for components.

3 similar comments
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has no components, please add labels for components.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has no components, please add labels for components.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55606 has no components, please add labels for components.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant