fix: format high-precision scientific numbers exactly - #1095
Open
He-Pin wants to merge 2 commits into
Open
Conversation
Motivation: Scientific formatting scaled binary64 values in Double and rounded through Long. Precision above 15 lost low-order digits, and precision 19 or greater could saturate at Long.MaxValue and silently replace the mantissa. Modification: Use the exact binary64 magnitude, BigInt scaling, half-even rounding, corrected decimal-exponent bounds, and carry handling for scientific precision above 15. Add regressions for precision 16 through 20, decade boundaries, ties, signed zero, and uppercase formatting. Result: High-precision %e/%E output no longer saturates and matches Python formatting from the same IEEE-754 value, including exact trailing digits. References: - databricks#1095 - https://jsonnet.org/ref/stdlib.html#std-format
He-Pin
force-pushed
the
fix/format-scientific-high-precision
branch
from
July 29, 2026 08:27
f291f79 to
eb3f89d
Compare
He-Pin
changed the title
Motivation: The original test only covered precision 16-20 with moderate values. Double.MaxValue, the smallest subnormal, and precision > 20 were verified manually but not regression-protected. Modification: Add assertions for Double.MaxValue (%.20e), smallest subnormal 5e-324 (%.20e), precision 40, precision 25, and pi at precision 30. Result: Extreme edge cases are now locked in against future regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The old scientific path scaled in
Doubleand rounded throughLong. Precision above 15 could lose meaningful binary64 digits; precision ≥19 saturated atLong.MaxValue, producing silent wrong answers (e.g.std.format("%.19e", 1.5)→1.000...e+00).Modification
BigDecimal.exact(number)withBigIntmantissa scaling.Longnarrowing.%E.Result
std.format("%.16e", 3.14159265358979323)3.1415926535897932e+003.1415926535897931e+003.1415926535897932e+003.1415926535897932e+003.1415926535897931e+00std.format("%.19e", 1.5)1.0000000000000000000e+001.5000000000000000000e+001.5000000000000000000e+001.5000000000000000000e+001.5000000000000000000e+00std.format("%.19e", 0.1)1.0000000000000000000e-011.0000000000000000555e-011.0000000000000000000e-011.0000000000000000000e-011.0000000000000000555e-01std.format("%.20e", 3.14)1.00000000000000000000e+003.14000000000000012434e+003.14000000000000000000e+003.09223372036854775807e+003.14000000000000012434e+00This PR matches Python's formatting model from the exact binary64 value. Full matrix (JVM/JS/Wasm/Native) +
checkFormatpassed.References
std.format