Skip to content

fix(spark): compute expm1 with fdlibm's algorithm as Spark does - #359

Open
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:expm1-fdlibm
Open

andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:expm1-fdlibm

Conversation

@andygrove

@andygrove andygrove commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

expm1 called Rust's f64::exp_m1, which uses the platform's C math library. Spark's Expm1 calls Java's StrictMath.expm1, which returns fdlibm's result on every platform. On macOS the results differ in the last bit for some inputs, and math/expm1.slt fails:

Query Spark 4.2.0 and this PR Before, on macOS
expm1(1) 1.718281828459045 1.7182818284590453
expm1(-2.5D) -0.9179150013761013 -0.9179150013761012
expm1(0.9D) 1.4596031111569499 1.4596031111569496
expm1(1.7D) 4.473947391727199 4.4739473917272

What changes are included in this PR?

expm1 calls libm::expm1. The libm crate is a pure Rust port of musl's math library, which comes from fdlibm, so it gives Spark's result on every platform. libm was already in Cargo.lock as a transitive dependency, and is now a direct dependency of datafusion-spark.

Are these changes tested?

Yes. New records in math/expm1.slt cover the three inputs above, small and large inputs, and overflow to Infinity. The existing expm1(1) record and the new ones for -2.5, 0.9 and 1.7 fail on macOS without this change. I checked expm1.slt against Spark 4.2.0 with the PySpark runner, and every record matches. I also compared libm::expm1 with Spark for 19 more inputs, including -0.0, NaN and both infinities, and every result matches.

Are there any user-facing changes?

expm1 returns Spark's result on every platform.

This PR was prepared with AI assistance (Claude Code) and reviewed by me.

Spark's Expm1 calls Java's StrictMath.expm1, which returns fdlibm's
result. f64::exp_m1 calls the platform's C math library, which differs
in the last bit on macOS. Use the libm crate, a port of musl's math
library, which comes from fdlibm.

Closes apache#358.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

expm1 result depends on the platform's math library and differs from Spark on macOS

1 participant