Skip to content

feat(spark): add Spark-compatible ln, log, log10, log2 and log1p - #334

Draft
andygrove wants to merge 7 commits into
apache:mainfrom
andygrove:feat-spark-log-functions
Draft

andygrove wants to merge 7 commits into
apache:mainfrom
andygrove:feat-spark-log-functions

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

These names currently resolve to DataFusion's built-ins, which differ from Spark:

Query Before Spark 4.2.0
ln(0), log10(0) -Infinity NULL
ln(-1), log2(-1) NaN NULL
log(2.0) (one argument) 0.301… (base 10) 0.6931471805599453 (natural log)
log(0, 1) -0 NULL
log(3, 81) 4 4.000000000000001
log1p(x) function not found ln(1 + x)

What changes are included in this PR?

  • math/log.rs: ln, log (one or two arguments), log10, log2 and log1p over DOUBLE, with numeric arguments cast to DOUBLE. Like Spark's UnaryLogExpression and Logarithm, an input at or below the asymptote (0, or −1 for log1p) returns NULL, and so does a base ≤ 0. NaN is not below the asymptote, so it returns NaN. log2 is ln(x) / ln(2), as in Spark, so log2(2^29) is 29.000000000000004. log(1, x) divides by zero and returns ±Infinity or NaN.
  • math/fdlibm.rs: Spark computes these with java.lang.StrictMath, which the Java spec requires to return exactly fdlibm's results. Rust's f64::ln calls the platform libm. That differs from fdlibm in the last bit on about 2% of inputs (about 5% for log10), and differently on macOS and Linux. The libm crate follows musl, which simplified fdlibm's final steps and also rounds differently. This file is a direct translation of fdlibm 5.3's e_log.c, e_log10.c and s_log1p.c from C to Rust. Constants are written as the bit patterns given in fdlibm's comments.
  • Licensing: the code is translated from Sun's original fdlibm C sources. Nothing comes from OpenJDK. Sun's notice ("Permission to use, copy, modify, and distribute this software is freely granted, provided that this notice is preserved") is kept verbatim in the file and added to LICENSE.txt. Reviewers, please confirm this is acceptable under ASF policy.
  • Each function is documented with #[user_doc].

Are these changes tested?

  • Ground truth:
    • The fdlibm translation returns the same bits as StrictMath.log, log10 and log1p on 2,000,000 random inputs: random bit patterns, [0, 100), (−1, 1) and integers. The StrictMath outputs were identical on JDK 17 and JDK 21.
    • The NULL/NaN semantics and the exact values (log(3, 81), log(10, 1000) = 2.9999999999999996, log2(2^29), ln(2)) were confirmed by evaluating Spark 4.2.0's Log, Log2, Log10, Log1p and Logarithm from the catalyst jar, through both the interpreted and generated-code paths.
  • Unit tests: pin the StrictMath bits at inputs where Rust's std differs, plus special values.
  • .slt files: the five placeholder files are now real tests, covering integer, float and decimal inputs, NULL, NaN, ±Infinity, subnormals, values ≤ 0, scalar and array paths, and inputs where f64::ln, f64::log10 and f64::ln_1p give a different last digit.
  • cargo fmt --all -- --check and cargo clippy --all-targets --all-features -- -D warnings: clean
  • cargo test -p datafusion-spark: 338 passed
  • cargo test -p datafusion-spark-sqllogictest: 249 of 249 files passed

Are there any user-facing changes?

Yes. ln, log, log10 and log2 now follow Spark instead of DataFusion's built-ins, and log1p is new. String arguments are cast to DOUBLE, as in Spark.

@andygrove
andygrove force-pushed the feat-spark-log-functions branch 2 times, most recently from 5b00309 to e18bd6d Compare November 16, 2026 16:24
@andygrove
andygrove requested a review from kosiew November 17, 2026 13:21

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. Matching StrictMath bit for bit is the right call here. I compared fdlibm.rs against JDK 17's StrictMath on 600,000 more random inputs (random bit patterns, subnormals, (-1, 1), near 1, and the expm1 range) and found no mismatches in log, log10, log1p, or expm1.

This also implements #266, which asks for log1p and lists these edge cases. Could the description say Closes #266?

Comment thread LICENSE.txt
Comment on lines +203 to +214

--------------------------------------------------------------------------------

This product includes code translated from fdlibm 5.3 (Freely Distributable
LIBM), in crates/spark/src/function/math/fdlibm.rs, under the following terms:

Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved.

Developed at SunSoft, a Sun Microsystems, Inc. business.
Permission to use, copy, modify, and distribute this
software is freely granted, provided that this notice
is preserved.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewers, please confirm this is acceptable under ASF policy.

The fdlibm notice is the SPDX SunPro license, which isn't named in the ASF's Category A list. There is precedent: Lucene ships fdlibm-derived code with the Apache header on top and the Sun notice in a comment (SloppyMath.java). The ASF header policy also says not to put the Apache header on third-party source files, and it leaves major modifications to the PMC. A line-by-line translation falls between those two cases.

Could you ask on legal-discuss@ whether SunPro is Category A and which header fdlibm.rs should carry, and link the answer here? I'd like that settled before this merges.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this. I'll ask on legal-discuss@ and link the thread here once there's an answer.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, while we wait on legal-discuss@: arrow already depends on code under this same notice. arrow-array enables half's num-traits feature, which enables num-traits' libm feature. Fifty of libm's source files carry the Sun notice, and its LICENSE reproduces it. That is a dependency rather than code in our source release, so it does not settle the question here. I also checked whether we could use libm instead. It differs from StrictMath on about 1% of log inputs and 11% of log10 inputs, but it matches expm1 exactly.

Comment thread crates/spark/src/function/math/log.rs Outdated
) -> Result<ArrayRef> {
let [arg] = take_function_args(name, args)?;
let result: Float64Array = arg
.as_primitive::<Float64Type>()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as_primitive panics if the array isn't Float64. SQL always coerces first, but a caller that builds the physical expression directly, as Comet does, can pass another type. With an Int32 array, SparkLn::new().invoke_with_args(...) panics with primitive array (arrow-array 60.1.0 cast.rs:898). spark_log has the same downcasts at lines 298-299. Could these return an error instead, the way expm1.rs does with unsupported_data_type_exec_err?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3cbe805. Both unary_log and spark_log now return unsupported_data_type_exec_err for a non-Float64 array, and non_float64_arguments_error covers each path.

Comment thread crates/spark/src/function/math/log.rs Outdated
Comment on lines +61 to +64
signature: Signature::exact(
vec![DataType::Float64],
Volatility::Immutable,
),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark implicitly casts a string argument to DOUBLE here (UnaryMathExpression has ImplicitCastInputTypes with Seq(DoubleType)). So SELECT ln('10') returns 2.302585092994046 in Spark 4.2.0 with ANSI mode on or off, and log('2', '8') returns 3.0. Here both fail at planning with coercion from Utf8 to the signature Exact(Float64) failed. #266 also lists STRING arguments among the cases to test.

Could the signatures accept strings the way date_from_unix_date.rs does, with Coercion::new_implicit and TypeSignatureClass::Native(logical_string())? With ANSI mode off, a malformed string returns NULL in Spark but errors here. That case could be a commented-out query citing #88, as in date_from_unix_date.slt.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3cbe805. All five functions now use Coercion::new_implicit with logical_string() among the allowed sources. ln.slt and log.slt cover string arguments and the ANSI error for a malformed string. The ANSI-off NULL case is commented out, citing #88.

Comment thread crates/spark/src/function/math/log.rs Outdated
Comment on lines +182 to +186
// Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
// base 2 logarithm, which differs in the last bit for some inputs.
unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| {
fdlibm::log(x) / fdlibm::log(2.0)
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fdlibm::log(2.0) is recomputed for every row. StrictMath.log(2.0) is exactly LN_2: the 2.0 row in fdlibm::tests expects 4604418534313441775, which is LN_2.to_bits(). So the constant can replace the call, with use std::f64::consts::LN_2; at the top. An assert_eq!(log(2.0), LN_2) in fdlibm::tests would check that equivalence by name.

Suggested change
// Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
// base 2 logarithm, which differs in the last bit for some inputs.
unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| {
fdlibm::log(x) / fdlibm::log(2.0)
});
// Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
// base 2 logarithm, which differs in the last bit for some inputs.
// `StrictMath.log(2.0)` is exactly `LN_2`.
unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| fdlibm::log(x) / LN_2);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3cbe805, with log_of_two_is_ln_2 in fdlibm::tests.

Comment on lines +297 to +313
[base, value] => {
let base = base.as_primitive::<Float64Type>();
let value = value.as_primitive::<Float64Type>();
let result: Float64Array = base
.iter()
.zip(value.iter())
.map(|(base, value)| match (base, value) {
// NaN is not <= 0, so a NaN argument yields NaN
// unless the other argument is <= 0.
(Some(base), Some(value)) if base <= 0.0 || value <= 0.0 => None,
(Some(base), Some(value)) => {
Some(fdlibm::log(value) / fdlibm::log(base))
}
_ => None,
})
.collect();
Ok(Arc::new(result))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a literal base, which is how log is usually called, make_scalar_function expands the base into an array, so fdlibm::log(base) runs on the same value for every row. Could invoke_with_args handle a scalar base before make_scalar_function? It would compute fdlibm::log(base) once, map value with unary_opt, and return all NULLs when the base is NULL or <= 0.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3cbe805. A scalar base with an array expr now takes fdlibm::log(base) once and maps value with unary_opt. A NULL or non-positive base returns all NULLs. A new query covers NULL, 0, negative and NaN bases.

Comment on lines +96 to +102
# Scalar base with an array expr
query R
SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x);
----
3
NULL
NULL

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file covers a scalar base with an array expr, but not an array base with a scalar expr. Could we add that case?

Suggested change
# Scalar base with an array expr
query R
SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x);
----
3
NULL
NULL
# Scalar base with an array expr
query R
SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x);
----
3
NULL
NULL
# Array base with a scalar expr
query R
SELECT log(b, 8::double) FROM (VALUES (2::double), (0::double), (NULL::double)) AS t(b);
----
3
NULL
NULL

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 3cbe805.


# -0.0 is above the asymptote, and log1p(-0.0) is -0.0
query R
SELECT log1p(-0.0::double);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Spark, -0.0 is a DECIMAL literal, and decimals have no negative zero. So -0.0::double is 0.0, and Spark 4.2.0 returns 0.0 for this query, not -0.0. The PySpark runner flags this line. '-0.0'::double is -0.0 in both engines. ln.slt line 43 has the same ln(-0.0::double), which tests 0.0 in Spark.

Suggested change
SELECT log1p(-0.0::double);
SELECT log1p('-0.0'::double);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3cbe805, along with the same literal in ln.slt.

@andygrove

Copy link
Copy Markdown
Member Author

Thanks for the extra fdlibm comparison. Updated the description to Closes #266.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. I checked the string arguments against PySpark 4.2.0 in both ANSI modes, and they match except for the case below. The fdlibm license question is still open in its thread.

Comment on lines +96 to +101
# Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors
# https://github.com/apache/datafusion-spark/issues/88
# query R
# SELECT ln('abc');
# ----
# NULL

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark parses a string with Java's Double.valueOf (Cast.scala L2237-L2240), which accepts a trailing d, D, f, or F. In PySpark 4.2.0, ln('10d') and ln('10F') return 2.302585092994046 with ANSI mode on and off. Here ln('10d'), ln('10D'), ln('10F'), and ln('1.5f') fail in both modes with Cannot cast string '10d' to value of Float64 type. Comet's string-to-float cast strips the suffix (string.rs L253-L258), so the port in #88 should cover it.

Could we record this next to the malformed-string case, and name it in the module doc in log.rs (lines 19-21) along with the ANSI-off NULL difference?

Suggested change
# Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors
# https://github.com/apache/datafusion-spark/issues/88
# query R
# SELECT ln('abc');
# ----
# NULL
# Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors
# https://github.com/apache/datafusion-spark/issues/88
# query R
# SELECT ln('abc');
# ----
# NULL
# Spark parses strings with Java's Double.valueOf, which accepts a d or f suffix in both
# ANSI modes; DataFusion errors
# https://github.com/apache/datafusion-spark/issues/88
# query R
# SELECT ln('10d');
# ----
# 2.302585092994046

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. e520f45 names both string differences in the module doc in log.rs. Since #368 landed I've recorded them as live records with # xfail spark: markers linking #88, rather than commented-out queries. ln('abc') with ANSI mode off and ln('10d') both assert DataFusion's current cast error, so the PySpark runner tracks them and will flag them once #88 fixes the cast. I also rebased onto main now that #333 has merged.

@andygrove
andygrove marked this pull request as draft November 18, 2026 16:40
Spark returns NULL for an input at or below the asymptote (0, or -1 for
log1p) where DataFusion's built-ins return -Infinity or NaN, and Spark's
one-argument log is the natural logarithm rather than base 10.

Spark computes these with java.lang.StrictMath, which is fdlibm. Add a
translation of fdlibm 5.3's e_log.c, e_log10.c and s_log1p.c, keeping
Sun's notice and adding it to LICENSE.txt, so the results match Spark
bit for bit. Rust's f64::ln differs in the last digit for some inputs.
Spark computes expm1 with java.lang.StrictMath, which is fdlibm. f64::exp_m1
calls the platform's libm, so expm1(1) gave 1.7182818284590453 on macOS where
Spark and Linux give 1.718281828459045. Translate fdlibm 5.3's s_expm1.c next
to the logarithms. It matches StrictMath.expm1 bit for bit on 1.24 million
inputs on JDK 17.
… non-Float64 arrays

Spark implicitly casts numeric and string arguments to DOUBLE. Return an
error instead of panicking when a caller passes a non-Float64 array,
divide log2 by LN_2, take the logarithm of a scalar base once, and fix
the -0.0 literals that Spark parses as DECIMAL 0.0.
… suffix

`cargo doc` with `-D warnings` failed because the public `log` module and
`SparkExpm1` linked to the private `fdlibm` module. Name it in plain text.

Spark parses a string argument with Java's `Double.valueOf`, which accepts a
trailing d, D, f or F in both ANSI modes. The cast DataFusion uses here errors
instead. Record the difference in the `log` module doc and as a commented-out
query in ln.slt, next to the ANSI-off NULL case, both waiting on apache#88.
Turn the commented-out ln('abc') (ANSI off) and ln('10d') records into live
records that assert DataFusion's current cast error, each with a
# xfail spark: marker that links apache#88.
@andygrove
andygrove force-pushed the feat-spark-log-functions branch from 94b6c4c to f1c3998 Compare November 19, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible log1p

2 participants