Repository navigation
Conversation
5b00309 to
e18bd6d
Compare
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. Matching StrictMath bit for bit is the right call here. I compared fdlibm.rs against JDK 17's StrictMath on 600,000 more random inputs (random bit patterns, subnormals, (-1, 1), near 1, and the expm1 range) and found no mismatches in log, log10, log1p, or expm1.
This also implements #266, which asks for log1p and lists these edge cases. Could the description say Closes #266?
|
|
||
| -------------------------------------------------------------------------------- | ||
|
|
||
| This product includes code translated from fdlibm 5.3 (Freely Distributable | ||
| LIBM), in crates/spark/src/function/math/fdlibm.rs, under the following terms: | ||
|
|
||
| Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. | ||
|
|
||
| Developed at SunSoft, a Sun Microsystems, Inc. business. | ||
| Permission to use, copy, modify, and distribute this | ||
| software is freely granted, provided that this notice | ||
| is preserved. |
There was a problem hiding this comment.
Reviewers, please confirm this is acceptable under ASF policy.
The fdlibm notice is the SPDX SunPro license, which isn't named in the ASF's Category A list. There is precedent: Lucene ships fdlibm-derived code with the Apache header on top and the Sun notice in a comment (SloppyMath.java). The ASF header policy also says not to put the Apache header on third-party source files, and it leaves major modifications to the PMC. A line-by-line translation falls between those two cases.
Could you ask on legal-discuss@ whether SunPro is Category A and which header fdlibm.rs should carry, and link the answer here? I'd like that settled before this merges.
There was a problem hiding this comment.
Thanks for digging into this. I'll ask on legal-discuss@ and link the thread here once there's an answer.
There was a problem hiding this comment.
For context, while we wait on legal-discuss@: arrow already depends on code under this same notice. arrow-array enables half's num-traits feature, which enables num-traits' libm feature. Fifty of libm's source files carry the Sun notice, and its LICENSE reproduces it. That is a dependency rather than code in our source release, so it does not settle the question here. I also checked whether we could use libm instead. It differs from StrictMath on about 1% of log inputs and 11% of log10 inputs, but it matches expm1 exactly.
| ) -> Result<ArrayRef> { | ||
| let [arg] = take_function_args(name, args)?; | ||
| let result: Float64Array = arg | ||
| .as_primitive::<Float64Type>() |
There was a problem hiding this comment.
as_primitive panics if the array isn't Float64. SQL always coerces first, but a caller that builds the physical expression directly, as Comet does, can pass another type. With an Int32 array, SparkLn::new().invoke_with_args(...) panics with primitive array (arrow-array 60.1.0 cast.rs:898). spark_log has the same downcasts at lines 298-299. Could these return an error instead, the way expm1.rs does with unsupported_data_type_exec_err?
There was a problem hiding this comment.
Done in 3cbe805. Both unary_log and spark_log now return unsupported_data_type_exec_err for a non-Float64 array, and non_float64_arguments_error covers each path.
| signature: Signature::exact( | ||
| vec![DataType::Float64], | ||
| Volatility::Immutable, | ||
| ), |
There was a problem hiding this comment.
Spark implicitly casts a string argument to DOUBLE here (UnaryMathExpression has ImplicitCastInputTypes with Seq(DoubleType)). So SELECT ln('10') returns 2.302585092994046 in Spark 4.2.0 with ANSI mode on or off, and log('2', '8') returns 3.0. Here both fail at planning with coercion from Utf8 to the signature Exact(Float64) failed. #266 also lists STRING arguments among the cases to test.
Could the signatures accept strings the way date_from_unix_date.rs does, with Coercion::new_implicit and TypeSignatureClass::Native(logical_string())? With ANSI mode off, a malformed string returns NULL in Spark but errors here. That case could be a commented-out query citing #88, as in date_from_unix_date.slt.
| // Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true | ||
| // base 2 logarithm, which differs in the last bit for some inputs. | ||
| unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| { | ||
| fdlibm::log(x) / fdlibm::log(2.0) | ||
| }); |
There was a problem hiding this comment.
fdlibm::log(2.0) is recomputed for every row. StrictMath.log(2.0) is exactly LN_2: the 2.0 row in fdlibm::tests expects 4604418534313441775, which is LN_2.to_bits(). So the constant can replace the call, with use std::f64::consts::LN_2; at the top. An assert_eq!(log(2.0), LN_2) in fdlibm::tests would check that equivalence by name.
| // Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true | |
| // base 2 logarithm, which differs in the last bit for some inputs. | |
| unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| { | |
| fdlibm::log(x) / fdlibm::log(2.0) | |
| }); | |
| // Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true | |
| // base 2 logarithm, which differs in the last bit for some inputs. | |
| // `StrictMath.log(2.0)` is exactly `LN_2`. | |
| unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| fdlibm::log(x) / LN_2); |
There was a problem hiding this comment.
Done in 3cbe805, with log_of_two_is_ln_2 in fdlibm::tests.
| [base, value] => { | ||
| let base = base.as_primitive::<Float64Type>(); | ||
| let value = value.as_primitive::<Float64Type>(); | ||
| let result: Float64Array = base | ||
| .iter() | ||
| .zip(value.iter()) | ||
| .map(|(base, value)| match (base, value) { | ||
| // NaN is not <= 0, so a NaN argument yields NaN | ||
| // unless the other argument is <= 0. | ||
| (Some(base), Some(value)) if base <= 0.0 || value <= 0.0 => None, | ||
| (Some(base), Some(value)) => { | ||
| Some(fdlibm::log(value) / fdlibm::log(base)) | ||
| } | ||
| _ => None, | ||
| }) | ||
| .collect(); | ||
| Ok(Arc::new(result)) |
There was a problem hiding this comment.
With a literal base, which is how log is usually called, make_scalar_function expands the base into an array, so fdlibm::log(base) runs on the same value for every row. Could invoke_with_args handle a scalar base before make_scalar_function? It would compute fdlibm::log(base) once, map value with unary_opt, and return all NULLs when the base is NULL or <= 0.
There was a problem hiding this comment.
Done in 3cbe805. A scalar base with an array expr now takes fdlibm::log(base) once and maps value with unary_opt. A NULL or non-positive base returns all NULLs. A new query covers NULL, 0, negative and NaN bases.
| # Scalar base with an array expr | ||
| query R | ||
| SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x); | ||
| ---- | ||
| 3 | ||
| NULL | ||
| NULL |
There was a problem hiding this comment.
The file covers a scalar base with an array expr, but not an array base with a scalar expr. Could we add that case?
| # Scalar base with an array expr | |
| query R | |
| SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x); | |
| ---- | |
| 3 | |
| NULL | |
| NULL | |
| # Scalar base with an array expr | |
| query R | |
| SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) AS t(x); | |
| ---- | |
| 3 | |
| NULL | |
| NULL | |
| # Array base with a scalar expr | |
| query R | |
| SELECT log(b, 8::double) FROM (VALUES (2::double), (0::double), (NULL::double)) AS t(b); | |
| ---- | |
| 3 | |
| NULL | |
| NULL |
|
|
||
| # -0.0 is above the asymptote, and log1p(-0.0) is -0.0 | ||
| query R | ||
| SELECT log1p(-0.0::double); |
There was a problem hiding this comment.
In Spark, -0.0 is a DECIMAL literal, and decimals have no negative zero. So -0.0::double is 0.0, and Spark 4.2.0 returns 0.0 for this query, not -0.0. The PySpark runner flags this line. '-0.0'::double is -0.0 in both engines. ln.slt line 43 has the same ln(-0.0::double), which tests 0.0 in Spark.
| SELECT log1p(-0.0::double); | |
| SELECT log1p('-0.0'::double); |
There was a problem hiding this comment.
Fixed in 3cbe805, along with the same literal in ln.slt.
|
Thanks for the extra fdlibm comparison. Updated the description to |
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. I checked the string arguments against PySpark 4.2.0 in both ANSI modes, and they match except for the case below. The fdlibm license question is still open in its thread.
| # Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors | ||
| # https://github.com/apache/datafusion-spark/issues/88 | ||
| # query R | ||
| # SELECT ln('abc'); | ||
| # ---- | ||
| # NULL |
There was a problem hiding this comment.
Spark parses a string with Java's Double.valueOf (Cast.scala L2237-L2240), which accepts a trailing d, D, f, or F. In PySpark 4.2.0, ln('10d') and ln('10F') return 2.302585092994046 with ANSI mode on and off. Here ln('10d'), ln('10D'), ln('10F'), and ln('1.5f') fail in both modes with Cannot cast string '10d' to value of Float64 type. Comet's string-to-float cast strips the suffix (string.rs L253-L258), so the port in #88 should cover it.
Could we record this next to the malformed-string case, and name it in the module doc in log.rs (lines 19-21) along with the ANSI-off NULL difference?
| # Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors | |
| # https://github.com/apache/datafusion-spark/issues/88 | |
| # query R | |
| # SELECT ln('abc'); | |
| # ---- | |
| # NULL | |
| # Spark 4.2.0 returns NULL for a malformed string when ANSI mode is off; DataFusion errors | |
| # https://github.com/apache/datafusion-spark/issues/88 | |
| # query R | |
| # SELECT ln('abc'); | |
| # ---- | |
| # NULL | |
| # Spark parses strings with Java's Double.valueOf, which accepts a d or f suffix in both | |
| # ANSI modes; DataFusion errors | |
| # https://github.com/apache/datafusion-spark/issues/88 | |
| # query R | |
| # SELECT ln('10d'); | |
| # ---- | |
| # 2.302585092994046 |
There was a problem hiding this comment.
Thanks. e520f45 names both string differences in the module doc in log.rs. Since #368 landed I've recorded them as live records with # xfail spark: markers linking #88, rather than commented-out queries. ln('abc') with ANSI mode off and ln('10d') both assert DataFusion's current cast error, so the PySpark runner tracks them and will flag them once #88 fixes the cast. I also rebased onto main now that #333 has merged.
Spark returns NULL for an input at or below the asymptote (0, or -1 for log1p) where DataFusion's built-ins return -Infinity or NaN, and Spark's one-argument log is the natural logarithm rather than base 10. Spark computes these with java.lang.StrictMath, which is fdlibm. Add a translation of fdlibm 5.3's e_log.c, e_log10.c and s_log1p.c, keeping Sun's notice and adding it to LICENSE.txt, so the results match Spark bit for bit. Rust's f64::ln differs in the last digit for some inputs.
Spark computes expm1 with java.lang.StrictMath, which is fdlibm. f64::exp_m1 calls the platform's libm, so expm1(1) gave 1.7182818284590453 on macOS where Spark and Linux give 1.718281828459045. Translate fdlibm 5.3's s_expm1.c next to the logarithms. It matches StrictMath.expm1 bit for bit on 1.24 million inputs on JDK 17.
… non-Float64 arrays Spark implicitly casts numeric and string arguments to DOUBLE. Return an error instead of panicking when a caller passes a non-Float64 array, divide log2 by LN_2, take the logarithm of a scalar base once, and fix the -0.0 literals that Spark parses as DECIMAL 0.0.
… suffix `cargo doc` with `-D warnings` failed because the public `log` module and `SparkExpm1` linked to the private `fdlibm` module. Name it in plain text. Spark parses a string argument with Java's `Double.valueOf`, which accepts a trailing d, D, f or F in both ANSI modes. The cast DataFusion uses here errors instead. Record the difference in the `log` module doc and as a commented-out query in ln.slt, next to the ANSI-off NULL case, both waiting on apache#88.
Turn the commented-out ln('abc') (ANSI off) and ln('10d') records into live
records that assert DataFusion's current cast error, each with a
# xfail spark: marker that links apache#88.
94b6c4c to
f1c3998
Compare
Which issue does this PR close?
log1p#266.Rationale for this change
These names currently resolve to DataFusion's built-ins, which differ from Spark:
ln(0),log10(0)-InfinityNULLln(-1),log2(-1)NaNNULLlog(2.0)(one argument)0.301…(base 10)0.6931471805599453(natural log)log(0, 1)-0NULLlog(3, 81)44.000000000000001log1p(x)ln(1 + x)What changes are included in this PR?
math/log.rs:ln,log(one or two arguments),log10,log2andlog1pover DOUBLE, with numeric arguments cast to DOUBLE. Like Spark'sUnaryLogExpressionandLogarithm, an input at or below the asymptote (0, or −1 forlog1p) returns NULL, and so does a base ≤ 0. NaN is not below the asymptote, so it returns NaN.log2isln(x) / ln(2), as in Spark, solog2(2^29)is29.000000000000004.log(1, x)divides by zero and returns ±Infinity or NaN.math/fdlibm.rs: Spark computes these withjava.lang.StrictMath, which the Java spec requires to return exactly fdlibm's results. Rust'sf64::lncalls the platform libm. That differs from fdlibm in the last bit on about 2% of inputs (about 5% forlog10), and differently on macOS and Linux. Thelibmcrate follows musl, which simplified fdlibm's final steps and also rounds differently. This file is a direct translation of fdlibm 5.3'se_log.c,e_log10.cands_log1p.cfrom C to Rust. Constants are written as the bit patterns given in fdlibm's comments.LICENSE.txt. Reviewers, please confirm this is acceptable under ASF policy.#[user_doc].Are these changes tested?
StrictMath.log,log10andlog1pon 2,000,000 random inputs: random bit patterns, [0, 100), (−1, 1) and integers. TheStrictMathoutputs were identical on JDK 17 and JDK 21.log(3, 81),log(10, 1000) = 2.9999999999999996,log2(2^29),ln(2)) were confirmed by evaluating Spark 4.2.0'sLog,Log2,Log10,Log1pandLogarithmfrom the catalyst jar, through both the interpreted and generated-code paths.StrictMathbits at inputs where Rust's std differs, plus special values..sltfiles: the five placeholder files are now real tests, covering integer, float and decimal inputs, NULL, NaN, ±Infinity, subnormals, values ≤ 0, scalar and array paths, and inputs wheref64::ln,f64::log10andf64::ln_1pgive a different last digit.cargo fmt --all -- --checkandcargo clippy --all-targets --all-features -- -D warnings: cleancargo test -p datafusion-spark: 338 passedcargo test -p datafusion-spark-sqllogictest: 249 of 249 files passedAre there any user-facing changes?
Yes.
ln,log,log10andlog2now follow Spark instead of DataFusion's built-ins, andlog1pis new. String arguments are cast to DOUBLE, as in Spark.