Repository navigation
feat(spark): implement Spark-compatible log1p - #344
kazantsev-maksim wants to merge 2 commits into
Conversation
# Conflicts: # crates/spark/src/function/math/mod.rs
andygrove
left a comment
There was a problem hiding this comment.
Thanks for picking up log1p. The <= -1 → NULL handling matches Spark's UnaryLogExpression, and I checked the values in this file against PySpark 4.2.0 with ANSI both on and off. They all agree.
When I ran cargo test -p datafusion-spark-sqllogictest -- math/log1p locally it failed, though. Could you rebase on main and run it? There are three issues:
- The slt runner prints doubles with 15 significant digits, so
0.4054651081081644needs to be0.405465108108164. The same goes for the0.693…values. - The two equality queries return a boolean, so they need
query Brather thanquery T. SELECT log1p('0.5')fails at planning time. See the comment on the signature.
The PySpark runner in crates/sqllogictest/scripts is a quick way to check the file against Spark once it passes.
| impl SparkLog1p { | ||
| pub fn new() -> Self { | ||
| Self { | ||
| signature: Signature::exact(vec![DataType::Float64], Volatility::Immutable), |
There was a problem hiding this comment.
Spark's Log1p declares inputTypes = Seq(DoubleType) with ImplicitCastInputTypes. That means it accepts strings (and errors on a malformed one under ANSI), and it rejects BOOLEAN, DATE and BINARY with UNEXPECTED_INPUT_TYPE. Signature::exact(Float64) rejects strings, so log1p('0.5') in this PR's own test fails.
Since #350 we have SparkInputTypes for exactly this case. Could you switch to SparkInputTypes::new(vec![SparkInputType::Double]) and add coerce_types and with_updated_config the way hash/sha2.rs on main does? Then the argument is always Float64, and the two unsupported_data_type_exec_err arms can go.
| query R | ||
| SELECT log1p('0.5'); | ||
| ---- | ||
| 0.4054651081081644 |
There was a problem hiding this comment.
Could you add a few cases for the inputs Spark rejects, like log1p(true), log1p(DATE '2020-01-01') and log1p(X'01'), as query error? A malformed string like log1p('abc') would also be useful. It returns NULL with ANSI off and raises CAST_INVALID_INPUT with ANSI on, so it needs the ANSI wrapper described in test_files/README.md.
Which issue does this PR close?
log1p#266.Rationale for this change
Spark SQL provides
log1p, which computes the natural logarithm of1 + xaccurately for values near zero. Adding this function to DataFusion Spark lets users run queries that uselog1pwith Spark-compatible behavior.What changes are included in this PR?
log1pscalar function.NULLfor inputs less than or equal to-1.0, including negative infinity.NULLinputs and handleNaN, positive infinity, and values near zero.Are these changes tested?
SQL logic tests have been added in
math/log1p.slt. They have not yet been run, and the expected results have not been independently verified against Spark.Are there any user-facing changes?
Yes. This PR adds the new
log1p(expr)SQL function. It returns aDOUBLEand returnsNULLfor inputs less than or equal to-1.0. No existing functions are changed.