Skip to content
12 changes: 12 additions & 0 deletions LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -200,3 +200,15 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

--------------------------------------------------------------------------------

This product includes code translated from fdlibm 5.3 (Freely Distributable
LIBM), in crates/spark/src/function/math/fdlibm.rs, under the following terms:

Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved.

Developed at SunSoft, a Sun Microsystems, Inc. business.
Permission to use, copy, modify, and distribute this
software is freely granted, provided that this notice
is preserved.
Comment on lines +203 to +214

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewers, please confirm this is acceptable under ASF policy.

The fdlibm notice is the SPDX SunPro license, which isn't named in the ASF's Category A list. There is precedent: Lucene ships fdlibm-derived code with the Apache header on top and the Sun notice in a comment (SloppyMath.java). The ASF header policy also says not to put the Apache header on third-party source files, and it leaves major modifications to the PMC. A line-by-line translation falls between those two cases.

Could you ask on legal-discuss@ whether SunPro is Category A and which header fdlibm.rs should carry, and link the answer here? I'd like that settled before this merges.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this. I'll ask on legal-discuss@ and link the thread here once there's an answer.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, while we wait on legal-discuss@: arrow already depends on code under this same notice. arrow-array enables half's num-traits feature, which enables num-traits' libm feature. Fifty of libm's source files carry the Sun notice, and its LICENSE reproduces it. That is a dependency rather than code in our source release, so it does not settle the question here. I also checked whether we could use libm instead. It differs from StrictMath on about 1% of log inputs and 11% of log10 inputs, but it matches expm1 exactly.

11 changes: 9 additions & 2 deletions crates/spark/src/function/math/expm1.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

use crate::function::error_utils::unsupported_data_type_exec_err;
use crate::function::math::fdlibm;
use arrow::array::{ArrayRef, AsArray};
use arrow::datatypes::{DataType, Float64Type};
use datafusion_common::utils::take_function_args;
Expand All @@ -17,6 +18,12 @@ use std::sync::Arc;
/// Spark-compatible `expm1` expression.
///
/// <https://spark.apache.org/docs/latest/api/sql/index.html#expm1>
///
/// Spark computes this with `java.lang.StrictMath.expm1`, so this uses the fdlibm
/// translation in the private `math::fdlibm` module rather than `f64::exp_m1`,
/// which calls the platform's libm and differs in the last digit on some platforms.
/// The module is named in plain text because rustdoc rejects a public item's link
/// to a private one.
#[user_doc(
doc_section(label = "Math Functions"),
description = "Returns `exp(expr) - 1` as a DOUBLE, computed without the precision loss of \
Expand Down Expand Up @@ -74,13 +81,13 @@ impl ScalarUDFImpl for SparkExpm1 {
let [arg] = take_function_args(self.name(), args.args)?;
match arg {
ColumnarValue::Scalar(ScalarValue::Float64(value)) => Ok(
ColumnarValue::Scalar(ScalarValue::Float64(value.map(|x| x.exp_m1()))),
ColumnarValue::Scalar(ScalarValue::Float64(value.map(fdlibm::expm1))),
),
ColumnarValue::Array(array) => match array.data_type() {
DataType::Float64 => Ok(ColumnarValue::Array(Arc::new(
array
.as_primitive::<Float64Type>()
.unary::<_, Float64Type>(|x| x.exp_m1()),
.unary::<_, Float64Type>(fdlibm::expm1),
)
as ArrayRef)),
other => Err(unsupported_data_type_exec_err(
Expand Down
Loading
Loading