diff --git a/.ai/skills/audit-datafusion-spark-expression/SKILL.md b/.ai/skills/audit-datafusion-spark-expression/SKILL.md index 50475c30..0ccf27ca 100644 --- a/.ai/skills/audit-datafusion-spark-expression/SKILL.md +++ b/.ai/skills/audit-datafusion-spark-expression/SKILL.md @@ -249,6 +249,11 @@ Note that `Signature::exact` is narrow. A Spark function accepting any string type needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function accepting any integral type needs the full width range. +If Spark's expression is `ImplicitCastInputTypes`, or needs one type for several +arguments, suggest the shared helpers in "Input types, casts and ANSI mode" in +`CONTRIBUTING.md`. Few functions use them yet, so report a hand-written +signature as a defect only when it accepts or rejects the wrong types. + **Return nullability.** Compare `return_field_from_args` (or `return_type`) against Spark's `nullable`. A function that reports non-nullable while Spark can return NULL produces wrong plans, not merely wrong values: downstream @@ -358,12 +363,12 @@ grep -rn "TODO" crates/spark/src/function//$ARGUMENTS.rs \ ``` Wiring ANSI up is usually mechanical rather than a semantics decision. The flag -is already threaded through `ScalarFunctionArgs`, so the implementation reads -it directly: +is already threaded through `ScalarFunctionArgs`. Prefer an `EvalMode` over a +`bool` (see `CONTRIBUTING.md`), as `parse_url` and `try_parse_url` do: ```rust let ScalarFunctionArgs { args, config_options, .. } = args; -let ansi_mode = config_options.execution.enable_ansi_mode; +let eval_mode = EvalMode::from_config(&config_options); ``` `crates/spark/src/function/math/abs.rs` and diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3a13bf9c..606232e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,6 +33,30 @@ have a physical implementation (`invoke_with_args()`) it is not usable by Comet. The functions in this crate need only support input types available to Spark; that is, they do not need to handle unsigned types or types such as `Float16` or `Decimal64`. +### Input types, casts and ANSI mode + +Spark casts a function's arguments before it runs the function, and an invalid value behaves +differently with ANSI mode on, off, and in the `try_*` functions. Prefer the shared helpers below +to per-function rules. They keep results and errors consistent, and their module docs have the +details. + +- **Declared input types.** When the Spark expression is `ImplicitCastInputTypes`, declare its + `inputTypes` with `SparkInputTypes` (`function/implicit_cast.rs`), which supplies `signature()`, + `coerce_types()` and `with_updated_config()` with Spark's errors and ANSI rules. See `sha2`. It + only chooses the target types, so the cast that then runs is not yet Spark's. +- **Common type of several arguments.** When Spark finds one type for several arguments, as + `array_position` does, call the rules in `function/type_coercion.rs` (for example + `tightest_common_type()`) from `coerce_types()`. +- **Casting a value.** To cast a value the way Spark's `CAST` does, call `spark_cast()` + (`function/conversion/cast/`) and not Arrow's `cast`. The casts are being ported from Comet + ([#88](https://github.com/apache/datafusion-spark/issues/88)). One that is not ported yet returns a + "not implemented" error, because Arrow's result would differ from Spark's. Port the missing cast + with its tests, or note in the PR that the function depends on it. +- **ANSI mode.** A function that depends on `spark.sql.ansi.enabled` should take an `EvalMode` + (`Legacy`, `Ansi` or `Try`) rather than a `bool`, read with + `EvalMode::from_config(&args.config_options)`. A `try_*` function then shares the implementation + with `EvalMode::Try`, as `parse_url` and `try_parse_url` do. + [sail]: https://github.com/lakehq/sail [comet]: https://github.com/apache/datafusion-comet