From 7490bac9b0e1b9e2ebb67730e951ba5ed7bdb616 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 18 Nov 2026 10:08:17 -0700 Subject: [PATCH 1/2] docs: explain input types, casts and ANSI mode in the contributor guide Describe SparkInputTypes, the common-type rules, spark_cast and EvalMode, which were added in #89, #304, #345 and #350, and point the audit skill at them. --- .../SKILL.md | 19 +++++++++-- CONTRIBUTING.md | 34 +++++++++++++++++++ 2 files changed, 50 insertions(+), 3 deletions(-) diff --git a/.ai/skills/audit-datafusion-spark-expression/SKILL.md b/.ai/skills/audit-datafusion-spark-expression/SKILL.md index 50475c30..651ac92b 100644 --- a/.ai/skills/audit-datafusion-spark-expression/SKILL.md +++ b/.ai/skills/audit-datafusion-spark-expression/SKILL.md @@ -249,6 +249,15 @@ Note that `Signature::exact` is narrow. A Spark function accepting any string type needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function accepting any integral type needs the full width range. +If Spark's expression is `ImplicitCastInputTypes`, the fix is usually to declare +its `inputTypes` with `SparkInputTypes` (`function/implicit_cast.rs`, with +`sha2` as the example) rather than to widen a hand-written `Signature`. For a +function that needs one type for several arguments, check whether it uses the +rules in `function/type_coercion.rs`. See "Input types, casts and ANSI mode" in +`CONTRIBUTING.md`. Report a function that still hand-rolls either as a finding, +and note that `SparkInputTypes` only chooses the target types. The cast that +then runs is not yet Spark's for every type (issue #88). + **Return nullability.** Compare `return_field_from_args` (or `return_type`) against Spark's `nullable`. A function that reports non-nullable while Spark can return NULL produces wrong plans, not merely wrong values: downstream @@ -358,14 +367,18 @@ grep -rn "TODO" crates/spark/src/function//$ARGUMENTS.rs \ ``` Wiring ANSI up is usually mechanical rather than a semantics decision. The flag -is already threaded through `ScalarFunctionArgs`, so the implementation reads -it directly: +is already threaded through `ScalarFunctionArgs`. Prefer an `EvalMode` +(`Legacy`, `Ansi`, `Try`) over a `bool`, so that the `try_*` function can share +the implementation. `parse_url` and `try_parse_url` are the example: ```rust let ScalarFunctionArgs { args, config_options, .. } = args; -let ansi_mode = config_options.execution.enable_ansi_mode; +let eval_mode = EvalMode::from_config(&config_options); ``` +A function that only needs the flag can still read +`config_options.execution.enable_ansi_mode`, as `abs` and `modulus` do. + `crates/spark/src/function/math/abs.rs` and `crates/spark/src/function/math/modulus.rs` are the in-repo precedents for both the plumbing and the test layout. DataFusion does not model Spark's error diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3a13bf9c..0920c87e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,6 +33,40 @@ have a physical implementation (`invoke_with_args()`) it is not usable by Comet. The functions in this crate need only support input types available to Spark; that is, they do not need to handle unsigned types or types such as `Float16` or `Decimal64`. +### Input types, casts and ANSI mode + +Spark casts a function's arguments before it runs the function, and a failed cast or an invalid +value behaves differently with ANSI mode on, off, and in the `try_*` functions. This crate has a +shared piece for each of these. Use them rather than writing a `Signature`, a `coerce_types()` +or an ANSI flag by hand, so that every function gets the same rules and the same errors. + +- **Declared input types.** When the Spark expression mixes in `ImplicitCastInputTypes`, declare + its `inputTypes` with `SparkInputType` and hold them in a `SparkInputTypes` + (`function/implicit_cast.rs`). The function then takes `signature()`, `coerce_types()` and + `with_updated_config()` from it, which gives Spark's casts, its `UNEXPECTED_INPUT_TYPE` and + `WRONG_NUM_ARGS` errors, and the ANSI or non-ANSI rules from the session config. `sha2` is a + complete example (`crates/spark/src/function/hash/sha2.rs`). This module only chooses the target + types. The cast that then runs is not yet Spark's, see the next point. +- **Common type of several arguments.** When Spark finds one type for several arguments, as + `array_position` does for the array element and the searched value, use the rules in + `function/type_coercion.rs` such as `tightest_common_type()`. They also provide the type names + that Spark's errors use. +- **Casting a value.** When a function needs to cast a value the way Spark's `CAST` does, call + `spark_cast()` (`function/conversion/cast/`) and not Arrow's `cast`. The casts are ported from + Comet one source type at a time ([#88]), and one that is not ported yet returns a "not + implemented" error. It does not fall back to Arrow's cast, because that would return a result + that differs from Spark's. If you need a cast that is missing, port it with its tests, or say in + the PR that the function depends on it. +- **ANSI mode.** A function whose result depends on `spark.sql.ansi.enabled` takes an `EvalMode` + (`Legacy`, `Ansi` or `Try`), and not a `bool`. Read it in `invoke_with_args()` with + `EvalMode::from_config(&args.config_options)`. Spark's `try_*` function then calls the same + implementation with `EvalMode::Try`, as `parse_url` and `try_parse_url` do. + +Test these through SQLLogicTests with ANSI mode both on and off (`set +datafusion.execution.enable_ansi_mode = true;`), because the coercion only happens when the +function is planned from SQL. + +[#88]: https://github.com/apache/datafusion-spark/issues/88 [sail]: https://github.com/lakehq/sail [comet]: https://github.com/apache/datafusion-comet From 2c9bd963f8a9fd3ebe508173daedc71d38b1a080 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 18 Nov 2026 10:46:15 -0700 Subject: [PATCH 2/2] docs: tighten the input types, casts and ANSI mode guidance Point to the module docs instead of restating them, and shorten the audit skill text to a pointer. --- .../SKILL.md | 20 +++---- CONTRIBUTING.md | 52 ++++++++----------- 2 files changed, 27 insertions(+), 45 deletions(-) diff --git a/.ai/skills/audit-datafusion-spark-expression/SKILL.md b/.ai/skills/audit-datafusion-spark-expression/SKILL.md index 651ac92b..0ccf27ca 100644 --- a/.ai/skills/audit-datafusion-spark-expression/SKILL.md +++ b/.ai/skills/audit-datafusion-spark-expression/SKILL.md @@ -249,14 +249,10 @@ Note that `Signature::exact` is narrow. A Spark function accepting any string type needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function accepting any integral type needs the full width range. -If Spark's expression is `ImplicitCastInputTypes`, the fix is usually to declare -its `inputTypes` with `SparkInputTypes` (`function/implicit_cast.rs`, with -`sha2` as the example) rather than to widen a hand-written `Signature`. For a -function that needs one type for several arguments, check whether it uses the -rules in `function/type_coercion.rs`. See "Input types, casts and ANSI mode" in -`CONTRIBUTING.md`. Report a function that still hand-rolls either as a finding, -and note that `SparkInputTypes` only chooses the target types. The cast that -then runs is not yet Spark's for every type (issue #88). +If Spark's expression is `ImplicitCastInputTypes`, or needs one type for several +arguments, suggest the shared helpers in "Input types, casts and ANSI mode" in +`CONTRIBUTING.md`. Few functions use them yet, so report a hand-written +signature as a defect only when it accepts or rejects the wrong types. **Return nullability.** Compare `return_field_from_args` (or `return_type`) against Spark's `nullable`. A function that reports non-nullable while Spark @@ -367,18 +363,14 @@ grep -rn "TODO" crates/spark/src/function//$ARGUMENTS.rs \ ``` Wiring ANSI up is usually mechanical rather than a semantics decision. The flag -is already threaded through `ScalarFunctionArgs`. Prefer an `EvalMode` -(`Legacy`, `Ansi`, `Try`) over a `bool`, so that the `try_*` function can share -the implementation. `parse_url` and `try_parse_url` are the example: +is already threaded through `ScalarFunctionArgs`. Prefer an `EvalMode` over a +`bool` (see `CONTRIBUTING.md`), as `parse_url` and `try_parse_url` do: ```rust let ScalarFunctionArgs { args, config_options, .. } = args; let eval_mode = EvalMode::from_config(&config_options); ``` -A function that only needs the flag can still read -`config_options.execution.enable_ansi_mode`, as `abs` and `modulus` do. - `crates/spark/src/function/math/abs.rs` and `crates/spark/src/function/math/modulus.rs` are the in-repo precedents for both the plumbing and the test layout. DataFusion does not model Spark's error diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0920c87e..606232e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,38 +35,28 @@ is, they do not need to handle unsigned types or types such as `Float16` or `Dec ### Input types, casts and ANSI mode -Spark casts a function's arguments before it runs the function, and a failed cast or an invalid -value behaves differently with ANSI mode on, off, and in the `try_*` functions. This crate has a -shared piece for each of these. Use them rather than writing a `Signature`, a `coerce_types()` -or an ANSI flag by hand, so that every function gets the same rules and the same errors. - -- **Declared input types.** When the Spark expression mixes in `ImplicitCastInputTypes`, declare - its `inputTypes` with `SparkInputType` and hold them in a `SparkInputTypes` - (`function/implicit_cast.rs`). The function then takes `signature()`, `coerce_types()` and - `with_updated_config()` from it, which gives Spark's casts, its `UNEXPECTED_INPUT_TYPE` and - `WRONG_NUM_ARGS` errors, and the ANSI or non-ANSI rules from the session config. `sha2` is a - complete example (`crates/spark/src/function/hash/sha2.rs`). This module only chooses the target - types. The cast that then runs is not yet Spark's, see the next point. +Spark casts a function's arguments before it runs the function, and an invalid value behaves +differently with ANSI mode on, off, and in the `try_*` functions. Prefer the shared helpers below +to per-function rules. They keep results and errors consistent, and their module docs have the +details. + +- **Declared input types.** When the Spark expression is `ImplicitCastInputTypes`, declare its + `inputTypes` with `SparkInputTypes` (`function/implicit_cast.rs`), which supplies `signature()`, + `coerce_types()` and `with_updated_config()` with Spark's errors and ANSI rules. See `sha2`. It + only chooses the target types, so the cast that then runs is not yet Spark's. - **Common type of several arguments.** When Spark finds one type for several arguments, as - `array_position` does for the array element and the searched value, use the rules in - `function/type_coercion.rs` such as `tightest_common_type()`. They also provide the type names - that Spark's errors use. -- **Casting a value.** When a function needs to cast a value the way Spark's `CAST` does, call - `spark_cast()` (`function/conversion/cast/`) and not Arrow's `cast`. The casts are ported from - Comet one source type at a time ([#88]), and one that is not ported yet returns a "not - implemented" error. It does not fall back to Arrow's cast, because that would return a result - that differs from Spark's. If you need a cast that is missing, port it with its tests, or say in - the PR that the function depends on it. -- **ANSI mode.** A function whose result depends on `spark.sql.ansi.enabled` takes an `EvalMode` - (`Legacy`, `Ansi` or `Try`), and not a `bool`. Read it in `invoke_with_args()` with - `EvalMode::from_config(&args.config_options)`. Spark's `try_*` function then calls the same - implementation with `EvalMode::Try`, as `parse_url` and `try_parse_url` do. - -Test these through SQLLogicTests with ANSI mode both on and off (`set -datafusion.execution.enable_ansi_mode = true;`), because the coercion only happens when the -function is planned from SQL. - -[#88]: https://github.com/apache/datafusion-spark/issues/88 + `array_position` does, call the rules in `function/type_coercion.rs` (for example + `tightest_common_type()`) from `coerce_types()`. +- **Casting a value.** To cast a value the way Spark's `CAST` does, call `spark_cast()` + (`function/conversion/cast/`) and not Arrow's `cast`. The casts are being ported from Comet + ([#88](https://github.com/apache/datafusion-spark/issues/88)). One that is not ported yet returns a + "not implemented" error, because Arrow's result would differ from Spark's. Port the missing cast + with its tests, or note in the PR that the function depends on it. +- **ANSI mode.** A function that depends on `spark.sql.ansi.enabled` should take an `EvalMode` + (`Legacy`, `Ansi` or `Try`) rather than a `bool`, read with + `EvalMode::from_config(&args.config_options)`. A `try_*` function then shares the implementation + with `EvalMode::Try`, as `parse_url` and `try_parse_url` do. + [sail]: https://github.com/lakehq/sail [comet]: https://github.com/apache/datafusion-comet