Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .ai/skills/audit-datafusion-spark-expression/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,11 @@ Note that `Signature::exact` is narrow. A Spark function accepting any string
type needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function
accepting any integral type needs the full width range.

If Spark's expression is `ImplicitCastInputTypes`, or needs one type for several
arguments, suggest the shared helpers in "Input types, casts and ANSI mode" in
`CONTRIBUTING.md`. Few functions use them yet, so report a hand-written
signature as a defect only when it accepts or rejects the wrong types.

**Return nullability.** Compare `return_field_from_args` (or `return_type`)
against Spark's `nullable`. A function that reports non-nullable while Spark
can return NULL produces wrong plans, not merely wrong values: downstream
Expand Down Expand Up @@ -358,12 +363,12 @@ grep -rn "TODO" crates/spark/src/function/<category>/$ARGUMENTS.rs \
```

Wiring ANSI up is usually mechanical rather than a semantics decision. The flag
is already threaded through `ScalarFunctionArgs`, so the implementation reads
it directly:
is already threaded through `ScalarFunctionArgs`. Prefer an `EvalMode` over a
`bool` (see `CONTRIBUTING.md`), as `parse_url` and `try_parse_url` do:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might have to make this a little bit stricter to make eval mode compulsory (at least in the top level function)


```rust
let ScalarFunctionArgs { args, config_options, .. } = args;
let ansi_mode = config_options.execution.enable_ansi_mode;
let eval_mode = EvalMode::from_config(&config_options);
```

`crates/spark/src/function/math/abs.rs` and
Expand Down
24 changes: 24 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,30 @@ have a physical implementation (`invoke_with_args()`) it is not usable by Comet.
The functions in this crate need only support input types available to Spark; that
is, they do not need to handle unsigned types or types such as `Float16` or `Decimal64`.

### Input types, casts and ANSI mode

Spark casts a function's arguments before it runs the function, and an invalid value behaves
differently with ANSI mode on, off, and in the `try_*` functions. Prefer the shared helpers below
to per-function rules. They keep results and errors consistent, and their module docs have the
details.

- **Declared input types.** When the Spark expression is `ImplicitCastInputTypes`, declare its
`inputTypes` with `SparkInputTypes` (`function/implicit_cast.rs`), which supplies `signature()`,
`coerce_types()` and `with_updated_config()` with Spark's errors and ANSI rules. See `sha2`. It
only chooses the target types, so the cast that then runs is not yet Spark's.
- **Common type of several arguments.** When Spark finds one type for several arguments, as
`array_position` does, call the rules in `function/type_coercion.rs` (for example
`tightest_common_type()`) from `coerce_types()`.
- **Casting a value.** To cast a value the way Spark's `CAST` does, call `spark_cast()`
(`function/conversion/cast/`) and not Arrow's `cast`. The casts are being ported from Comet
([#88](https://github.com/apache/datafusion-spark/issues/88)). One that is not ported yet returns a
"not implemented" error, because Arrow's result would differ from Spark's. Port the missing cast
with its tests, or note in the PR that the function depends on it.
- **ANSI mode.** A function that depends on `spark.sql.ansi.enabled` should take an `EvalMode`
(`Legacy`, `Ansi` or `Try`) rather than a `bool`, read with
`EvalMode::from_config(&args.config_options)`. A `try_*` function then shares the implementation
with `EvalMode::Try`, as `parse_url` and `try_parse_url` do.

[sail]: https://github.com/lakehq/sail
[comet]: https://github.com/apache/datafusion-comet

Expand Down
Loading