Skip to content

Pass fail_on_error to datafusion-spark functions through ConfigOptions #7669

Open

Description

@mbutrovich

What is the problem the feature request solves?

create_scalar_function_expr builds every ScalarFunctionExpr with ConfigOptions::default(). The datafusion-spark functions read ANSI mode from config_options.execution.enable_ansi_mode at execution time, for example abs and next_day. Under Comet they always see false, so they return NULL or a wrapped value where Spark with ANSI mode on raises an error.

Comet's own functions don't have this problem, because the planner passes expr.fail_on_error to their constructors. The "Port ... from Comet" issues in datafusion-spark plan for Comet to use the datafusion-spark versions and remove its copies (for example apache/datafusion-spark#125). apache/datafusion-spark#329 ports make_date and reads the session config in place of Comet's fail_on_error constructor argument. Each such switch turns Comet's ANSI errors for that function into NULLs.

Describe the potential solution

In create_scalar_function_expr, pass a ConfigOptions with execution.enable_ansi_mode set to expr.fail_on_error instead of the default. Two shared Arc<ConfigOptions> values, one per mode, avoid an allocation per expression.

The same ConfigOptions can carry the datafusion-spark config extension that apache/datafusion-spark#92 proposes. That issue notes that Comet sees only the defaults, and concludes that every flag also needs a constructor argument or a public function. If Comet populates ConfigOptions itself, the flags that are read at execution time wouldn't need that second way in.

Additional context

Spark's try_* functions are a third mode (EvalMode::Try). A single bool can't express it, so this change covers ANSI and legacy only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions