You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Pass fail_on_error to datafusion-spark functions through ConfigOptions #7669
create_scalar_function_expr builds every ScalarFunctionExpr with ConfigOptions::default(). The datafusion-spark functions read ANSI mode from config_options.execution.enable_ansi_mode at execution time, for example abs and next_day. Under Comet they always see false, so they return NULL or a wrapped value where Spark with ANSI mode on raises an error.
Comet's own functions don't have this problem, because the planner passes expr.fail_on_error to their constructors. The "Port ... from Comet" issues in datafusion-spark plan for Comet to use the datafusion-spark versions and remove its copies (for example apache/datafusion-spark#125). apache/datafusion-spark#329 ports make_date and reads the session config in place of Comet's fail_on_error constructor argument. Each such switch turns Comet's ANSI errors for that function into NULLs.
Describe the potential solution
In create_scalar_function_expr, pass a ConfigOptions with execution.enable_ansi_mode set to expr.fail_on_error instead of the default. Two shared Arc<ConfigOptions> values, one per mode, avoid an allocation per expression.
The same ConfigOptions can carry the datafusion-spark config extension that apache/datafusion-spark#92 proposes. That issue notes that Comet sees only the defaults, and concludes that every flag also needs a constructor argument or a public function. If Comet populates ConfigOptions itself, the flags that are read at execution time wouldn't need that second way in.
Additional context
Spark's try_* functions are a third mode (EvalMode::Try). A single bool can't express it, so this change covers ANSI and legacy only.
Activity