Skip to content

nanvl needs simplify for a lazy expr2 and for string arguments #376

Open

Description

@andygrove

nanvl gets two of its Spark behaviors only from its simplify rewrite to CASE WHEN isnan(expr1) THEN expr2 ELSE expr1 END. When simplify doesn't run, invoke_with_args handles the call. That happens with the sqllogictest evaluate mode (#316), and with a caller such as Comet that builds the physical expression directly. In that path:

  • expr2 is evaluated for every row. With ANSI mode on, SELECT nanvl(c2, 1 / 0) FROM (VALUES (0), (1)) t(c2) raises Divide by zero. Spark returns 0, 1, because it only evaluates 1 / 0 on rows where c2 is NaN.
  • String arguments are rejected. SELECT nanvl('123.5', 1.0::double) fails with nanvl expects Float32 or Float64 arguments at this point, got Utf8, where Spark returns 123.5. The ANSI-aware cast (NULL with ANSI off, an error with it on) lives only in simplify.

The string case can be fixed in invoke_with_args by casting a string argument to Float64, with safe set from enable_ansi_mode. A scalar UDF always receives its arguments already evaluated, so a lazy expr2 can't come from invoke_with_args alone.

The seven records in conditional/nanvl.slt that need simplify are marked skipif evaluate until this is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions