Skip to content

Decide how functions handle dictionary and run-end-encoded arguments #349

Open

Description

@andygrove

Is your feature request related to a problem or challenge?

Spark has no dictionary or run-end-encoded types, but DataFusion and Comet pass them to functions: a Parquet string column is often read as a dictionary. Functions here handle them unevenly. The shared common-type rules in function/type_coercion.rs (#345) now give a dictionary argument the type of its values, so array_position, array_remove, array_insert, greatest and least accept one and cast it to the value type. Other functions handle them on their own: dayofweek maps the dictionary values and hash has a dictionary path, while others reject them. Run-end-encoded arguments are not handled in the shared rules at all.

Describe the solution you'd like

Decide how the crate should treat these encodings, and apply it consistently. Questions to settle:

  • Should coercion keep the encoding where a function can work on it directly (for example, compare or hash the dictionary values once and map the keys), instead of casting to the value type?
  • Should run-end-encoded arguments be accepted the same way as dictionaries?
  • Should this be part of the shared input-type coercion proposed in Declare Spark input types per function and coerce them in one place #346, so each function does not handle encodings itself?
  • What do Comet's call paths need, since Comet calls invoke_with_args directly without coercion?

Describe alternatives you've considered

Keep handling encodings per function, as now.

Additional context

Raised while moving array_insert onto the shared rules in #70, which had unwrapped dictionaries itself.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions