You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Decide how functions handle dictionary and run-end-encoded arguments #349
Is your feature request related to a problem or challenge?
Spark has no dictionary or run-end-encoded types, but DataFusion and Comet pass them to functions: a Parquet string column is often read as a dictionary. Functions here handle them unevenly. The shared common-type rules in function/type_coercion.rs (#345) now give a dictionary argument the type of its values, so array_position, array_remove, array_insert, greatest and least accept one and cast it to the value type. Other functions handle them on their own: dayofweek maps the dictionary values and hash has a dictionary path, while others reject them. Run-end-encoded arguments are not handled in the shared rules at all.
Describe the solution you'd like
Decide how the crate should treat these encodings, and apply it consistently. Questions to settle:
Should coercion keep the encoding where a function can work on it directly (for example, compare or hash the dictionary values once and map the keys), instead of casting to the value type?
Should run-end-encoded arguments be accepted the same way as dictionaries?
Activity
refactor(spark): share Spark's common-type rules between functions (#345
feat(spark): cast function arguments to Spark's declared input types (#…