You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] collect_list(x ORDER BY ...) fails with an internal error #112
collect_list with an ORDER BY inside the call fails with an internal error:
SELECT collect_list(a ORDER BY a DESC) FROM (VALUES (3), (1), (2)) t(a);
-- DataFusion error: Internal error: Assertion failed: values.len() == 1 (left: 2, right: 1): expects single batch.
Spark SQL has no ORDER BY inside collect_list, so there is no Spark result to match, but DataFusion's SQL accepts the syntax and an internal error is never the right answer.
Expected behavior
Either support the ordering (as DataFusion's array_agg(a ORDER BY ...) does), or reject it at planning time with a clear error. collect_set(a ORDER BY a DESC) does not fail: it ignores the ordering and returns the values in arbitrary order ([2, 1, 3] for the query above). Since a set has no order that is harmless, but rejecting it would be clearer.
Activity
docs(spark): note avg and collect_list gaps with issue links
docs(spark): document array, collection and aggregate functions with …