Description
Describe the enhancement requested
Many compute functions accept decimal128 and decimal256 but raise ArrowNotImplementedError for decimal32 and decimal64. I checked each function on main (27.0.0.dev) with decimal32(5, 1), decimal64(5, 1) and decimal128(5, 1) arrays from pyarrow. The following work for decimal128 and fail for decimal32 and decimal64:
- Aggregations:
min_max,min,max,index,mode,sum,mean,product - Grouped aggregations:
hash_min_max,hash_sum,hash_list - Hashing:
unique,value_counts,dictionary_encode - Set lookup:
is_in,index_in - Sorting:
sort_indices,rank,select_k_unstable - Arithmetic and rounding:
abs,negate,sign,round,ceil,floor - Selection:
choose
Already working for all widths: add, equal, if_else, coalesce, case_when, take, drop_null, count_distinct, hash_distinct. The replace kernels (replace_with_mask, fill_null_forward, fill_null_backward) are covered by #55256 / #55257.
For many of these, the kernel code is already templated over the decimal type and only the registration lists decimal128 and decimal256. For example, hash_min_max registers decimal128(1, 1), decimal256(1, 1) in hash_aggregate.cc, and the scalar min_max registration in aggregate_basic.cc does the same. Those should mostly need the extra registrations plus tests. Others, like the sort comparators and rounding, may need more work.
sum, mean and product need a decision first. Today sum of decimal128(5, 1) returns decimal128(38, 1), widening to the maximum precision of the same type (#35166), while mean and product keep the input type. For decimal32 the maximum precision is only 9, so following the same rule would overflow much sooner. Should sum of decimal32 / decimal64 stay within the input type, as decimal32(9, s) / decimal64(18, s), or promote to decimal128(38, s)?
I'd like to work on this and split it into small PRs, one per kernel family, starting with min_max / hash_min_max. I'll leave sum / mean / product until the output type is agreed here.
Component(s)
C++
Activity