Describe the bug, including details regarding any error messages, version, or platform.
is_null(arr, nan_is_null=True) does not mark NaN values as null when arr is dictionary-encoded with floating-point dictionary values.
IsNullExec decides whether to take the NaN path by looking at the top-level type id. For a dictionary-encoded array that id is DICTIONARY, so the floating-point check is skipped entirely and NaN entries in the dictionary are never marked null. The same call on the decoded (non-dictionary) array returns the expected result.
Related: null values inside the dictionary (as opposed to null indices) are also not reflected, so a valid index pointing at a null dictionary slot is reported as non-null.
Discussion and a first attempt at a fix are in #51000; @pitrou suggested computing is_null on the dictionary values, take-ing by the indices, and OR-ing the result with the index validity bitmap, which covers both the NaN case and null dictionary values in one pass.
Component(s)
C++
Describe the bug, including details regarding any error messages, version, or platform.
is_null(arr, nan_is_null=True)does not mark NaN values as null whenarris dictionary-encoded with floating-point dictionary values.IsNullExecdecides whether to take the NaN path by looking at the top-level type id. For a dictionary-encoded array that id isDICTIONARY, so the floating-point check is skipped entirely and NaN entries in the dictionary are never marked null. The same call on the decoded (non-dictionary) array returns the expected result.Related: null values inside the dictionary (as opposed to null indices) are also not reflected, so a valid index pointing at a null dictionary slot is reported as non-null.
Discussion and a first attempt at a fix are in #51000; @pitrou suggested computing
is_nullon the dictionary values,take-ing by the indices, and OR-ing the result with the index validity bitmap, which covers both the NaN case and null dictionary values in one pass.Component(s)
C++