Repository navigation
Conversation
|
What happens if you append different NaNs values? For example quiet_NaN vs. signaling_NaN. |
|
@pitrou, the current run-end logic considers both quiet and signaling NaNs equal. Should we add an option to distinguish between them? |
|
I checked arrow-rs. Its behavior seems inconsistent with ours. It uses
If we want to allow the merging of
While checking Arrow's code, I discovered another issue. When constructing dictionary encoded arrays for Parquet in #50807, I compare the physical values of NaNs, but Arrow's dictionary builder does not do this (see |
|
@HuaHuaY @pitrou there seems to be a paradox in the run-end encode compute kernel. The comment below mentions that NaN values are not merged for nested types: arrow/cpp/src/arrow/compute/kernels/vector_run_end_encode.cc Lines 307 to 308 in 5c2ff72 For nested types, the following code shows that consecutive NaN values are not merged: TEST(MyTest, Struct) {
auto struct_type = struct_({field("a", float32())});
auto array = ArrayFromJSON(struct_type, R"([{"a":NaN},{"a":NaN}])");
Datum datum(array);
auto result =
RunEndEncode(datum).ValueOrDie().array_as<RunEndEncodedArray>();
ARROW_LOGGER_INFO("", result->ToString());
}The result is: However, for non-nested types, consecutive NaN values are merged: TEST(MyTest, Float) {
std::shared_ptr<Array> array;
ArrayFromVector<FloatType>({NAN, NAN}, &array);
Datum datum(array);
auto result =
RunEndEncode(datum).ValueOrDie().array_as<RunEndEncodedArray>();
ARROW_LOGGER_INFO("", result->ToString());
}The result is: So it seems that the behavior differs between nested and non-nested types, despite the comment suggesting that NaN values should not be merged for nested types. Is this the expected behavior? |
The code at I am not familiar with Arrow Compute. But I don't think there are any correctness issues with either behavior; both allow the original data to be recovered, with the difference being encoding efficiency. If we want to merge NaNs with identical physical representations, perhaps we could open a separate PR to do that for nested types. |
|
@pitrou, I could not find anything in the spec that says whether NaNs should be merged. I suggest that we should add options for both the compute kernel and the RunEndEncoded builder to control whether NaNs, signed zeros, |
That sounds overkill to me, especially as NaNs are rare in practice. The only constraint here is that run-end-encoding must be entirely lossless: decoding must reproduce the original array. So we should only merge identical values, not values that are close to each other. |
|
As the result of this discussion is that float values should be merged only when they are identical in their physical representation, I suggest adding another flag to What is your opinion? |
I'm not sure. I think we can instead special-case all fixed-width types for the purpose of comparing REE values. This can also make the REE builder faster than allocating a new Scalar for every logical value. Something like this in ree_util.h (untested, just a sketch): struct PhysicalValue {
friend bool operator==(const PhysicalValue& left, const PhysicalValue& right) {
return ...;
}
static PhysicalValueForComparison LookupArray(const Array& array, int64_t index) {
if (array.IsNull(index)) {
return {std::shared_ptr<Scalar>{}};
}
auto typed_lookup = [&](auto* concrete_type) {
using ArrowType = std::decay_t<decltype(*concrete_type)>;
if constexpr(is_fixed_width(ArrowType::id) && !is_boolean(ArrowType::id)) {
const int64_t byte_width = concrete_type->byte_width();
const ArrayData& data = *array.data();
return {data.buffers[1]->span_as<uint8_t>.subspan(
(index + data.offset) * byte_width, byte_width)};
} else {
return array.GetScalar(index);
}
};
return VisitType(*array.type(), typed_lookup);
}
// Three possible states:
// - A span of bytes for a non-null non-boolean primitive value
// - A nullptr Scalar for a null value
// - A non-null generic Scalar otherwise
std::variant<std::shared_ptr<Scalar>, std::span<const uint8_t>> payload_;
}; |
|
@pitrou, I implemented my approach hereand your approach here. Please consider both patches as prototypes; they pass all existing tests. As far as I can tell, your suggested approach does not actually avoid any scalar allocations. In fact, I couldn't find any scalar being created in the first place. It also seems that What do you think would be the better direction: adding a new option to |
|
@andishgar Thanks for doing this! I think the @zanmato1984 Do you have any opinions? |
4cbaac3 to
94aa469
Compare
|
@pitrou I've applied the changes, and the PR is ready for review. |
zanmato1984
left a comment
There was a problem hiding this comment.
The EqualOptions approach makes sense to me. It reuses the existing comparison machinery and aligns the builder and nested REE path with the lossless treatment of floating-point representations already used by the primitive encoding path.
I have two remaining concerns about support for the new public option and regression coverage, detailed in the inline comments.
| /// Whether to compare floating-point values by their physical representation. | ||
| bool physical_repr_equal() const { return physical_repr_equal_; } | ||
|
|
||
| /// Return a copy of this EqualOptions object with the "physical_repr_equal" | ||
| /// property set to the specified value. | ||
| /// If both "physical_repr_equal" and "nans_equal" are enabled, two values | ||
| /// are considered equal if either condition is satisfied. | ||
| EqualOptions physical_repr_equal(bool v) const { |
There was a problem hiding this comment.
Could we also propagate this new option to sparse floating-point equality? FloatSparseTensorDataEquals currently only checks nans_equal(), so it ignores physical_repr_equal().
For independently allocated float32 and float64 tensors containing identical NaN bits, EqualOptions::Defaults().signed_zeros_equal(false).physical_repr_equal(true) gives true for dense tensor equality but false after conversion to COO sparse tensors. Enabling nans_equal(true) is not equivalent, since that also accepts different NaN payloads.
Since both equality APIs accept EqualOptions and this new flag has no documented sparse exclusion, could the sparse implementation reuse the common floating-point equality visitor, with regression coverage for identical and different NaN payloads?
| ASSERT_OK(builder.AppendScalar( | ||
| **MakeScalar(float32(), std::numeric_limits<float>::quiet_NaN()), 25)); | ||
| ASSERT_OK(builder.AppendScalar( | ||
| **MakeScalar(float32(), std::numeric_limits<float>::signaling_NaN()), 5)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 3), 15)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), std::nanf("")), 20)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 4), 25)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), +0.0), 10)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 5.0), 15)); | ||
| ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), -0.0), 25)); |
There was a problem hiding this comment.
Could this test append the same NaN in two separate AppendScalar calls? The consecutive NaNs here are quiet versus signaling, which should remain separate. The other NaN is isolated, and +0 and -0 are separated by 5.
As a narrow negative control, reverting only the builder's Equals(scalar, options_) call to Equals(scalar), while keeping the rest of the current code and tests unchanged, still leaves this test passing. An independent repeated-NaN probe fails under that same reversion.
Two calls to AppendScalar(nan, 3) with identical NaN bits should produce run ends [6], not [3, 6]. Could we cover that case, two quiet NaNs with different payloads, and adjacent signed zeros, asserting both the run boundaries and exact decoded bits? This would protect the intended merge without allowing payload or sign loss.
Rationale for this change
RunEndEncodedBuilderdoes not merge consecutive NaN values into a single run.What changes are included in this PR?
Enable merging consecutive NaN values into a single run.
Are these changes tested?
Yes, I added a unit test covering consecutive NaN values.
Are there any user-facing changes?
No.