Repository navigation
fix: Read unaligned buffers safely in ArrowArrayView getters - #1011
Open
JonSnow1807 wants to merge 1 commit into
Open
JonSnow1807 wants to merge 1 commit into
JonSnow1807 wants to merge 1 commit into
Conversation
The ArrowArrayView getters read values, offsets and binary views through typed pointers (e.g., data.as_int32[i]). Buffers that arrive through the C Data interface do not have to be aligned, and for those each of these reads is a misaligned load that UBSan reports. This adds memcpy-based ArrowBufferViewGet*Unsafe() helpers for the remaining fixed-width types next to the 32- and 64-bit ones from apache#946 and uses them in ArrowArrayViewGetIntUnsafe(), ArrowArrayViewGetUIntUnsafe(), ArrowArrayViewGetDoubleUnsafe(), ArrowArrayViewGetStringUnsafe(), ArrowArrayViewGetBytesUnsafe(), ArrowArrayViewListChildOffset() and ArrowArrayViewUnionChildOffset(). Binary views are copied before their fields are read. The helpers index the buffer as an array of byte arrays instead of adding i * sizeof(T) to a byte pointer. With the multiplication, clang stopped inlining ArrowArrayViewGetIntUnsafe() and ArrowArrayViewGetDoubleUnsafe() in a file with more than one call site. Part of apache#950.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1011 +/- ##
==========================================
+ Coverage 78.48% 78.72% +0.24%
==========================================
Files 106 106
Lines 17245 17273 +28
Branches 2065 2065
==========================================
+ Hits 13535 13599 +64
+ Misses 2446 2410 -36
Partials 1264 1264 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #950. The ArrowArrayView getters in
inline_array.hread values, offsets and binary views through typed pointers, which is a misaligned load when a buffer from the C Data interface isn’t aligned. This addsArrowBufferViewGet*Unsafe()helpers for the remaining fixed-width types next to the two from #946, uses them in the integer, double, string, bytes, list child offset and union child offset getters, and copies binary views before reading their fields.New tests read from buffers misaligned by 1 to 7 bytes in sliced arrays. On main UBSan reports 56 misaligned loads on 43 lines; with this change it reports none, and ctest passes under ASan and UBSan with clang and gcc.
The helpers index the buffer as an array of byte arrays because with
i * sizeof(T)clang stopped inliningArrowArrayViewGetIntUnsafe()andArrowArrayViewGetDoubleUnsafe()in files with more than one call site. With this form clang generates the same code as on main, and the array benchmarks are unchanged. The remaining sites (array.c, the IPC decoder, the C++ helpers) can follow separately.