Repository navigation
Conversation
…ounting
### Rationale for this change
Counting how many of a run of int16_t levels match a predicate (e.g. how
many definition levels equal the max definition level, or how many
repetition levels are 0) is a hot loop in the Parquet reader and writer.
It is currently open-coded as a scalar loop or a std::count() call.
`std::count` accumulates into a 64-bit integer, which forces the compiler
to widen every 16-bit compare result before accumulating and prevents
efficient vectorization. Accumulating into a `uint16_t` instead lets the
compiler keep the whole reduction in 16-bit lanes (e.g. `cmeq.8h` +
`sub.8h` + `addv.8h` on Neon, 32 levels per iteration), and the chunked
loop keeps that accumulator from overflowing: at most `0xFFFF` levels are
counted before the accumulator is folded into the 64-bit total.
The chunk bookkeeping (clamping the chunk length, folding the partial
count, advancing the pointer) is a fixed cost per call, and for inputs
shorter than one vector iteration it dominates: counting 7 levels that
way is measurably slower than `std::count`, even though both end up
running the same scalar loop. Such short inputs are therefore counted
directly, without going through the chunk loop.
On an Apple M-series machine, counting 2048 int16 levels where every
tenth is missing drops from ~232 ns to ~32 ns (about 7x), while short
inputs are at least as fast as `std::count`.
### What changes are included in this PR?
- Add `parquet::internal::CountIf16` next to the other level scanning
primitives in `parquet/level_comparison.h`. It is a header-only inline
template, since the level conversion translation unit is compiled twice
(standard and BMI2).
- Add unit tests covering empty input, a mixed predicate, short inputs
around the vectorization threshold, and the `0xFFFF` chunk boundaries.
- Add `BM_CountPresentLevels{StdCount,CountIf16}` to the level conversion
benchmark to track the improvement.
### Are these changes tested?
New `CountIf16.*` tests in `level_conversion_test.cc`, run as part of
`parquet-reader-test`.
### Are there any user-facing changes?
No.
|
|
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the shared Parquet level-counting helper and its integrations.
No open issues 🎉
What changed in this PR
This PR addresses GH-55399 by introducing a vectorization-friendly helper for counting Parquet definition and repetition levels, then applying it across reading, writing, and size statistics.
Changes:
- Add
CountIf16with 16-bit accumulators, overflow-safe chunking, and a short-input path. - Replace existing counting loops and
std::countcalls with the shared helper. - Add tests for predicates, short inputs, and chunk boundaries.
| File | Description |
|---|---|
cpp/src/parquet/size_statistics.cc |
Use the helper for binary-level histogram counts. |
cpp/src/parquet/level_conversion_test.cc |
Test counting correctness and chunk boundaries. |
cpp/src/parquet/level_comparison.h |
Implement CountIf16 and its chunked fallback. |
cpp/src/parquet/file_reader.cc |
Use the helper to count rows during file scans. |
cpp/src/parquet/column_writer.cc |
Replace value and row counting loops with the helper. |
cpp/src/parquet/column_reader.cc |
Apply the helper to value counting, record delimiting, and skipping. |
Review effort: Balanced
|
It looks like an interesting change. It might be worth submitting to LLVM to optimize the implementation of |
|
There're existing discussion about this, see: llvm/llvm-project#204897 But anyway if you want to benefit from this, you need at least |
| // unrolled ladder of 8/16/24-wide compares instead of scalar code. | ||
| constexpr int64_t kSmallLevels = 32; | ||
| constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max(); | ||
| if (num_levels < kSmallLevels) { |
There was a problem hiding this comment.
Does it make sense to keep this if branch, rather than letting LLVM automatically generate optimized code for short loops?
There was a problem hiding this comment.
I don't know, it would making the very short loops (9-31 elements?) a bit slower ( like 1.5ns -> 3ns ) on my Mac M4 Pro. If this doesn't matter just counting by chunk is ok.
…ader and writer ### Rationale for this change Follow-up to the `CountIf16` helper: use it for the level counting loops in the Parquet column reader/writer and in the file scanner, replacing open-coded scalar loops and `std::count` calls. Converted sites: - `TypedColumnReaderImpl::ReadLevelsInCurrentPage`: count non-null values. - `TypedRecordReader::SkipRecordsInBufferNonRepeated`: count values to skip. - `TypedRecordReader::DelimitRecords`: count records (rep level == 0) and values seen (def level == max). - `TypedRecordReader::ReadDenseForOptionalInBuffer`: count values to read. - `TypedColumnWriterImpl::WriteLevels`: count values to write (both overloads) and records written (rep level == 0). - `TypedColumnWriterImpl::MaybeCalculateValidityBits`: count values to write and spaced values to write. - `ScanFileContents`: count rows (rep level == 0). Sites are left as is where counting is not the bottleneck: the per-element bitmap loops in `DefRepLevelsToListInfo` also build validity bitmaps and write list offsets for every level, `page_index.cc` counts pages rather than levels, and the non-nullable validation loop in `WriteArrow` needs to `break` early on the first mismatch. `UpdateLevelHistogram`'s `max_level == 1` fast path is also left as is: its levels are known to be 0 or 1, so it sums them with `std::accumulate` in the `int16_t` space, which is a single 16-bit add per vector, while counting the 1s costs a compare and a subtract. ### What changes are included in this PR? Use `CountIf16` for the level counting call sites above. ### Are these changes tested? Covered by the existing Parquet reader, writer and Arrow reader/writer tests. ### Are there any user-facing changes? No.
e1291b9 to
7ce5324
Compare
|
Does this even make a difference in a benchmark? There are so many expensive things happening when reading or writing Parquet (such as compression or encoding) that I doubt the cost of |
|
@ursabot please benchmark lang=C++ |
|
Benchmark runs are scheduled for commit 7ce5324. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete. |
Generally speaking, no. Some optimization might benifit from same optimization in #55308 , write side previously counting the numbers toooo slow. It has only a few optimization during read =_= |
|
Hmm, there seem to be some small but real speedups on the benchmarks. |
| /// It is kept out of line so that the common path (a single Parquet page chunk, | ||
| /// always short enough for one accumulator) is a plain loop without any chunk | ||
| /// bookkeeping in front of it. |
There was a problem hiding this comment.
Was it measured to make a difference?
Was this AI-generated?
There was a problem hiding this comment.
Was it measured to make a difference?
Generally speaking yes. The asm and benchmark shows this could benefit the performance.
Was this AI-generated?
Yes, but all changes are verified and reviewed myself
There was a problem hiding this comment.
This is measured to make a difference. Generally see https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion
A code sample is here: https://godbolt.org/z/56oWjTGWW
LLVM will hoist some checking to outside, and it will trying to broadcast the level, and trying to simd comparing, which is useful for large loop but in-efficient for short loop.
I think keeping or reming it is both ok for me, if we think just making short inputs a bit slower is ok, we can just keep a while (num_levels > 0) loop.
| // Short inputs are counted first, before the wide loop and before the chunk | ||
| // clamp, because a handful of levels cannot amortize either. Running the | ||
| // small case first is also how folly::simd::detail::simdForEach orders its | ||
| // single steps before the unrolled ones, so that short inputs never pay for | ||
| // the unrolled set up. | ||
| // | ||
| // 32 is the widest per-iteration body this reduction gets: a packed int16_t | ||
| // register holds 8 levels with SSE2/NEON and 16 with AVX2, but clang runs up | ||
| // to 32 levels per iteration on NEON (four registers), so shorter inputs | ||
| // never reach a full iteration on the narrow targets. The bound also has to | ||
| // be a compile-time power of two, which is what turns the loop into an | ||
| // unrolled ladder of 8/16/24-wide compares instead of scalar code. |
There was a problem hiding this comment.
This lengthy comment is AI-generated, right?
There was a problem hiding this comment.
Yeah, I'll rewrite. it myself
| // unrolled ladder of 8/16/24-wide compares instead of scalar code. | ||
| constexpr int64_t kSmallLevels = 32; | ||
| constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max(); | ||
| if (num_levels < kSmallLevels) { |
There was a problem hiding this comment.
What's the point of this if? It's exactly the same code as the other loop below.
There was a problem hiding this comment.
I found this nice for small benchmark, actuall when if (n < 32), directly loop is slower than std::count.
n < 32 will tell the compiler to unroll the loop for it. when n <= 8 it's slower, when n in (8, 32), this making the counting faster.
Without n < 32 ( or this small check ), it just fall into loop.
Detail see: https://godbolt.org/z/nj1vKvqPY
Performance on my MacOS:
| n | u16_guard | u16 | i64_guard | std_count | i64_loop |
|---|---|---|---|---|---|
| 1 | 2.38 | 1.53 | 1.93 | 1.43 | 1.14 |
| 4 | 3.16 | 2.29 | 3.15 | 1.89 | 2.06 |
| 7 | 4.12 | 3.82 | 4.00 | 3.45 | 3.83 |
| 8 | 1.76 | 2.35 | 2.00 | 2.29 | 2.65 |
| 13 | 3.73 | 3.14 | 4.13 | 3.78 | 3.18 |
| 16 | 2.00 | 3.43 | 2.80 | 3.52 | 3.81 |
| 24 | 1.99 | 4.02 | 4.21 | 4.51 | 4.41 |
| 28 | 3.81 | 4.28 | 5.00 | 5.12 | 5.02 |
| 31 | 4.69 | 5.87 | 5.87 | 6.17 | 6.15 |
| 32 | 2.05 | 2.35 | 6.76 | 6.63 | 6.56 |
| 256 | 5.58 | 5.72 | 37.60 | 38.70 | 37.81 |
| 2048 | 38.70 | 39.25 | 278.86 | 277.19 | 283.70 |
I can remove this if we think it's a benchmark hack
There was a problem hiding this comment.
Detail see: https://godbolt.org/z/nj1vKvqPY
Micro-benchmarks like this do not tell us about performance when called from Parquet C++ reader code. Inlining may be different, code placement may be different, etc.
There was a problem hiding this comment.
Yeah, the main difference is that it helps unroll n<32's vectorization, and tells the compiler the loop is for element >32 && <15536. But I can remove this, any way CountIf16 mainly benefit from Chunked counting.
|
|
||
| MinMax PARQUET_EXPORT FindMinMax(const int16_t* levels, int64_t num_levels); | ||
|
|
||
| /// Counts the levels that satisfy `pred`, in chunks small enough that a |
There was a problem hiding this comment.
Can you move these functions to a private/internal header?
There was a problem hiding this comment.
Emmm it's already parquet::internal in a internal header?
There was a problem hiding this comment.
Hmm, the header is not internal, but that can be a separate issue.
| return count; | ||
| } | ||
| if (num_levels > kMaxCount) { | ||
| return CountIf16Chunked(levels, num_levels, pred); |
There was a problem hiding this comment.
Should use perfect forwarding?
| return CountIf16Chunked(levels, num_levels, pred); | |
| return CountIf16Chunked(levels, num_levels, std::forward<Predicate>(pred)); |
|
It's a bit late in my tz, I'll resolve comments tomorrow night |
|
I'm a bit busy this two days, I'll solve the comments this Sunday, sorry for delaying =_= |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… · New These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… · New These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… · New The PR description mentions introducing anum_levels < 32fast-path because small inputs may not… · New
Review effort: Lite
| for (int64_t i = levels_position_; i < position_end; ++i) { | ||
| records_read += rep_levels[i] == 0; | ||
| } | ||
| records_read += CountLevels(rep_levels + levels_position_, stride, 0); |
| *values_seen = CountLevels(def_levels + level, levels_position_ - level, | ||
| this->max_def_level()); |
| *values_to_read += CountLevels(def_levels + start_levels_position, | ||
| levels_position_ - start_levels_position, | ||
| this->max_def_level()); |
| inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels, | ||
| Predicate&& pred) { | ||
| constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max(); | ||
| if (num_levels > kMaxCount) { | ||
| return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred)); | ||
| } | ||
| uint16_t count = 0; | ||
| for (int64_t i = 0; i < num_levels; ++i) { | ||
| count += pred(levels[i]) ? 1 : 0; | ||
| } | ||
| return count; | ||
| } |
| } | ||
| *out_values_to_write += | ||
| internal::CountLevels(def_levels, batch_size, level_info_.def_level); | ||
| *out_spaced_values_to_write += internal::CountLevelsGreaterEqual( |
There was a problem hiding this comment.
This function is used only once, I don't know whether I should extract it or not ( DType was dispatch multiple times, so I think it's ok to extract it)
| /// It is kept out of line so that the common path (a single Parquet page chunk, | ||
| /// always short enough for one accumulator) is a plain loop without any chunk | ||
| /// bookkeeping in front of it. |
There was a problem hiding this comment.
This is measured to make a difference. Generally see https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion
A code sample is here: https://godbolt.org/z/56oWjTGWW
LLVM will hoist some checking to outside, and it will trying to broadcast the level, and trying to simd comparing, which is useful for large loop but in-efficient for short loop.
I think keeping or reming it is both ok for me, if we think just making short inputs a bit slower is ok, we can just keep a while (num_levels > 0) loop.
| } | ||
|
|
||
| /// \brief Count the levels equal to `value`. | ||
| inline int64_t CountLevels(const int16_t* levels, int64_t num_levels, int16_t value) { |
There was a problem hiding this comment.
Here I found CountLevelsIf will be compiled into DType multiple times, which making output binary 0.5% more larger whichout linker's help. So I extract a short CountLevels here to help compiler able to not copy and paste this function everywhere 😅
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
10 open issues
The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… · New The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… · New The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… · New The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… · New The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… · New These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same…CountLevelsIfChunkedis a templated function defined in a header; even withARROW_NOINLINE, it… · New The PR description mentions introducing anum_levels < 32fast-path because small inputs may not…
Review effort: Lite
| ARROW_DCHECK_LE(*num_def_levels, std::numeric_limits<int32_t>::max()); | ||
| *non_null_values_to_read += static_cast<int32_t>( | ||
| std::count(def_levels, def_levels + *num_def_levels, this->max_def_level())); | ||
| internal::CountLevels(def_levels, *num_def_levels, this->max_def_level())); |
| CountLevels(def_levels() + levels_position_, | ||
| remaining_levels_pos - levels_position_, this->max_def_level())); |
| for (int64_t i = levels_position_; i < position_end; ++i) { | ||
| records_read += rep_levels[i] == 0; | ||
| } | ||
| records_read += CountLevels(rep_levels + levels_position_, stride, 0); |
| *values_seen = | ||
| CountLevels(def_levels + level, levels_position_ - level, this->max_def_level()); |
| CountLevels(def_levels + start_levels_position, | ||
| levels_position_ - start_levels_position, this->max_def_level()); |
| // Fallback for num_levels > uint16_t::max(), counting in chunks to avoid overflow. | ||
| template <typename Predicate> | ||
| ARROW_NOINLINE int64_t CountLevelsIfChunked(const int16_t* levels, int64_t num_levels, | ||
| Predicate&& pred) { | ||
| constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max(); | ||
| int64_t total = 0; | ||
| while (num_levels > 0) { | ||
| const int64_t chunk_size = std::min(num_levels, kMaxChunkSize); | ||
| uint16_t chunk_count = 0; | ||
| for (int64_t i = 0; i < chunk_size; ++i) { | ||
| chunk_count += pred(levels[i]) ? 1 : 0; | ||
| } | ||
| total += chunk_count; | ||
| levels += chunk_size; | ||
| num_levels -= chunk_size; | ||
| } | ||
| return total; | ||
| } | ||
|
|
||
| /// \brief Count the levels that satisfy `pred`. | ||
| /// | ||
| /// Accumulates matches in a `uint16_t` so the compiler can vectorize the loop | ||
| /// using 16-bit lanes without widening each comparison result to 64 bits. | ||
| /// Inputs longer than `uint16_t::max()` are counted in chunks to avoid overflow. | ||
| template <typename Predicate> | ||
| inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels, | ||
| Predicate&& pred) { | ||
| constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max(); | ||
| if (num_levels > kMaxChunkSize) { | ||
| return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred)); | ||
| } |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
16 open issues
The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… The file mixesinternal::CountLevels(...)(line 1551) with unqualifiedCountLevels(...)calls… These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same… These calls useCountLevelswithout theinternal::qualifier, while earlier changes in the same…CountLevelsis called unqualified here, while other changes in this file use… · New These calls use unqualifiedCountLevels, but other call sites in the same file use… · New These calls use unqualifiedCountLevels, but other call sites in the same file use… · New The PR description mentions that non-template wrappers allow ODR deduplication of the out-of-line… · New The PR description mentions that non-template wrappers allow ODR deduplication of the out-of-line… · NewCountLevelsIfChunkedis a templated function defined in a header; even withARROW_NOINLINE, it… The PR description mentions introducing anum_levels < 32fast-path because small inputs may not… Same as above: this unqualifiedCountLevelscall is inconsistent with other call sites using… · New
Review effort: Lite
| CountLevels(def_levels() + levels_position_, | ||
| remaining_levels_pos - levels_position_, this->max_def_level())); |
| for (int64_t i = levels_position_; i < position_end; ++i) { | ||
| records_read += rep_levels[i] == 0; | ||
| } | ||
| records_read += CountLevels(rep_levels + levels_position_, stride, 0); |
| *values_seen = | ||
| CountLevels(def_levels + level, levels_position_ - level, this->max_def_level()); |
| // Fallback for num_levels > uint16_t::max(), counting in chunks to avoid overflow. | ||
| template <typename Predicate> | ||
| ARROW_NOINLINE int64_t CountLevelsIfChunked(const int16_t* levels, int64_t num_levels, | ||
| Predicate&& pred) { | ||
| constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max(); | ||
| int64_t total = 0; | ||
| while (num_levels > 0) { | ||
| const int64_t chunk_size = std::min(num_levels, kMaxChunkSize); | ||
| uint16_t chunk_count = 0; | ||
| for (int64_t i = 0; i < chunk_size; ++i) { | ||
| chunk_count += pred(levels[i]) ? 1 : 0; | ||
| } | ||
| total += chunk_count; | ||
| levels += chunk_size; | ||
| num_levels -= chunk_size; | ||
| } | ||
| return total; | ||
| } |
| inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels, | ||
| Predicate&& pred) { | ||
| constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max(); | ||
| if (num_levels > kMaxChunkSize) { | ||
| return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred)); | ||
| } | ||
| uint16_t count = 0; | ||
| for (int64_t i = 0; i < num_levels; ++i) { | ||
| count += pred(levels[i]) ? 1 : 0; | ||
| } | ||
| return count; | ||
| } | ||
|
|
||
| /// \brief Count the levels equal to `value`. | ||
| inline int64_t CountLevels(const int16_t* levels, int64_t num_levels, int16_t value) { | ||
| return CountLevelsIf(levels, num_levels, | ||
| [value](int16_t level) { return level == value; }); | ||
| } | ||
|
|
||
| /// \brief Count the levels greater than or equal to `min_level`. | ||
| inline int64_t CountLevelsGreaterEqual(const int16_t* levels, int64_t num_levels, | ||
| int16_t min_level) { | ||
| return CountLevelsIf(levels, num_levels, | ||
| [min_level](int16_t level) { return level >= min_level; }); | ||
| } |
| CountLevels(def_levels + start_levels_position, | ||
| levels_position_ - start_levels_position, this->max_def_level()); |
|
Some file are already inside namespace parquet::internal; Copilot Lite mode just missed the surrounding file context. I think it's f**king mad 😅 |



Rationale for this change
std::countoverint16_tlevels widens every comparison result to a 64-bit accumulator, preventing efficient 16-bit lane vectorization. Accumulating inuint16_tchunks keeps the reduction in 16-bit lanes without overflowing:-O2x86 AVX2 & ARM NEON): https://godbolt.org/z/rvobWjrfaBefore x86:
After x86:
Before ARM:
After ARM:
Some notes:
uint16_t::max()(65535) use an out-of-line chunked fallback (CountLevelsIfChunked), so the common single-page path avoids outer-loop LICM hoisting of vector broadcasts (vpbroadcastw+vzeroupperon x86 AVX2 /dup.8hon NEON) on short inputs.CountLevels/CountLevelsGreaterEqualwrappers are provided on top ofCountLevelsIfto keep call sites concise and let ODR deduplicate the out-of-line fallback across physical type instantiations.What changes are included in this PR?
Add
CountLevels,CountLevelsGreaterEqual, andCountLevelsIfto optimizeint16_tlevel counting in Parquet reader, writer, and scanner.Are these changes tested?
Yes, unit tests in
level_conversion_test.ccand covered by existing Parquet reader/writer tests.Are there any user-facing changes?
No
Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: