Skip to content

GH-55399: [C++][Parquet] Minor optimize level counting with CountLevels - #55408

Open
mapleFU wants to merge 5 commits into
apache:mainfrom
mapleFU:parquet/gh-55399-level-count-if16
Open

mapleFU wants to merge 5 commits into
apache:mainfrom
mapleFU:parquet/gh-55399-level-count-if16

Conversation

@mapleFU

@mapleFU mapleFU commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

std::count over int16_t levels widens every comparison result to a 64-bit accumulator, preventing efficient 16-bit lane vectorization. Accumulating in uint16_t chunks keeps the reduction in 16-bit lanes without overflowing:

  1. x86: https://godbolt.org/z/Gqc7fxe55
  2. ARM: https://godbolt.org/z/sGhG8crYa
  3. Fast-path vs out-of-line chunked fallback (-O2 x86 AVX2 & ARM NEON): https://godbolt.org/z/rvobWjrfa

Before x86:

vmovq      (%rdi,%rax,2), %xmm6      # Load 8 bytes = 4  levels
vpcmpeqw   %xmm1, %xmm6, %xmm6
vpmovzxwq  %xmm6, %ymm6              # 8×16 bits→ 4×64 bits,only 4 lanes
vpand      %ymm2, %ymm6, %ymm6       # & 1
vpaddq     %ymm6, %ymm0, %ymm0       # 64 bits adds

After x86:

vpcmpeqw   (%rdi,%r8,2), %ymm1, %ymm5   # compare 16 levels
vpsubw     %ymm5, %ymm0, %ymm0          # add 16 lanes

Before ARM:

ldp q18,q19,[x8,#-16]      ; 2 cmeq 16 bits mask
cmeq v18.8h, v18.8h, v16.8h
cmeq v19.8h, v19.8h, v16.8h
ushll2 v20.4s, v18.8h, #0  ; 16→32 bits
ushll  v18.4s, v18.4h, #0
...
and    v22.16b, v22.16b, v17.16b   ;
add    v4.2d, v4.2d, v20.2d        ;
subs x14,x14,#16 ; b.ne

After ARM:

ldp q3,q4,[x9,#-16]
cmeq v3.8h, v3.8h, v2.8h
cmeq v4.8h, v4.8h, v2.8h
sub  v0.8h, v0.8h, v3.8h
sub  v1.8h, v1.8h, v4.8h
subs/add/b.ne

Some notes:

  1. Inputs larger than uint16_t::max() (65535) use an out-of-line chunked fallback (CountLevelsIfChunked), so the common single-page path avoids outer-loop LICM hoisting of vector broadcasts (vpbroadcastw + vzeroupper on x86 AVX2 / dup.8h on NEON) on short inputs.
  2. Non-template CountLevels / CountLevelsGreaterEqual wrappers are provided on top of CountLevelsIf to keep call sites concise and let ODR deduplicate the out-of-line fallback across physical type instantiations.
  3. Optimizes both read and write paths (though level counting is only a small part of end-to-end Parquet I/O).

What changes are included in this PR?

Add CountLevels, CountLevelsGreaterEqual, and CountLevelsIf to optimize int16_t level counting in Parquet reader, writer, and scanner.

Are these changes tested?

Yes, unit tests in level_conversion_test.cc and covered by existing Parquet reader/writer tests.

Are there any user-facing changes?

No

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human

…ounting

### Rationale for this change

Counting how many of a run of int16_t levels match a predicate (e.g. how
many definition levels equal the max definition level, or how many
repetition levels are 0) is a hot loop in the Parquet reader and writer.
It is currently open-coded as a scalar loop or a std::count() call.

`std::count` accumulates into a 64-bit integer, which forces the compiler
to widen every 16-bit compare result before accumulating and prevents
efficient vectorization. Accumulating into a `uint16_t` instead lets the
compiler keep the whole reduction in 16-bit lanes (e.g. `cmeq.8h` +
`sub.8h` + `addv.8h` on Neon, 32 levels per iteration), and the chunked
loop keeps that accumulator from overflowing: at most `0xFFFF` levels are
counted before the accumulator is folded into the 64-bit total.

The chunk bookkeeping (clamping the chunk length, folding the partial
count, advancing the pointer) is a fixed cost per call, and for inputs
shorter than one vector iteration it dominates: counting 7 levels that
way is measurably slower than `std::count`, even though both end up
running the same scalar loop. Such short inputs are therefore counted
directly, without going through the chunk loop.

On an Apple M-series machine, counting 2048 int16 levels where every
tenth is missing drops from ~232 ns to ~32 ns (about 7x), while short
inputs are at least as fast as `std::count`.

### What changes are included in this PR?

- Add `parquet::internal::CountIf16` next to the other level scanning
  primitives in `parquet/level_comparison.h`. It is a header-only inline
  template, since the level conversion translation unit is compiled twice
  (standard and BMI2).
- Add unit tests covering empty input, a mixed predicate, short inputs
  around the vectorization threshold, and the `0xFFFF` chunk boundaries.
- Add `BM_CountPresentLevels{StdCount,CountIf16}` to the level conversion
  benchmark to track the improvement.

### Are these changes tested?

New `CountIf16.*` tests in `level_conversion_test.cc`, run as part of
`parquet-reader-test`.

### Are there any user-facing changes?

No.
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55399 has been automatically assigned in GitHub to PR creator.

@mapleFU
mapleFU marked this pull request as ready for review November 5, 2026 07:10
Copilot AI balanced review requested due to automatic review settings November 5, 2026 07:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the shared Parquet level-counting helper and its integrations.

No open issues 🎉

What changed in this PR

This PR addresses GH-55399 by introducing a vectorization-friendly helper for counting Parquet definition and repetition levels, then applying it across reading, writing, and size statistics.

Changes:

  • Add CountIf16 with 16-bit accumulators, overflow-safe chunking, and a short-input path.
  • Replace existing counting loops and std::count calls with the shared helper.
  • Add tests for predicates, short inputs, and chunk boundaries.
File Description
cpp/​src/​parquet/​size_statistics.cc Use the helper for binary-level histogram counts.
cpp/​src/​parquet/​level_conversion_test.cc Test counting correctness and chunk boundaries.
cpp/​src/​parquet/​level_comparison.h Implement CountIf16 and its chunked fallback.
cpp/​src/​parquet/​file_reader.cc Use the helper to count rows during file scans.
cpp/​src/​parquet/​column_writer.cc Replace value and row counting loops with the helper.
cpp/​src/​parquet/​column_reader.cc Apply the helper to value counting, record delimiting, and skipping.

Review effort: Balanced

@HuaHuaY

HuaHuaY commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

It looks like an interesting change. It might be worth submitting to LLVM to optimize the implementation of std::count in the standard library.

@mapleFU

mapleFU commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

There're existing discussion about this, see: llvm/llvm-project#204897

But anyway if you want to benefit from this, you need at least clang > 23.1, and I don't know when could LLVM implement this :-(

Comment thread cpp/src/parquet/level_comparison.h Outdated
// unrolled ladder of 8/16/24-wide compares instead of scalar code.
constexpr int64_t kSmallLevels = 32;
constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max();
if (num_levels < kSmallLevels) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to keep this if branch, rather than letting LLVM automatically generate optimized code for short loops?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, it would making the very short loops (9-31 elements?) a bit slower ( like 1.5ns -> 3ns ) on my Mac M4 Pro. If this doesn't matter just counting by chunk is ok.

Comment thread cpp/src/parquet/size_statistics.cc Resolved
Copilot AI balanced review requested due to automatic review settings November 5, 2026 12:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Parquet level-counting optimization.

No open issues 🎉

Review effort: Balanced

…ader and writer

### Rationale for this change

Follow-up to the `CountIf16` helper: use it for the level counting loops
in the Parquet column reader/writer and in the file scanner, replacing
open-coded scalar loops and `std::count` calls.

Converted sites:

- `TypedColumnReaderImpl::ReadLevelsInCurrentPage`: count non-null values.
- `TypedRecordReader::SkipRecordsInBufferNonRepeated`: count values to skip.
- `TypedRecordReader::DelimitRecords`: count records (rep level == 0) and
  values seen (def level == max).
- `TypedRecordReader::ReadDenseForOptionalInBuffer`: count values to read.
- `TypedColumnWriterImpl::WriteLevels`: count values to write (both
  overloads) and records written (rep level == 0).
- `TypedColumnWriterImpl::MaybeCalculateValidityBits`: count values to
  write and spaced values to write.
- `ScanFileContents`: count rows (rep level == 0).

Sites are left as is where counting is not the bottleneck: the
per-element bitmap loops in `DefRepLevelsToListInfo` also build validity
bitmaps and write list offsets for every level, `page_index.cc` counts
pages rather than levels, and the non-nullable validation loop in
`WriteArrow` needs to `break` early on the first mismatch.

`UpdateLevelHistogram`'s `max_level == 1` fast path is also left as is: its
levels are known to be 0 or 1, so it sums them with `std::accumulate` in the
`int16_t` space, which is a single 16-bit add per vector, while counting the
1s costs a compare and a subtract.

### What changes are included in this PR?

Use `CountIf16` for the level counting call sites above.

### Are these changes tested?

Covered by the existing Parquet reader, writer and Arrow reader/writer
tests.

### Are there any user-facing changes?

No.
@mapleFU
mapleFU force-pushed the parquet/gh-55399-level-count-if16 branch from e1291b9 to 7ce5324 Compare November 5, 2026 12:35
Copilot AI balanced review requested due to automatic review settings November 5, 2026 12:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the Parquet level-counting optimization.

No open issues 🎉

Review effort: Balanced

@pitrou

pitrou commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Does this even make a difference in a benchmark? There are so many expensive things happening when reading or writing Parquet (such as compression or encoding) that I doubt the cost of std::count is important...

@pitrou

pitrou commented Nov 9, 2026

Copy link
Copy Markdown
Member

@ursabot please benchmark lang=C++

@rok

rok commented Nov 9, 2026

Copy link
Copy Markdown
Member

Benchmark runs are scheduled for commit 7ce5324. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

@mapleFU

mapleFU commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Member
Does this even make a difference in a benchmark? There are so many expensive things happening when reading or writing Parquet (such as compression or encoding) that I doubt the cost of std::count is important...

Generally speaking, no. Some optimization might benifit from same optimization in #55308 , write side previously counting the numbers toooo slow. It has only a few optimization during read =_=

@pitrou

pitrou commented Nov 9, 2026

Copy link
Copy Markdown
Member

Hmm, there seem to be some small but real speedups on the benchmarks.

Comment thread cpp/src/parquet/level_comparison.h Outdated
Comment on lines +43 to +45
/// It is kept out of line so that the common path (a single Parquet page chunk,
/// always short enough for one accumulator) is a plain loop without any chunk
/// bookkeeping in front of it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it measured to make a difference?
Was this AI-generated?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it measured to make a difference?

Generally speaking yes. The asm and benchmark shows this could benefit the performance.

Was this AI-generated?

Yes, but all changes are verified and reviewed myself

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is measured to make a difference. Generally see https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion

A code sample is here: https://godbolt.org/z/56oWjTGWW

LLVM will hoist some checking to outside, and it will trying to broadcast the level, and trying to simd comparing, which is useful for large loop but in-efficient for short loop.

I think keeping or reming it is both ok for me, if we think just making short inputs a bit slower is ok, we can just keep a while (num_levels > 0) loop.

Comment thread cpp/src/parquet/level_comparison.h Outdated
Comment on lines +82 to +93
// Short inputs are counted first, before the wide loop and before the chunk
// clamp, because a handful of levels cannot amortize either. Running the
// small case first is also how folly::simd::detail::simdForEach orders its
// single steps before the unrolled ones, so that short inputs never pay for
// the unrolled set up.
//
// 32 is the widest per-iteration body this reduction gets: a packed int16_t
// register holds 8 levels with SSE2/NEON and 16 with AVX2, but clang runs up
// to 32 levels per iteration on NEON (four registers), so shorter inputs
// never reach a full iteration on the narrow targets. The bound also has to
// be a compile-time power of two, which is what turns the loop into an
// unrolled ladder of 8/16/24-wide compares instead of scalar code.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lengthy comment is AI-generated, right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'll rewrite. it myself

Comment thread cpp/src/parquet/level_comparison.h Outdated
// unrolled ladder of 8/16/24-wide compares instead of scalar code.
constexpr int64_t kSmallLevels = 32;
constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max();
if (num_levels < kSmallLevels) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of this if? It's exactly the same code as the other loop below.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this nice for small benchmark, actuall when if (n < 32), directly loop is slower than std::count.
n < 32 will tell the compiler to unroll the loop for it. when n <= 8 it's slower, when n in (8, 32), this making the counting faster.

Without n < 32 ( or this small check ), it just fall into loop.

Detail see: https://godbolt.org/z/nj1vKvqPY

Performance on my MacOS:

n u16_guard u16 i64_guard std_count i64_loop
1 2.38 1.53 1.93 1.43 1.14
4 3.16 2.29 3.15 1.89 2.06
7 4.12 3.82 4.00 3.45 3.83
8 1.76 2.35 2.00 2.29 2.65
13 3.73 3.14 4.13 3.78 3.18
16 2.00 3.43 2.80 3.52 3.81
24 1.99 4.02 4.21 4.51 4.41
28 3.81 4.28 5.00 5.12 5.02
31 4.69 5.87 5.87 6.17 6.15
32 2.05 2.35 6.76 6.63 6.56
256 5.58 5.72 37.60 38.70 37.81
2048 38.70 39.25 278.86 277.19 283.70

I can remove this if we think it's a benchmark hack

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detail see: https://godbolt.org/z/nj1vKvqPY

Micro-benchmarks like this do not tell us about performance when called from Parquet C++ reader code. Inlining may be different, code placement may be different, etc.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the main difference is that it helps unroll n<32's vectorization, and tells the compiler the loop is for element >32 && <15536. But I can remove this, any way CountIf16 mainly benefit from Chunked counting.

Comment thread cpp/src/parquet/level_comparison.h Outdated

MinMax PARQUET_EXPORT FindMinMax(const int16_t* levels, int64_t num_levels);

/// Counts the levels that satisfy `pred`, in chunks small enough that a

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move these functions to a private/internal header?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emmm it's already parquet::internal in a internal header?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, the header is not internal, but that can be a separate issue.

Comment thread cpp/src/parquet/level_comparison.h Resolved
Comment thread cpp/src/parquet/level_comparison.h Outdated
return count;
}
if (num_levels > kMaxCount) {
return CountIf16Chunked(levels, num_levels, pred);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use perfect forwarding?

Suggested change
return CountIf16Chunked(levels, num_levels, pred);
return CountIf16Chunked(levels, num_levels, std::forward<Predicate>(pred));

Comment thread cpp/src/parquet/size_statistics.cc Resolved
@mapleFU

mapleFU commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

It's a bit late in my tz, I'll resolve comments tomorrow night

@mapleFU

mapleFU commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

I'm a bit busy this two days, I'll solve the comments this Sunday, sorry for delaying =_=

@mapleFU mapleFU changed the title GH-55399: [C++][Parquet] Minor optimize level counting with CounfIf16 GH-55399: [C++][Parquet] Minor optimize level counting with CountIf16 Nov 16, 2026
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:44
@mapleFU mapleFU changed the title GH-55399: [C++][Parquet] Minor optimize level counting with CountIf16 GH-55399: [C++][Parquet] Minor optimize level counting with CountLevels Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

Review effort: Lite

for (int64_t i = levels_position_; i < position_end; ++i) {
records_read += rep_levels[i] == 0;
}
records_read += CountLevels(rep_levels + levels_position_, stride, 0);
Comment thread cpp/src/parquet/column_reader.cc Outdated
Comment on lines +2147 to +2148
*values_seen = CountLevels(def_levels + level, levels_position_ - level,
this->max_def_level());
Comment thread cpp/src/parquet/column_reader.cc Outdated
Comment on lines +2237 to +2239
*values_to_read += CountLevels(def_levels + start_levels_position,
levels_position_ - start_levels_position,
this->max_def_level());
Comment on lines +66 to +77
inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels,
Predicate&& pred) {
constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max();
if (num_levels > kMaxCount) {
return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred));
}
uint16_t count = 0;
for (int64_t i = 0; i < num_levels; ++i) {
count += pred(levels[i]) ? 1 : 0;
}
return count;
}
}
*out_values_to_write +=
internal::CountLevels(def_levels, batch_size, level_info_.def_level);
*out_spaced_values_to_write += internal::CountLevelsGreaterEqual(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is used only once, I don't know whether I should extract it or not ( DType was dispatch multiple times, so I think it's ok to extract it)

Comment thread cpp/src/parquet/level_comparison.h Outdated
Comment on lines +43 to +45
/// It is kept out of line so that the common path (a single Parquet page chunk,
/// always short enough for one accumulator) is a plain loop without any chunk
/// bookkeeping in front of it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is measured to make a difference. Generally see https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion

A code sample is here: https://godbolt.org/z/56oWjTGWW

LLVM will hoist some checking to outside, and it will trying to broadcast the level, and trying to simd comparing, which is useful for large loop but in-efficient for short loop.

I think keeping or reming it is both ok for me, if we think just making short inputs a bit slower is ok, we can just keep a while (num_levels > 0) loop.

}

/// \brief Count the levels equal to `value`.
inline int64_t CountLevels(const int16_t* levels, int64_t num_levels, int16_t value) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I found CountLevelsIf will be compiled into DType multiple times, which making output binary 0.5% more larger whichout linker's help. So I extract a short CountLevels here to help compiler able to not copy and paste this function everywhere 😅

Copilot AI balanced review requested due to automatic review settings November 16, 2026 14:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARROW_DCHECK_LE(*num_def_levels, std::numeric_limits<int32_t>::max());
*non_null_values_to_read += static_cast<int32_t>(
std::count(def_levels, def_levels + *num_def_levels, this->max_def_level()));
internal::CountLevels(def_levels, *num_def_levels, this->max_def_level()));
Comment on lines +1966 to +1967
CountLevels(def_levels() + levels_position_,
remaining_levels_pos - levels_position_, this->max_def_level()));
for (int64_t i = levels_position_; i < position_end; ++i) {
records_read += rep_levels[i] == 0;
}
records_read += CountLevels(rep_levels + levels_position_, stride, 0);
Comment on lines +2146 to +2147
*values_seen =
CountLevels(def_levels + level, levels_position_ - level, this->max_def_level());
Comment on lines +2237 to +2238
CountLevels(def_levels + start_levels_position,
levels_position_ - start_levels_position, this->max_def_level());
Comment on lines +40 to +70
// Fallback for num_levels > uint16_t::max(), counting in chunks to avoid overflow.
template <typename Predicate>
ARROW_NOINLINE int64_t CountLevelsIfChunked(const int16_t* levels, int64_t num_levels,
Predicate&& pred) {
constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max();
int64_t total = 0;
while (num_levels > 0) {
const int64_t chunk_size = std::min(num_levels, kMaxChunkSize);
uint16_t chunk_count = 0;
for (int64_t i = 0; i < chunk_size; ++i) {
chunk_count += pred(levels[i]) ? 1 : 0;
}
total += chunk_count;
levels += chunk_size;
num_levels -= chunk_size;
}
return total;
}

/// \brief Count the levels that satisfy `pred`.
///
/// Accumulates matches in a `uint16_t` so the compiler can vectorize the loop
/// using 16-bit lanes without widening each comparison result to 64 bits.
/// Inputs longer than `uint16_t::max()` are counted in chunks to avoid overflow.
template <typename Predicate>
inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels,
Predicate&& pred) {
constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max();
if (num_levels > kMaxChunkSize) {
return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred));
}
Copilot AI balanced review requested due to automatic review settings November 16, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

16 open issues

Review effort: Lite

Comment on lines +1985 to +1986
CountLevels(def_levels() + levels_position_,
remaining_levels_pos - levels_position_, this->max_def_level()));
for (int64_t i = levels_position_; i < position_end; ++i) {
records_read += rep_levels[i] == 0;
}
records_read += CountLevels(rep_levels + levels_position_, stride, 0);
Comment on lines +2165 to +2166
*values_seen =
CountLevels(def_levels + level, levels_position_ - level, this->max_def_level());
Comment on lines +40 to +57
// Fallback for num_levels > uint16_t::max(), counting in chunks to avoid overflow.
template <typename Predicate>
ARROW_NOINLINE int64_t CountLevelsIfChunked(const int16_t* levels, int64_t num_levels,
Predicate&& pred) {
constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max();
int64_t total = 0;
while (num_levels > 0) {
const int64_t chunk_size = std::min(num_levels, kMaxChunkSize);
uint16_t chunk_count = 0;
for (int64_t i = 0; i < chunk_size; ++i) {
chunk_count += pred(levels[i]) ? 1 : 0;
}
total += chunk_count;
levels += chunk_size;
num_levels -= chunk_size;
}
return total;
}
Comment on lines +65 to +89
inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels,
Predicate&& pred) {
constexpr int64_t kMaxChunkSize = std::numeric_limits<uint16_t>::max();
if (num_levels > kMaxChunkSize) {
return CountLevelsIfChunked(levels, num_levels, std::forward<Predicate>(pred));
}
uint16_t count = 0;
for (int64_t i = 0; i < num_levels; ++i) {
count += pred(levels[i]) ? 1 : 0;
}
return count;
}

/// \brief Count the levels equal to `value`.
inline int64_t CountLevels(const int16_t* levels, int64_t num_levels, int16_t value) {
return CountLevelsIf(levels, num_levels,
[value](int16_t level) { return level == value; });
}

/// \brief Count the levels greater than or equal to `min_level`.
inline int64_t CountLevelsGreaterEqual(const int16_t* levels, int64_t num_levels,
int16_t min_level) {
return CountLevelsIf(levels, num_levels,
[min_level](int16_t level) { return level >= min_level; });
}
Comment on lines +2256 to +2257
CountLevels(def_levels + start_levels_position,
levels_position_ - start_levels_position, this->max_def_level());
@mapleFU

mapleFU commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Some file are already inside namespace parquet::internal; Copilot Lite mode just missed the surrounding file context. I think it's f**king mad 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants