Repository navigation
GH-51223: [C++] Fix copying sliced boolean arrays - #51240
Conversation
|
|
There was a problem hiding this comment.
🟢 Approval recommended
The boolean ArraySpan copy now correctly accounts for slice offsets and the added regression tests directly cover the reported failure modes.
Pull request overview
Fixes incorrect bit indexing when copying from sliced boolean ArraySpans by ensuring the slice offset is applied during bitmap copies, and adds regression tests covering the affected compute kernels.
Changes:
- Apply
ArraySpan::offsetto the bit index when copying boolean values viaCopyDataUtils<BooleanType>. - Add regression tests for sliced boolean inputs in
replace_with_mask,fill_null_forward, andfill_null_backward.
File summaries
| File | Description |
|---|---|
| cpp/src/arrow/compute/kernels/copy_data_internal.h | Fix boolean ArraySpan copy to include in.offset in the bit index passed to CopyBitmap. |
| cpp/src/arrow/compute/kernels/vector_replace_test.cc | Add regression tests that exercise non-byte-aligned (offset=3) sliced boolean arrays for replace_with_mask and fill-null kernels. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Tested this against the reproducer from #51223: with the branch built locally, |
… verified against the reporter's reproducer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The fix correctly accounts for boolean slice offsets in the copy path and is backed by targeted regression tests for the reported failing kernels.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
Thanks for validating this against the reproducer and for spotting the misplaced comment. Fixed in efb7d1f. |
| static void CopyData(const DataType&, const uint8_t* in, const int64_t in_offset, | ||
| uint8_t* out, const int64_t out_offset, const int64_t length) { | ||
| arrow::internal::CopyBitmap(in, in_offset, length, out, out_offset); | ||
| } |
There was a problem hiding this comment.
Is this one ever called directly without accounting for the input offset?
If so, then we must fix those use cases as well. Otherwise, perhaps make this method protected to avoid external usage?
There was a problem hiding this comment.
Yes, I checked the direct uses of the uint8_t* overload. They already pass an absolute bit offset, including the source ArraySpan::offset (for example, CopyValues in scalar_if_else.cc passes array.offset + in_offset).
The affected callers use the ArraySpan overload and pass an offset relative to that span, so adding in.offset there matches the fixed-width specializations and fixes the sliced boolean case. The raw-pointer overload still needs to remain accessible for callers that already supply an absolute offset.
…ibutor proposes the type fix on apache/arrow-go#1298
…ainer's offset question
pitrou
left a comment
There was a problem hiding this comment.
Thanks for the explanation @A-makarim . This looks good to me, I'll wait for additional CI and then we can merge.
Fixes apache#51223 Signed-off-by: Abdul Azeem Makarim <114302821+A-makarim@users.noreply.github.com>
efb7d1f to
398dcdf
Compare
|
Rebased. |
…#368) ## Summary Fixes a wrong answer on main since ADR 0041's SQ7a (#360), found by `diff-random`. A select item's CASE over an IN subquery gave one row the wrong value under OFFSET: over the synthetic fixtures, `SELECT ct_id, CASE WHEN ct_id IN (SELECT rd_city FROM riders WHERE rd_city IS NOT NULL) THEN 1 ELSE 0 END FROM cities OFFSET 2` gave `4|1` for a city that has no rider. **Cause: Arrow 25's `replace_with_mask` kernel**, which `Scatter` in `src/exec/compute.cc` calls. - It reads a sliced BOOLEAN's values from the first bit of its buffer and ignores the array's offset. It reads the validity and the mask correctly. This is upstream [apache/arrow#51223](apache/arrow#51223), fixed for Arrow 26 by [#51240](apache/arrow#51240). - Without ORDER BY, the CASE is computed above the LIMIT. The LIMIT's window slices the mark join's batch, so the mark reaches the CASE with an offset, and the kernel reads values as many rows early as the offset. **Fix.** `Scatter` copies a BOOLEAN that has an offset to offset 0 (its bitmaps only) before calling the kernel, for both of the kernel's arrays (`UnsliceBoolean`). **What it broke.** A mark under OFFSET, with or without LIMIT, wherever a scatter hands it over whole: - a CASE's WHEN or a later WHEN; - an OR's or AND's argument after the first, while every row is still undecided; - COALESCE over such a CASE; - the `= ANY` and `<> ALL` forms, and a three-valued mark over a set that holds a NULL. These were right before the fix: - NOT over a mark, and a mark as an OR's first argument; - marks in WHERE, ORDER BY items, GROUP BY keys and aggregate arguments; - ORDER BY ... OFFSET, where the CASE is computed below the sort. Every other Arrow kernel antb1 runs on BOOLEANs was checked at offsets 0 to 19, with and without NULLs, and reads the offset: and, and_not, and_kleene, or, or_kleene, invert, coalesce, is_valid, is_null, if_else, equal, filter, take and cast. Before SQ7a's marks, no SQL could put a BOOLEAN column under a scatter. ## Type of change - [ ] feat: new SQL, CLI or engine capability - [x] fix: bug fix - [ ] perf: performance improvement - [ ] refactor, test, docs, build, ci or chore - [ ] Breaking change (CLI, output format or semantics); also add the `breaking-change` label ## Verification ```text $ pixi run test 100% tests passed out of 4027 ANTB1-TESTS: PASS preset=dev $ pixi run ci 100% tests passed out of 4027 (Clang Debug -Werror, no warning: lines) $ clang-tidy -p build/tidy --warnings-as-errors='*' src/exec/compute.cc src/exec/tests/compute_test.cc; pixi run tidy exit 0; exit 0 $ pixi run lint every hook passes but shellcheck, whose findings are the known macOS-only SC2218 in scripts this PR does not touch $ ASan build (pixi run asan stops at test discovery on macOS; suites run from build/ci-asan, detect_leaks=0) antb1_exec_tests 526 passed, antb1_engine_tests 32 antb1-slt run subqueries/*.slt, limit.slt, case.slt, boolean.slt: 2367 records, PASS at 1 and 4 threads $ pixi run diff-random (default seed), ANTB1_DIFF_SEED=3, ANTB1_DIFF_SEED=6 DIFF: PASS queries=2000 failed=0 unsupported=0 (each) # The five failing diff-random cases, over the twelve star tables: seed 3 cases 21273 and 22971, seed 4 case # 27508, seed 6 cases 24720 and 39177 each failed before the fix and passes after it # Star-table runs of 40,000 queries: seeds 3, 4 and 6; seed 3 again with --batch-size 97 --threads 4 DIFF: PASS failed=0 (each) ``` **Tests added:** - `exec.ComputeTest.ABooleanOfASlicedBatchIsScatteredFromItsOwnRows` checks every scatter of a sliced BOOLEAN at offsets 1 to 9, with and without NULLs, against the same rows unsliced. It failed before the fix. - `exec.ComputeTest.ArrowsReplaceWithMaskReadsASlicedBooleanFromItsBuffersFirstBit` pins the kernel's misread. It fails once the Arrow in use is fixed (Arrow 26), and its message says the workaround and the test can then go. - 13 records in `tests/slt/cases/subqueries/in_marks.slt`: - 10 under OFFSET without ORDER BY. They rely on file order, as `limit/limit.slt` does, and each fails without the fix; - NOT IN as the contrast; - two under ORDER BY ... OFFSET. ## Checklist - [x] `pixi run check` passes locally: its lint step stops on the macOS-only shellcheck findings, so `pixi run lint` and the tests were run separately (above) - [x] Tests cover the change (above) - [x] Docs updated where behavior, commands or architecture changed, or not needed: not needed, no documented behavior changes - [x] No ClickBench-derived data is committed: no Parquet files, query answers or values from `hits` (ADR-0006) - [x] Nothing derived from TPC-H or TPC-DS is committed: no query text or fragments, data, answers or TPC tools (ADR-0006) - [x] Changes to governance paths (see `.github/CODEOWNERS`) were agreed with a maintainer: none touched ## AI assistance - [ ] No AI assistance - [x] AI-assisted. Tools and what they did: Claude Code (Claude Opus 5.5) found the failures with `diff-random`, traced them to the Arrow kernel, wrote the fix, the unit tests and the records, checked the other BOOLEAN kernels, and reviewed the diff with a reviewer agent. - Accountable human (has read and understands the whole diff): @Hor911 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Rationale for this change
Boolean values are bit-packed. When copying values from a sliced boolean array,
the copy path did not include the array's slice offset. This caused
fill_null_forward,fill_null_backward, andreplace_with_maskto readvalues from earlier positions in the parent array.
What changes are included in this PR?
ArraySpan::offsetto the bit index when copying boolean values froman
ArraySpan.replace_with_mask,fill_null_forward, andfill_null_backward.Are these changes tested?
Yes. I ran
arrow-compute-vector-testlocally.Are there any user-facing changes?
Only a bugfix.
This PR contains a "Critical Fix".
This fixes a bug that caused compute operations on sliced boolean arrays with
a non-zero offset to produce incorrect values.