Skip to content

GH-51223: [C++] Fix copying sliced boolean arrays - #51240

Merged
pitrou merged 2 commits into
apache:mainfrom
A-makarim:GH-51223-fix-sliced-boolean-arrays
Sep 23, 2026
Merged

pitrou merged 2 commits into
apache:mainfrom
A-makarim:GH-51223-fix-sliced-boolean-arrays

Conversation

@A-makarim

Copy link
Copy Markdown
Contributor

Rationale for this change

Boolean values are bit-packed. When copying values from a sliced boolean array,
the copy path did not include the array's slice offset. This caused
fill_null_forward, fill_null_backward, and replace_with_mask to read
values from earlier positions in the parent array.

What changes are included in this PR?

  • Apply ArraySpan::offset to the bit index when copying boolean values from
    an ArraySpan.
  • Add regression coverage for sliced boolean inputs in replace_with_mask,
    fill_null_forward, and fill_null_backward.

Are these changes tested?

Yes. I ran arrow-compute-vector-test locally.

Are there any user-facing changes?

Only a bugfix.

This PR contains a "Critical Fix".

This fixes a bug that caused compute operations on sliced boolean arrays with
a non-zero offset to produce incorrect values.

@A-makarim
A-makarim requested a review from pitrou as a code owner September 8, 2026 16:17
Copilot AI lite review requested due to automatic review settings September 8, 2026 16:17
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #51223 has no components, please add labels for components.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The boolean ArraySpan copy now correctly accounts for slice offsets and the added regression tests directly cover the reported failure modes.

Pull request overview

Fixes incorrect bit indexing when copying from sliced boolean ArraySpans by ensuring the slice offset is applied during bitmap copies, and adds regression tests covering the affected compute kernels.

Changes:

  • Apply ArraySpan::offset to the bit index when copying boolean values via CopyDataUtils<BooleanType>.
  • Add regression tests for sliced boolean inputs in replace_with_mask, fill_null_forward, and fill_null_backward.
File summaries
File Description
cpp/src/arrow/compute/kernels/copy_data_internal.h Fix boolean ArraySpan copy to include in.offset in the bit index passed to CopyBitmap.
cpp/src/arrow/compute/kernels/vector_replace_test.cc Add regression tests that exercise non-byte-aligned (offset=3) sliced boolean arrays for replace_with_mask and fill-null kernels.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@singhpratech

Copy link
Copy Markdown
Contributor

Tested this against the reproducer from #51223: with the branch built locally, fill_null_forward,
fill_null_backward and replace_with_mask give the same result for full.slice(3, 5) as for a
fresh copy of the same values, and both new tests fail with copy_data_internal.h from main, so
they cover the bug. One small thing: FillNullSlicedArray landed between the comment that describes
FillNullForwardChunkedArray and that test, so the comment now sits above the wrong one.

singhpratech added a commit to singhpratech/ArrowMetal that referenced this pull request Sep 9, 2026
… verified against the reporter's reproducer

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 9, 2026 22:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fix correctly accounts for boolean slice offsets in the copy path and is backed by targeted regression tests for the reported failing kernels.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@A-makarim

Copy link
Copy Markdown
Contributor Author

Thanks for validating this against the reproducer and for spotting the misplaced comment. Fixed in efb7d1f.

Comment on lines 38 to 41
static void CopyData(const DataType&, const uint8_t* in, const int64_t in_offset,
uint8_t* out, const int64_t out_offset, const int64_t length) {
arrow::internal::CopyBitmap(in, in_offset, length, out, out_offset);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this one ever called directly without accounting for the input offset?
If so, then we must fix those use cases as well. Otherwise, perhaps make this method protected to avoid external usage?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I checked the direct uses of the uint8_t* overload. They already pass an absolute bit offset, including the source ArraySpan::offset (for example, CopyValues in scalar_if_else.cc passes array.offset + in_offset).

The affected callers use the ArraySpan overload and pass an offset relative to that span, so adding in.offset there matches the fixed-width specializations and fixes the sliced boolean case. The raw-pointer overload still needs to remain accessible for callers that already supply an absolute offset.

singhpratech added a commit to singhpratech/ArrowMetal that referenced this pull request Sep 17, 2026
singhpratech added a commit to singhpratech/ArrowMetal that referenced this pull request Sep 18, 2026

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation @A-makarim . This looks good to me, I'll wait for additional CI and then we can merge.

Copilot AI review requested due to automatic review settings September 23, 2026 12:48
@pitrou
pitrou force-pushed the GH-51223-fix-sliced-boolean-arrays branch from efb7d1f to 398dcdf Compare September 23, 2026 12:48
@pitrou

pitrou commented Sep 23, 2026

Copy link
Copy Markdown
Member

Rebased.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved issues were identified, and regression coverage is included.

Review effort: Lite
Findings: None

@pitrou
pitrou merged commit 0efbdec into apache:main Sep 23, 2026
60 checks passed
Hor911 added a commit to ydb-campus/antb1 that referenced this pull request Nov 14, 2026
…#368)

## Summary

Fixes a wrong answer on main since ADR 0041's SQ7a (#360), found by
`diff-random`. A select item's CASE over an IN
subquery gave one row the wrong value under OFFSET: over the synthetic
fixtures,
`SELECT ct_id, CASE WHEN ct_id IN (SELECT rd_city FROM riders WHERE
rd_city IS NOT NULL) THEN 1 ELSE 0 END FROM cities
OFFSET 2` gave `4|1` for a city that has no rider.

**Cause: Arrow 25's `replace_with_mask` kernel**, which `Scatter` in
`src/exec/compute.cc` calls.
- It reads a sliced BOOLEAN's values from the first bit of its buffer
and ignores the array's offset. It reads the
validity and the mask correctly. This is upstream
[apache/arrow#51223](apache/arrow#51223),
fixed for Arrow 26 by
[#51240](apache/arrow#51240).
- Without ORDER BY, the CASE is computed above the LIMIT. The LIMIT's
window slices the mark join's batch, so the mark
reaches the CASE with an offset, and the kernel reads values as many
rows early as the offset.

**Fix.** `Scatter` copies a BOOLEAN that has an offset to offset 0 (its
bitmaps only) before calling the kernel, for
both of the kernel's arrays (`UnsliceBoolean`).

**What it broke.** A mark under OFFSET, with or without LIMIT, wherever
a scatter hands it over whole:
- a CASE's WHEN or a later WHEN;
- an OR's or AND's argument after the first, while every row is still
undecided;
- COALESCE over such a CASE;
- the `= ANY` and `<> ALL` forms, and a three-valued mark over a set
that holds a NULL.

These were right before the fix:
- NOT over a mark, and a mark as an OR's first argument;
- marks in WHERE, ORDER BY items, GROUP BY keys and aggregate arguments;
- ORDER BY ... OFFSET, where the CASE is computed below the sort.

Every other Arrow kernel antb1 runs on BOOLEANs was checked at offsets 0
to 19, with and without NULLs, and reads the
offset: and, and_not, and_kleene, or, or_kleene, invert, coalesce,
is_valid, is_null, if_else, equal, filter, take and
cast. Before SQ7a's marks, no SQL could put a BOOLEAN column under a
scatter.

## Type of change

- [ ] feat: new SQL, CLI or engine capability
- [x] fix: bug fix
- [ ] perf: performance improvement
- [ ] refactor, test, docs, build, ci or chore
- [ ] Breaking change (CLI, output format or semantics); also add the
`breaking-change` label

## Verification

```text
$ pixi run test
100% tests passed out of 4027
ANTB1-TESTS: PASS preset=dev

$ pixi run ci
100% tests passed out of 4027   (Clang Debug -Werror, no warning: lines)

$ clang-tidy -p build/tidy --warnings-as-errors='*' src/exec/compute.cc src/exec/tests/compute_test.cc; pixi run tidy
exit 0; exit 0

$ pixi run lint
every hook passes but shellcheck, whose findings are the known macOS-only SC2218 in scripts this PR does not touch

$ ASan build (pixi run asan stops at test discovery on macOS; suites run from build/ci-asan, detect_leaks=0)
antb1_exec_tests 526 passed, antb1_engine_tests 32
antb1-slt run subqueries/*.slt, limit.slt, case.slt, boolean.slt: 2367 records, PASS at 1 and 4 threads

$ pixi run diff-random (default seed), ANTB1_DIFF_SEED=3, ANTB1_DIFF_SEED=6
DIFF: PASS queries=2000 failed=0 unsupported=0 (each)

# The five failing diff-random cases, over the twelve star tables: seed 3 cases 21273 and 22971, seed 4 case
# 27508, seed 6 cases 24720 and 39177
each failed before the fix and passes after it

# Star-table runs of 40,000 queries: seeds 3, 4 and 6; seed 3 again with --batch-size 97 --threads 4
DIFF: PASS failed=0 (each)
```

**Tests added:**
- `exec.ComputeTest.ABooleanOfASlicedBatchIsScatteredFromItsOwnRows`
checks every scatter of a sliced BOOLEAN at
offsets 1 to 9, with and without NULLs, against the same rows unsliced.
It failed before the fix.
-
`exec.ComputeTest.ArrowsReplaceWithMaskReadsASlicedBooleanFromItsBuffersFirstBit`
pins the kernel's misread. It fails
once the Arrow in use is fixed (Arrow 26), and its message says the
workaround and the test can then go.
- 13 records in `tests/slt/cases/subqueries/in_marks.slt`:
- 10 under OFFSET without ORDER BY. They rely on file order, as
`limit/limit.slt` does, and each fails without the
    fix;
  - NOT IN as the contrast;
  - two under ORDER BY ... OFFSET.

## Checklist

- [x] `pixi run check` passes locally: its lint step stops on the
macOS-only shellcheck findings, so `pixi run lint` and
      the tests were run separately (above)
- [x] Tests cover the change (above)
- [x] Docs updated where behavior, commands or architecture changed, or
not needed: not needed, no documented behavior
      changes
- [x] No ClickBench-derived data is committed: no Parquet files, query
answers or values from `hits` (ADR-0006)
- [x] Nothing derived from TPC-H or TPC-DS is committed: no query text
or fragments, data, answers or TPC tools
      (ADR-0006)
- [x] Changes to governance paths (see `.github/CODEOWNERS`) were agreed
with a maintainer: none touched

## AI assistance

- [ ] No AI assistance
- [x] AI-assisted. Tools and what they did: Claude Code (Claude Opus
5.5) found the failures with `diff-random`, traced
them to the Arrow kernel, wrote the fix, the unit tests and the records,
checked the other BOOLEAN kernels, and
      reviewed the diff with a reviewer agent.
- Accountable human (has read and understands the whole diff): @Hor911

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants