Skip to content

fix(ipc): Zero the padding of footer Blocks - #986

Merged
paleolimbot merged 3 commits into
apache:mainfrom
pedrobtz:fix-ipc-footer-block-padding
Nov 11, 2026
Merged

paleolimbot merged 3 commits into
apache:mainfrom
pedrobtz:fix-ipc-footer-block-padding

Conversation

@pedrobtz

Copy link
Copy Markdown
Contributor

ArrowIpcEncoderEncodeFooter() builds each Block of the footer on the stack with an initializer, then copies it into the flatbuffer vector. Block has four bytes of padding after metaDataLength, and an initializer doesn't set padding. So the footer carried whatever bytes were on the stack. That makes files non-reproducible, and the stack bytes end up in the file. With Apple clang 17 at -O0, filling the stack with 0xAB before the call gives padding abababab.

The vector is now zeroed after extend, and each element is set with the generated Block_assign(). Both the record batch and the dictionary block loops are changed.

The new NanoarrowIpcFooterEncodingIsDeterministic test encodes the same footer twice, after filling the stack with 0x00 and then 0xFF, and requires identical bytes. In a plain Debug build it fails without the fix and passes with it. Under ASan it passes either way, because ASan moves stack frames.

Found through a byte-for-byte determinism test in the zuarrow R package, which vendors nanoarrow.

ctest passes in a plain Debug build and with ASan and UBSan (410 tests each). The changed files are formatted with the pinned clang-format 16.0.6.

🤖 Generated with Claude Code

Each Block was built on the stack with an initializer and copied into the
footer, so its four bytes of padding after metaDataLength were whatever
was on the stack. Zero the vector and set each field instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.54%. Comparing base (08d23f6) to head (43d9a98).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #986      +/-   ##
==========================================
+ Coverage   78.52%   78.54%   +0.02%     
==========================================
  Files         106      106              
  Lines       17195    17293      +98     
  Branches     2064     2066       +2     
==========================================
+ Hits        13502    13583      +81     
- Misses       2431     2446      +15     
- Partials     1262     1264       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

The test I think is out of scope, although I do appreciate the effort. With the test removed I think the fix is great and we can merge 🚀

Comment thread src/nanoarrow/ipc/encoder_test.cc Outdated
Comment on lines +111 to +115
// Fill a stretch of the stack so that uninitialized stack bytes used by a
// subsequent call are not zero by chance
#if defined(_MSC_VER)
__declspec(noinline)
#else

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate the thoroughness here, but this test still passes in most cases even without the fix. I think this PR is stronger without the test.

The test can pass without the fix, so it does not guard against
regressions (per review).
@pedrobtz

Copy link
Copy Markdown
Contributor Author

Thanks! I've dropped the test, so the PR is now just the fix in encoder.c.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit ee22361 into apache:main Nov 11, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants