Skip to content

RobustUnpacker: release all memoryviews of the resync buffer before resizing it - #10517

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:robustunpacker-pypy-bufferror
Oct 11, 2026
Merged

ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:robustunpacker-pypy-bufferror

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Fixes the "Tests (pypy-3.11-nightly, ubuntu-26.04)" CI job, which fails since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9): 12 tests in archive_test.py (test_missing_chunk, test_corrupt_chunk, test_resync_*) die with

BufferError: Existing exports of data: object cannot be re-sized

at del self._buffered_data[:drop_len] in RobustUnpacker._search().

Cause

_search() scans the buffer inside with memoryview(self._buffered_data) as data: and called valid_msgpacked_dict(data[offset:]) for every offset; that function sliced the key out of the view as well. CPython frees these unreferenced child views at once, so the bytearray has no exports left when the with block releases the parent view. PyPy has no refcounting, the child views live until the GC runs. PyPy 8.0.0 did not count child views as exports of the bytearray, the new nightly does (as CPython always did), so the resize after the with block fails.

Upstream background

The nightly behaviour is a deliberate PyPy fix, not a regression: PyPy 8.0.0 dropped the bytearray's export when the parent view was released even while child slices were still alive, which could segfault (pypy/pypy#5613, fixed by pypy/pypy@49f894f0f541 "keep a memoryview's export alive while its slices and casts use it", followed by pypy/pypy@f61a05f029e5 "refactor buffer exports to use a BufferExport class"). CPython 3.11 raises the same BufferError for this pattern if the child view is still referenced. PyPy's docs (pypy/pypy@225bffc76d6d, for pypy/pypy#5612) now say that views are kept alive until the GC frees them, so views must be released explicitly (view.release() or with memoryview(buf) as view:) before the viewed object is resized. For _search() that means the child slices data[offset:] need releasing too, or must not be created at all; the with on the parent view alone is not enough.

Fix

valid_msgpacked_dict(d, offset=0) only indexes d and never slices it, and _search() passes (data, offset) instead of a slice. With the already explicitly released slice in _try_item_start(), no unreleased view of _buffered_data exists when it is resized. The algorithm and its linear-time behaviour (#10459) are unchanged.

Verification

  • PyPy 8.0.2-alpha0 nightly (544bd2f768f9, macOS arm64): archive_test.py 12 failed / 80 passed before, 93 passed after.
  • PyPy 8.0.0 release: 93 passed.
  • CPython 3.11: archive_test.py 93 passed, archiver/check_cmd_test.py 120 passed (binary variants skipped).
  • CPython scan of 2 MiB garbage after resync(): 0.179 s before, 0.148 s after (no slice object per offset).

🤖 Generated with Claude Code

…esizing it

Since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9), 12 tests
in archive_test.py (test_missing_chunk, test_corrupt_chunk, test_resync_*) fail with:

    BufferError: Existing exports of data: object cannot be re-sized

at `del self._buffered_data[:drop_len]` in RobustUnpacker._search().

_search() scanned the buffer inside `with memoryview(self._buffered_data) as data:`
and called valid_msgpacked_dict(data[offset:]) for every offset, which also sliced the
key out of that view. CPython frees these unreferenced child views immediately, so the
bytearray is exportless again as soon as the with block releases the parent view.
PyPy has no refcounting: the child views stay alive until the GC runs. PyPy 8.0.0 did
not count child views as exports of the bytearray, the new nightly does (like CPython
always did), so the resize after the with block fails.

Fix: valid_msgpacked_dict(d, offset=0) only indexes d, it never slices it, and _search()
passes (data, offset) instead of a slice. Together with the explicitly released slice in
_try_item_start(), no unreleased view of _buffered_data exists anymore when it is
resized. The scan loop also gets a bit faster on CPython (no slice object per offset).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.29%. Comparing base (dc473d2) to head (4da7021).
⚠️ Report is 25 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10517      +/-   ##
==========================================
+ Coverage   89.24%   89.29%   +0.04%     
==========================================
  Files         104      104              
  Lines       19927    19941      +14     
  Branches     3124     3127       +3     
==========================================
+ Hits        17784    17806      +22     
+ Misses       1483     1478       -5     
+ Partials      660      657       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann
ThomasWaldmann merged commit 15738b8 into borgbackup:master Oct 11, 2026
45 of 46 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the robustunpacker-pypy-bufferror branch October 11, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant