Repository navigation
RobustUnpacker: release all memoryviews of the resync buffer before resizing it - #10517
Merged
ThomasWaldmann merged 1 commit intoOct 11, 2026
Conversation
…esizing it
Since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9), 12 tests
in archive_test.py (test_missing_chunk, test_corrupt_chunk, test_resync_*) fail with:
BufferError: Existing exports of data: object cannot be re-sized
at `del self._buffered_data[:drop_len]` in RobustUnpacker._search().
_search() scanned the buffer inside `with memoryview(self._buffered_data) as data:`
and called valid_msgpacked_dict(data[offset:]) for every offset, which also sliced the
key out of that view. CPython frees these unreferenced child views immediately, so the
bytearray is exportless again as soon as the with block releases the parent view.
PyPy has no refcounting: the child views stay alive until the GC runs. PyPy 8.0.0 did
not count child views as exports of the bytearray, the new nightly does (like CPython
always did), so the resize after the with block fails.
Fix: valid_msgpacked_dict(d, offset=0) only indexes d, it never slices it, and _search()
passes (data, offset) instead of a slice. Together with the explicitly released slice in
_try_item_start(), no unreleased view of _buffered_data exists anymore when it is
resized. The scan loop also gets a bit faster on CPython (no slice object per offset).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10517 +/- ##
==========================================
+ Coverage 89.24% 89.29% +0.04%
==========================================
Files 104 104
Lines 19927 19941 +14
Branches 3124 3127 +3
==========================================
+ Hits 17784 17806 +22
+ Misses 1483 1478 -5
+ Partials 660 657 -3 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the "Tests (pypy-3.11-nightly, ubuntu-26.04)" CI job, which fails since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9): 12 tests in
archive_test.py(test_missing_chunk,test_corrupt_chunk,test_resync_*) die withat
del self._buffered_data[:drop_len]inRobustUnpacker._search().Cause
_search()scans the buffer insidewith memoryview(self._buffered_data) as data:and calledvalid_msgpacked_dict(data[offset:])for every offset; that function sliced the key out of the view as well. CPython frees these unreferenced child views at once, so the bytearray has no exports left when thewithblock releases the parent view. PyPy has no refcounting, the child views live until the GC runs. PyPy 8.0.0 did not count child views as exports of the bytearray, the new nightly does (as CPython always did), so the resize after thewithblock fails.Upstream background
The nightly behaviour is a deliberate PyPy fix, not a regression: PyPy 8.0.0 dropped the bytearray's export when the parent view was released even while child slices were still alive, which could segfault (pypy/pypy#5613, fixed by pypy/pypy@49f894f0f541 "keep a memoryview's export alive while its slices and casts use it", followed by pypy/pypy@f61a05f029e5 "refactor buffer exports to use a BufferExport class"). CPython 3.11 raises the same
BufferErrorfor this pattern if the child view is still referenced. PyPy's docs (pypy/pypy@225bffc76d6d, for pypy/pypy#5612) now say that views are kept alive until the GC frees them, so views must be released explicitly (view.release()orwith memoryview(buf) as view:) before the viewed object is resized. For_search()that means the child slicesdata[offset:]need releasing too, or must not be created at all; thewithon the parent view alone is not enough.Fix
valid_msgpacked_dict(d, offset=0)only indexesdand never slices it, and_search()passes(data, offset)instead of a slice. With the already explicitly released slice in_try_item_start(), no unreleased view of_buffered_dataexists when it is resized. The algorithm and its linear-time behaviour (#10459) are unchanged.Verification
archive_test.py12 failed / 80 passed before, 93 passed after.archive_test.py93 passed,archiver/check_cmd_test.py120 passed (binary variants skipped).resync(): 0.179 s before, 0.148 s after (no slice object per offset).🤖 Generated with Claude Code