You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
pipe_file() fails on non-contiguous memoryviews and replaces the object with an empty one #997
S3FileSystem.pipe_file() accepts bytes, bytearray, and memoryview values.
Data within the block size goes to a single PutObject request through bytes(value), which accepts any memoryview.
Larger data goes through fsspec's AbstractFileSystem.pipe_file(), which runs with self.open(path, "wb") as f: f.write(value).
Two defects follow on that path:
A non-contiguous memoryview fails.S3File.write() passes the value to io.BytesIO.write(), which raises BufferError: memoryview: underlying buffer is not C-contiguous. The same value within the block size is uploaded successfully.
The failed write replaces the object with an empty one. When write() raises, the with block closes the file, and close() commits it. Nothing was buffered, so S3File.commit() calls touch(), and a PutObject request with no body overwrites the existing object. The caller sees the original exception after the object has already been replaced.
Expected: pipe_file() uploads any bytes-like value it accepts, whichever path it takes, and a failed write leaves the existing object unchanged.
Reproduction
Run offline: the client is a real botocore S3 client with Stubber activated, so no request leaves the process.
The filesystem is built as in TestS3FileSystem._make_fs() in tests/pyathena/filesystem/test_s3.py, with default_cache_type = "bytes", and its real _call is used.
BufferError: memoryview: underlying buffer is not C-contiguous
calls == [('PutObject', ['Bucket', 'Key'])] # the empty object from touch()
Environment
PyAthena master 92c9e3e. v3.37.0 has the same pipe_file() dispatch and buffered path (code reading; not run on v3.37.0).
AioS3FileSystem._pipe_file() delegates to the sync pipe_file(), and inside an fsspec transaction it writes through the same with self.open(...) pattern in _pipe_file_in_transaction() (code reading).
fsspec 2026.9.0.
Proposed fix (optional)
Convert a non-contiguous memoryview to bytes before the buffered path, as the PutObject path already does.
On the buffered path, write the value without fsspec's with block, and discard the file instead of committing it when the write fails.
Add offline tests for a non-contiguous memoryview larger than the block size, and for a failing write that must not send PutObject.
Activity
Write pipe_file() data without committing a failed write
Write pipe_file() data without committing a failed write