Skip to content

pipe_file() fails on non-contiguous memoryviews and replaces the object with an empty one #997

Closed

Description

@laughingman7743

Problem

S3FileSystem.pipe_file() accepts bytes, bytearray, and memoryview values.
Data within the block size goes to a single PutObject request through bytes(value), which accepts any memoryview.
Larger data goes through fsspec's AbstractFileSystem.pipe_file(), which runs with self.open(path, "wb") as f: f.write(value).
Two defects follow on that path:

  1. A non-contiguous memoryview fails. S3File.write() passes the value to io.BytesIO.write(), which raises BufferError: memoryview: underlying buffer is not C-contiguous. The same value within the block size is uploaded successfully.
  2. The failed write replaces the object with an empty one. When write() raises, the with block closes the file, and close() commits it. Nothing was buffered, so S3File.commit() calls touch(), and a PutObject request with no body overwrites the existing object. The caller sees the original exception after the object has already been replaced.

Expected: pipe_file() uploads any bytes-like value it accepts, whichever path it takes, and a failed write leaves the existing object unchanged.

Reproduction

Run offline: the client is a real botocore S3 client with Stubber activated, so no request leaves the process.
The filesystem is built as in TestS3FileSystem._make_fs() in tests/pyathena/filesystem/test_s3.py, with default_cache_type = "bytes", and its real _call is used.

import boto3
from botocore.stub import Stubber

fs._client = boto3.client("s3", region_name="us-west-2", aws_access_key_id="x", aws_secret_access_key="x")
stubber = Stubber(fs._client)
stubber.activate()
stubber.add_response("put_object", {"ETag": '"e"'})
calls = []
fs._client.meta.events.register(
    "provide-client-params.s3.*",
    lambda params, model, **kw: calls.append((model.name, sorted(params))),
)

value = memoryview(b"a" * (12 * 2**20))[::2]  # 6 MiB, larger than the 5 MiB default block
fs.pipe_file("s3://bucket/key", value)

Output on master 92c9e3e:

BufferError: memoryview: underlying buffer is not C-contiguous
calls == [('PutObject', ['Bucket', 'Key'])]   # the empty object from touch()

Environment

  • PyAthena master 92c9e3e. v3.37.0 has the same pipe_file() dispatch and buffered path (code reading; not run on v3.37.0).
  • AioS3FileSystem._pipe_file() delegates to the sync pipe_file(), and inside an fsspec transaction it writes through the same with self.open(...) pattern in _pipe_file_in_transaction() (code reading).
  • fsspec 2026.9.0.

Proposed fix (optional)

  • Convert a non-contiguous memoryview to bytes before the buffered path, as the PutObject path already does.
  • On the buffered path, write the value without fsspec's with block, and discard the file instead of committing it when the write fails.
  • Add offline tests for a non-contiguous memoryview larger than the block size, and for a failing write that must not send PutObject.

Found while reviewing #968.

Activity

  1. added 2 commits that reference this issue on Oct 3, 2026
    6ed7822
    cbafc9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions