Repository navigation
Move the AioS3FileSystem multipart copy onto the multipart write session #1230
Description
Sub-issue of #1211, which records the design decisions. Depends on the session added for S3File.
Use case
AioS3FileSystem._copy_object_with_multipart_upload bounds its part copies with a semaphore and a failed flag, and aborts in a filesystem thread that waits for the creation, the copies, and the completion.
Proposed change
Move the asynchronous multipart copy onto _S3WriteSession:
- The session's admission replaces the semaphore and the
failedflag, keeping the per-copymax_workerslimit and the filesystem's shared thread pool. - A completion that has not started can still be cancelled, as today.
- The prepared parameters and annotations are handled as in the synchronous copy.
Behavior changes are listed in the PR for the 4.2.0 release notes.
Validation plan (if implementing)
- The existing asynchronous copy tests, including cancellation, loop shutdown, concurrent copies sharing one pool, and a queued completion.
- The filesystem integration tests against the CI bucket.
Activity
laughingman7743 commentedon Oct 18, 2026
From #1229 (PR #1269): the synchronous multipart copy no longer calls
S3FileSystem._abort_multipart_upload(), but it is kept because the asynchronous copy still calls it (AioS3FileSystem._copy_object_with_multipart_upload,pyathena/filesystem/s3_async.py, the_abort()that runs in a filesystem thread). When this issue moves the asynchronous copy onto_S3WriteSession, remove_abort_multipart_upload()together with its tests if nothing else uses it.Move the S3FileSystem multipart copy onto the write session (GH-1229)
laughingman7743 commentedon Oct 18, 2026
Added to the scope of this issue, from the review of #1272 (#1228):
AioS3StreamedFile._commit()(pyathena/filesystem/s3_async.py) reserves the commit with_S3WriteSession.begin_complete()and then queues_complete_uploadwithrun_shielded(). AKeyboardInterruptthat arrives between the two leaves a reservation that nothing runs: the session staysCOMPLETING,abort()leaves the write to the commit, and the upload is neither completed nor aborted. The_committingflag before #1272 had the same gap.The asynchronous multipart copy moved by this issue reserves its completion the same way ("a completion that has not started can still be cancelled"), so both are fixed here with one mechanism: a reservation whose finalizer was never queued is released, and the write is aborted as best effort. This keeps the interrupt policy of #1211: an interrupt during the later cleanup escapes and keeps the upload for a retry.
Describe every commit that withdraw_complete() withdraws (GH-1230)
Abort a multipart upload only if the interrupted file has one (GH-1230)
Say that an interrupted file aborts its multipart upload, if any (GH-…
Complete the async copy once its parts have settled (GH-1230)
Synchronize the async copy tests with the step they check (GH-1230)
Bound the wait for the cancelled copy in the queued-parts test (GH-1230)
Make the async filesystem tests fail instead of hanging on a blocked …
Release blocked copy cleanups within the stub and without holding fut…