Repository navigation
Move the S3FileSystem multipart copy onto the multipart write session #1229
Description
Sub-issue of #1211, which records the design decisions. Depends on the session added for S3File.
Use case
S3FileSystem._copy_object_with_multipart_upload tracks its creation and part copies itself, with _finish_multipart_upload and _abort_multipart_upload, and submits every part copy at once to its executor.
Proposed change
Move the synchronous multipart copy onto _S3WriteSession:
- Part copies are admitted against
max_workersinstead of all being queued at once. - The prepared parameters of
plan_multipart_copy()keep reaching the requests unfiltered, so that botocore still rejects an unknown parameter. - Annotations stay outside the session and use the version and ETag of the completion.
_finish_multipart_uploadand_abort_multipart_uploadare removed if nothing else uses them.
Behavior changes are listed in the PR for the 4.2.0 release notes.
Validation plan (if implementing)
- The existing multipart copy tests, including interrupts during creation and abort failures.
- The filesystem integration tests against the CI bucket.
Activity
laughingman7743 commentedon Oct 17, 2026
From #1254 (#1240):
S3FileSystem._finish_multipart_upload(..., abort=True)has no caller (_copy_object_with_multipart_uploadpassesabort=Falseand runs its own cleanup), and after #1254 no test exercises that branch. When this issue removes the method, nothing else needs it; until then theabortparameter and its branch are untested.Wait for the session's cleanup wait in the copy scheduling test (GH-1229
Move the S3FileSystem multipart copy onto the write session (GH-1229)
Wait for the session's cleanup wait in the copy scheduling test (GH-1229