Skip to content

Free failed part bodies without the GC and abort an async copy's upload created during loop shutdown #1186

Closed

Description

@laughingman7743

Problem

Two cleanup gaps of the S3 filesystem's executors, left out of #1159 (PR #1184).

1. The body of a failed part stays allocated until the garbage collector runs

This is option 2 of #1173.
A future that fails through S3Executor.submit_to() is in a reference cycle:

  • the run() closure holds the future;
  • the future holds the exception;
  • the exception's traceback holds the frame of run(), and the frames below it hold the part's body.

The body of a failed part is therefore freed by the garbage collector, not when the error is raised or the future is dropped.
Measured on 3a50182 with the reproduction from #1173 (max_workers=4, 8 MiB parts, every part failing): about 32 MiB stay allocated before gc.collect() and 0.2 MiB after.

#1176 bounds this to at most max_workers failed parts, because the write stops at its wait.
After #1184, S3AioThreadPool inherits submit_to(), so the parts of AioS3StreamedFile (open_async()) are affected the same way.

2. An AioS3FileSystem multipart copy leaves its upload when the event loop shuts down during the creation

AioS3FileSystem._copy_object_with_multipart_upload() runs CreateMultipartUpload as a task, and its _abort() cleanup treats a cancelled creation task as proof that no upload exists:

await asyncio.wait([creation])
if creation.cancelled() or creation.exception() is not None:
    # No upload was created.
    return

asyncio.run() cancels every remaining task when it shuts down, including the creation task. The blocking request in its thread can still succeed, but the cleanup returns without the upload ID and does not abort the upload. The incomplete multipart upload stays in the bucket until a lifecycle rule removes it.

This is one of the event-loop shutdown edges deferred in #1046 (PR #1056). It was found again by the independent review of #1184, and it also exists on master.

Reproduction

  1. Run the reproduction of Failed parts of an S3File multipart write keep their data and do not stop the write #1173, then compare the memory allocated before and after gc.collect().
  2. Run a copy of an object over 5 GiB with asyncio.run(), and stop the run during CreateMultipartUpload, for example with a fake core whose creation blocks until the loop shuts down. list_multipart_uploads() then still lists the upload.

Environment

master after #1184; PyAthena 4.1.0+, Python 3.11–3.15.

Proposed fix (optional)

  1. In S3Executor.submit_to(), drop the references of run() to the callable and its arguments, and clear the traceback frames below run(), before set_exception(). Note that traceback.clear_frames() cannot clear the running frame of run() itself. Test it with the garbage collector disabled, and check which traceback locals are lost.
  2. Keep the outcome of the creation request independently of the task, for example as the concurrent.futures.Future of its thread, so that _abort() can wait for a creation that its task no longer reports, recover the upload ID, and abort it.

Activity

  1. added 2 commits that reference this issue on Oct 16, 2026
    fd8b7b3
    4ac521d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions