You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Free failed part bodies without the GC and abort an async copy's upload created during loop shutdown #1186
Two cleanup gaps of the S3 filesystem's executors, left out of #1159 (PR #1184).
1. The body of a failed part stays allocated until the garbage collector runs
This is option 2 of #1173.
A future that fails through S3Executor.submit_to() is in a reference cycle:
the run() closure holds the future;
the future holds the exception;
the exception's traceback holds the frame of run(), and the frames below it hold the part's body.
The body of a failed part is therefore freed by the garbage collector, not when the error is raised or the future is dropped.
Measured on 3a50182 with the reproduction from #1173 (max_workers=4, 8 MiB parts, every part failing): about 32 MiB stay allocated before gc.collect() and 0.2 MiB after.
#1176 bounds this to at most max_workers failed parts, because the write stops at its wait.
After #1184, S3AioThreadPool inherits submit_to(), so the parts of AioS3StreamedFile (open_async()) are affected the same way.
2. An AioS3FileSystem multipart copy leaves its upload when the event loop shuts down during the creation
AioS3FileSystem._copy_object_with_multipart_upload() runs CreateMultipartUpload as a task, and its _abort() cleanup treats a cancelled creation task as proof that no upload exists:
awaitasyncio.wait([creation])
ifcreation.cancelled() orcreation.exception() isnotNone:
# No upload was created.return
asyncio.run() cancels every remaining task when it shuts down, including the creation task. The blocking request in its thread can still succeed, but the cleanup returns without the upload ID and does not abort the upload. The incomplete multipart upload stays in the bucket until a lifecycle rule removes it.
This is one of the event-loop shutdown edges deferred in #1046 (PR #1056). It was found again by the independent review of #1184, and it also exists on master.
Run a copy of an object over 5 GiB with asyncio.run(), and stop the run during CreateMultipartUpload, for example with a fake core whose creation blocks until the loop shuts down. list_multipart_uploads() then still lists the upload.
Environment
master after #1184; PyAthena 4.1.0+, Python 3.11–3.15.
Proposed fix (optional)
In S3Executor.submit_to(), drop the references of run() to the callable and its arguments, and clear the traceback frames below run(), before set_exception(). Note that traceback.clear_frames() cannot clear the running frame of run() itself. Test it with the garbage collector disabled, and check which traceback locals are lost.
Keep the outcome of the creation request independently of the task, for example as the concurrent.futures.Future of its thread, so that _abort() can wait for a creation that its task no longer reports, recover the upload ID, and abort it.
Activity
Free failed parts without the GC and abort copies at loop shutdown
Free failed parts without the GC and abort copies at loop shutdown