Repository navigation
Unclosed multipart S3File collected in a request-pool thread can block its cleanup #1364
Description
Problem
An S3File opened for writing that is never closed is finalized by the garbage collector, which runs its cleanup in whatever thread triggers the collection. Since #1359, the parts of a multipart upload run in the request pool of the filesystem. Two cases can block that cleanup:
- The only thread of the pool. The collection runs in the only thread of the request pool (
max_workers=1), while a part of the file is queued behind it. The cleanup waits for the queued part, which needs that thread. - Inside its own running part. A cyclic, unclosed file is collected inside one of its own running part requests. Its finalizer waits for that request, which cannot finish until the finalizer returns. Cancelling the queued parts does not help here.
Both break T1 of the specification: no call in the request pool waits for a future of the request pool.
Open questions
- Whether a collected unclosed file should commit, as it does now when the cleanup can run, or discard its upload. Cancelling the queued parts turns a commit into a discard.
- Where the cleanup runs: inline in the finalizer thread for the parts that have not started (as
_S3PoolScope._nextdoes when the pool rejects a call), in an owned context outside the request pool, or deferred while a request of the file is running.
Context
Deferred from #1359 and from the step 3 design of #1332, where the first proposal (cancel the queued parts in the finalizer) was found insufficient by review.
Activity
laughingman7743 commentedon Nov 12, 2026
Design (decided)
Both cases reproduce with a stubbed client: the finalizer waits forever in
commit()→_S3WriteSession.complete()→_complete_parts().Decisions on the open questions
Commit or discard. A collected unclosed file keeps today's behavior: it is closed as
close()does, committing it unlessautocommit=False. The one exception is a file whose write buffer the same collection has already finalized. Its data is incomplete, so it is discarded (its multipart upload is aborted), with a logged warning. Discarding every collected file, asAioS3StreamedFiledoes, would be a broader behavior change and is left out of this issue.Where the cleanup runs. In a new thread, with an empty context, whenever waiting in the finalizer is unsafe:
gc.callbacks), since the collecting thread may hold a lock of the pool, a scope, or a session, or be the request that the file waits for;S3ThreadPool, also when the last reference is dropped there.A file whose last reference is dropped in a user thread outside a collection is still closed inline, so
fs.open(path, "wb").write(data)still commits before the statement ends.Details
IOBasefinalization closes the originalBytesIO. This was verified on CPython 3.10, 3.13, and 3.14: a thread started from__del__finds the buffer closed. If the buffer is already closed when the finalizer runs, the file is discarded as above.close()called during a collection by another finalizer, such as that of aTextIOWrapperin text mode, is handed over in the same way, and only once.threading._shutdownruns and is joined before the process exits (verified on 3.10–3.14). AfterThreadPoolExecutorstops accepting calls, a final part cannot be submitted, so such a file is discarded instead of committed.AioS3File, which inheritsS3File, gets the same behavior.AioS3StreamedFilekeeps its own finalizer.Limits (documented)
executor=whose threads are notS3ThreadPoolthreads: a last reference dropped in one of them is closed inline.Review
Codex (static, against 68fc3be) found the buffer finalization, the lock taken by the pending-call check, collections in the cleanup thread, text wrappers, and custom executors. All are addressed or recorded above. The first idea, cancelling the queued parts in the finalizer, was rejected in #1332 because it does not cover the second case.
Close an unclosed S3File outside the pools when it is finalized (GH-1364
Close a finalized S3File inline during interpreter finalization (GH-1364
Document how a file is closed at interpreter finalization (GH-1364)
laughingman7743 commentedon Nov 12, 2026
Addendum: interpreter finalization
The decisions above are unchanged except for the bullet on files for which no thread can be started, which #1396 replaces:
Thread.start()waits forever during finalization; on 3.12 and later, it raises. No other thread runs requests then, and the request pools take no more calls, so the close does not wait for a part queued behind it.ThreadPoolExecutor, which named the wait atthreading._shutdownrather than finalization.Checked with exit scripts and a stubbed client on CPython 3.11, 3.14, and 3.15:
threading._shutdownis committed by the new thread, which runs and is joined.ValueErrorfrom its closed buffer instead, and its multipart upload was left incomplete.Start a handed-over close once the collection has ended (GH-1364)
Close handed-over files in _thread threads and never wait where unsafe (
Run a handed-over close that could not start on the next call (GH-1364)
Wake waiters on a failed start and close after the exit wait inline (G…
Check close() apart from commit() and document the wrapper write wait (…
laughingman7743 commentedon Nov 12, 2026
Addendum 2: where and when a handed-over close runs, and later calls
The independent reviews of #1396 led to these changes to the design above:
gcstop callback. The collector may run the finalizer of the file before that of itsTextIOWrapper, and a close started from the finalizer of the file raced with the wrapper writing its remaining text._threadmodule rather than athreading.Thread.Thread.start()takes locks ofthreadingthat the finalizing or collecting thread may hold (_active_limbo_lockon every version, held bythreading.enumerate()while it allocates;_shutdown_locks_lockon 3.11 and 3.12, taken by the new thread beforeThread.start()returns), so a start from a finalizer could wait forever. The interpreter does not join such threads, so anatexithandler waits for the handed-over closes; a fork clears them in the child, and a close queued in the parent is not run in a child.close(),commit(), ordiscard()of a file whose close was handed over waits for that close to end. Without the wait, a transaction could complete the upload without the data that the handed-over close uploads, and an explicit close could shut down the executor of the file during the handed-over close. During a collection or in a thread of anS3ThreadPool, where the wait could last forever (the end of the collection starts the close; the close may submit a part behind that pool thread),close()returns without waiting, andcommit()anddiscard()raiseRuntimeError.close(),commit(), ordiscard()of the file outside a collection and the pool threads, also one that already waits, rather than marked as ended, so that a transaction does not commit the file without the data that the close uploads; an exit does not wait for it.As a result,
close()called during a collection, such as by a finalizer that writes a file in awithblock, returns before the file is closed.Limits, in addition to those above:
S3ThreadPooloutside a collection closes its file in that thread, as before. No PyAthena pool call holds a text wrapper.S3ThreadPoolcan wait forever for a part queued behind that thread, before the close is reached, as before; it is now documented.threadsafety2).The exit results of the first addendum were measured at e5911ed and depended on the order in which the collector finalizes the file and its buffer. At aac7cf8, exit scripts on CPython 3.11, 3.12, 3.13, 3.14, and 3.15 commit a file dropped by a pool thread during
threading._shutdownand files collected at module teardown with no part left to upload, and abort the upload of a file collected at teardown with a part left to upload; on 3.11 and 3.15, a file collected by an exit handler registered before PyAthena is committed.Close an unclosed S3File outside the pools when it is finalized (GH-1364
Close a finalized S3File inline during interpreter finalization (GH-1364
Document how a file is closed at interpreter finalization (GH-1364)
Start a handed-over close once the collection has ended (GH-1364)
Close handed-over files in _thread threads and never wait where unsafe (
Run a handed-over close that could not start on the next call (GH-1364)
Wake waiters on a failed start and close after the exit wait inline (G…
Check close() apart from commit() and document the wrapper write wait (…