You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Move S3File writes onto a multipart write session #1227
Sub-issue of #1211, which records the design decisions.
Use case
S3File (also used by AioS3File) tracks its own part futures, in-flight sets, failure checks, and abort. Its lifecycle state is implicit, which lets discard() followed by commit() upload the buffered last part alone, lets a commit() after an abort failure complete the discarded upload, and keeps failed part futures in reference cycles.
Proposed change
Add the internal _S3WriteSession and S3MultipartStateError to pyathena/filesystem/s3_writer.py as specified in #1211, and move S3File onto it:
_initiate_upload, _upload_chunk, commit, discard, and _close_without_commit delegate creation, part uploads and copies, bounded concurrency, the single-request path, completion, and abort to the session.
multipart_upload becomes read-only; multipart_upload_parts is replaced by a snapshot of the completed parts.
commit() after a discard or a failure raises S3MultipartStateError; a write after a failure raises before buffering.
Behavior changes are listed in the PR for the 4.2.0 release notes.
Validation plan (if implementing)
Unit tests of the session without AWS: state transitions, admission and readiness, waiter cancellation, scheduling failures before and after a request starts, late callbacks during an abort, two abort callers, queued versus running completion, abort failure and retry, interrupts, and one-worker pools.
GC-disabled weak-reference tests that the session releases failed request futures and part bodies after the error is consumed.
The existing S3File and AioS3File tests, rewritten where they injected multipart_upload_parts or set multipart_upload, with both S3ThreadPoolExecutor and S3AioExecutor.
The filesystem integration tests against the CI bucket (append, exclusive create, transactions).
Activity