test_write_failed_part_while_waiting fails intermittently since queued parts are cancelled #1237

tests/pyathena/filesystem/test_s3_async.py::TestAioS3StreamedFile::test_write_failed_part_while_waiting fails intermittently in CI since #1231 was merged.

Failures

• Run 39014099850, on feat/1227-write-session.
• Run 39014636778, on Read UNLOAD timestamps at microseconds #1233, which does not change the filesystem.

>       assert sorted(c.kwargs["part_number"] for c in fs.core.upload_part.call_args_list) == [
            1,
            2,
        ]
E       assert [2] == [1, 2]

Run alone on master 59169c5, 30 times locally, the test passed every time. Both CI failures ran under pytest-xdist with the whole suite.

Cause

upload_part always raises PermissionError, and the file has max_workers=2. The test expects both parts of the first two blocks to call upload_part.

Since f8ffed1 (#1231, "Cancel queued S3 requests before setting their stop"), a write that finds a failed part calls AioS3StreamedFile._close_without_commit() (pyathena/filesystem/s3_async.py:1603). That method:

1. cancels the creation and the parts that have not started;
2. then sets _parts_stop.

If part 2 starts first and fails before part 1's thread picks it up, part 1 can be cancelled while it is still queued, and then it never calls upload_part. The behavior is the intended one: no request is sent after a sibling failed. Only the test assumes that both parts start.

Expected

The test checks what the contract guarantees, whatever the order in which the two threads start:

• the write raises the part's error;
• the file is closed;
• the upload is aborted and not completed;
• at least one part ran.

Alternatively, the test makes both parts start before either fails, for example with a barrier in the upload_part mock, if it is meant to cover two running parts.