You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
AioS3FileSystem batch operations do not reject conflicting pairs #1398
Since #1344, the batch operations of S3FileSystem raise ValueError before any transfer for pairs whose result would depend on the order of their concurrent transfers:
two pairs of get, put, copy, or pipe(dict) with the same destination after normalization;
a copy destination that is the source of another pair;
get destinations that are the same existing local file (os.path.samefile) or the same file-like object.
The batch coroutines of AioS3FileSystem run their transfers concurrently too (_get, _put, _copy, and _pipe, through fsspec), but check none of these. At d03e034, with stubbed requests:
_pipe({"s3://bucket/k": b"1", "bucket/k": b"2"}) sends two concurrent PutObject requests for k; S3FileSystem.pipe() raises ValueError: Cannot pipe several values to the same destination.
_copy(["s3://bucket/a", "s3://bucket/b"], ["s3://bucket/b", "s3://bucket/c"]) copies a to b and b to c at the same time, so c gets the old or the new contents of b.
_put([l1, l2], ["s3://bucket/p", "bucket/p"]) uploads both files to p.
_mv() already checks its pairs (S3FileSystem._move_pairs()).
Direction
Apply the checks of S3FileSystem (the duplicate destinations, S3PathPairing.check_copy_pairs() with the lookups of _conflict_lookups(), and _check_local_destinations()) in the overrides of AioS3FileSystem before fsspec runs the transfers. This changes behavior for callers that pass such pairs, as it did for S3FileSystem in #1344, and needs a release note.
Context
Left out of #1395 (step 4b of #1332), which aligned the cancellation and the arguments of AioS3FileSystem with S3FileSystem.
Activity