Skip to content

AioS3FileSystem batch operations do not reject conflicting pairs #1398

Open

Description

@laughingman7743

Problem

Since #1344, the batch operations of S3FileSystem raise ValueError before any transfer for pairs whose result would depend on the order of their concurrent transfers:

  • two pairs of get, put, copy, or pipe(dict) with the same destination after normalization;
  • a copy destination that is the source of another pair;
  • get destinations that are the same existing local file (os.path.samefile) or the same file-like object.

The batch coroutines of AioS3FileSystem run their transfers concurrently too (_get, _put, _copy, and _pipe, through fsspec), but check none of these. At d03e034, with stubbed requests:

  • _pipe({"s3://bucket/k": b"1", "bucket/k": b"2"}) sends two concurrent PutObject requests for k; S3FileSystem.pipe() raises ValueError: Cannot pipe several values to the same destination.
  • _copy(["s3://bucket/a", "s3://bucket/b"], ["s3://bucket/b", "s3://bucket/c"]) copies a to b and b to c at the same time, so c gets the old or the new contents of b.
  • _put([l1, l2], ["s3://bucket/p", "bucket/p"]) uploads both files to p.

_mv() already checks its pairs (S3FileSystem._move_pairs()).

Direction

Apply the checks of S3FileSystem (the duplicate destinations, S3PathPairing.check_copy_pairs() with the lookups of _conflict_lookups(), and _check_local_destinations()) in the overrides of AioS3FileSystem before fsspec runs the transfers. This changes behavior for callers that pass such pairs, as it did for S3FileSystem in #1344, and needs a release note.

Context

Left out of #1395 (step 4b of #1332), which aligned the cancellation and the arguments of AioS3FileSystem with S3FileSystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions