Recursive copy() and mv() send CopyObject for directories, and mv() can delete overlapping copies #1008

Problem

Recursive copy() and mv() pass directory entries to cp_file(), and the recursive mv() that fsspec provides can delete data. The two problems are linked: fixing the first one exposes more of the second.

1. cp_file() sends CopyObject for directory entries

fsspec's recursive copy() passes the source directory and its subdirectories to cp_file(). S3FileSystem.cp_file() and AioS3FileSystem._cp_file() send CopyObject for them, which fails with NoSuchKey because no such object exists.

• copy(recursive=True) still succeeds, because fsspec ignores FileNotFoundError in recursive copies. Every directory entry still costs a failed CopyObject.
• mv(recursive=True) calls copy(..., on_error="raise"), so it raises FileNotFoundError.
  • S3FileSystem copies and deletes nothing.
  • AioS3FileSystem runs the copies concurrently, so the files are copied but the source is not deleted.

This was item 2 of #974; item 1 (get_file()) is fixed separately.

2. fsspec's mv() removes what it copied when the destination overlaps the source

AbstractFileSystem.mv() (fsspec 2026.9.0, spec.py:1292-1301) runs copy(path1, path2, recursive=recursive, on_error="raise") and then rm(path1, recursive=recursive). The removal expands path1 again after the copy. If the copies landed where path1 matches them, they are removed too.

Measured on master 28826be against real S3:

Today, problem 1 stops most directory moves before the removal. Making cp_file() skip directory entries would let these moves reach the removal:

• mv("p/src", "p/src/archive", recursive=True) would remove both trees.
• mv("p/src/sub", "p/src", recursive=True) would remove src/sub/a, which was reported with live S3 in the PR Follow fsspec's get_file() contract for directories and file objects #990 review.

The PR #990 review also found other cases that need a decision. The table above does not measure them:

• maxdepth: the removal deletes files below the depth that were never copied.
• List sources: fsspec flattens a listed directory onto base names, so copies can overwrite each other.
• Duplicate destinations: mv("p/src/*.txt", ["p/dst", "p/dst"], recursive=True).
• Keys ending in /: info() reports such a key as a directory, so skipping directory entries would drop a real object, possibly one larger than a single CopyObject allows.

Expected

• copy(recursive=True) and mv(recursive=True) move a directory tree without a failed CopyObject per directory.
• mv() never deletes data that it has not copied to a separate location. Either it rejects such moves before copying anything, or it moves them safely.

Decision needed

How much of fsspec's copy-then-remove mv() PyAthena should guard. Options:

• Narrow guard: reject a destination that is, or lies under, a path the source matches.
• Overlap guard: reject any destination that overlaps the source in either direction, compared by path segment, plus maxdepth and list sources. PR Follow fsspec's get_file() contract for directories and file objects #990 tried this before it was reduced to get_file(); the attempt is at c6782a5. It also rejects some moves that work on master, such as files-only lists.
• Own implementation: have mv() remove exactly the keys it copied, instead of expanding path1 again.

Environment

• PyAthena master 28826be, fsspec 2026.9.0, Python 3.13.1.
• The table was measured with live S3 in the CI test bucket. The other cases come from mocked responses in the PR Follow fsspec's get_file() contract for directories and file objects #990 review.