ls() returns [] for a missing key prefix instead of raising FileNotFoundError #1330

Part of #1326. Found by fsspec's conformance tests (#1329).

Summary

S3FileSystem.ls() and AioS3FileSystem._ls() return [] for a key prefix that matches no object in an existing bucket. fsspec's filesystems raise FileNotFoundError for a missing path, and so does s3fs. Four of fsspec's conformance tests fail on this alone: test_copy_directory_to_new_directory and test_put_directory_to_new_directory, for both filesystems.

Behavior

Measured on S3 with s3fs 2026.9.0, in an existing bucket:

With versions=True, _ls_object_versions() also returns [] for a missing path, where s3fs raises.

Cause

ls() (pyathena/filesystem/s3.py) lists the prefix with _ls_dirs(), then looks up the path as an object with _head_object() when the listing is empty, and returns the empty list when neither finds it. AioS3FileSystem._ls() runs the same method. This has been the behavior since the filesystem was added (207c8f02).

Compatibility

A caller that lists a path which may not exist and expects [] will get FileNotFoundError, so the change needs a release note. Through the methods that both filesystems inherit from fsspec, it also changes:

• walk() of a missing path, which yields (path, [], []) now and would yield nothing, as fsspec's walk omits a path whose ls raises with the default on_error="omit"; with on_error="raise" it would raise.
• listdir() and tree(), which call ls().

find, glob, du, info, exists, isdir, and expand_path do not call ls() and keep their results; find() of a missing path returns [].

These tests assert the current behavior and change with it:

• test_ls_dirs asserts len(fs.ls(f"{dir_}/prefix/test_")) == 0.
• test_ls_empty_refresh_evicts_cached_listing, test_refresh_evicts_cached_object_and_bucket_not_found, and test_missing_object_drops_cached_parent_listing[ls] call ls() on a missing path to check the cache eviction; they would expect FileNotFoundError and keep checking the eviction.

Validation

Remove the xfail marks of the four conformance tests in tests/pyathena/filesystem/test_fsspec_conformance.py; they pass with the change.