Repository navigation
S3FileSystem.find(maxdepth=...) descends one level deeper than fsspec #933
Description
Problem
S3FileSystem.find(path, maxdepth=n) descends one level deeper than fsspec's maxdepth:
maxdepth |
fsspec AbstractFileSystem.find / walk |
PyAthena S3FileSystem.find |
|---|---|---|
0 |
ValueError("maxdepth must be at least 1") |
entries directly under path |
1 |
entries directly under path |
also the entries one level below |
n |
n levels |
n + 1 levels |
fsspec documents maxdepth as "the maximum number of levels to descend", and its walk() (fsspec 2026.9.0) rejects maxdepth < 1. Code written against fsspec that passes maxdepth=1 gets more files than expected from PyAthena.
From the code on master (775874c): _find() lists the current level with delimiter="/" and recurses into each subdirectory while maxdepth > 0, with maxdepth - 1. test_find_maxdepth in tests/pyathena/filesystem/test_s3.py encodes the current behavior (maxdepth=0 gives the root files, maxdepth=1 adds level1/).
Found by code reading during the review of #928.
Reproduction
fs.touch("s3://BUCKET/tmp/d/direct")
fs.touch("s3://BUCKET/tmp/d/sub/nested")
print(fs.find("s3://BUCKET/tmp/d", maxdepth=1))
# fsspec semantics: [".../d/direct"]
# PyAthena: [".../d/direct", ".../d/sub/nested"]Environment
- PyAthena master 775874c (behavior since
maxdepthsupport was added in v3.15.0), fsspec 2026.9.0.
Proposed fix (optional)
Align with fsspec: reject maxdepth < 1 and recurse only while more than one level remains. This changes results for existing callers that rely on the current numbering, so it needs a release note (and a decision on whether it belongs in 4.0.0 only).
Activity