You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
S3FileSystem keeps the cached object after rm_file() deletes a specific version #931
With version_aware=True, deleting a specific version through rm_file("bucket/dir/key?versionId=...") leaves the cached HeadObject entry of the unversioned path bucket/dir/key. After that, exists() and info() on bucket/dir/key keep returning the cached object:
If the deleted version was the only one, exists("bucket/dir/key") stays True.
If it was the latest version, info("bucket/dir/key") keeps returning the deleted version's metadata instead of the version that became current.
Expected: after a version is deleted through the filesystem, the next exists()/info() of the object path reflects S3.
rm_file() calls invalidate_cache(path) with the version-qualified path (pyathena/filesystem/s3.py, rm_file).
invalidate_cache() pops the path itself and walks _parent(). For bucket/dir/key?versionId=v1, the parent is bucket/dir, so the bucket/dir/key entry that _head_object() stored is never popped.
exists() returns True when _ls_from_cache() finds that entry.
Found by code reading during the review of #928 (not reproduced live yet).
Reproduction
frompyathenaimportconnectfrompyathena.filesystem.s3importS3FileSystemfs=S3FileSystem(connect(), version_aware=True, skip_instance_cache=True)
path="s3://VERSIONED_BUCKET/tmp/key"fs.pipe(path, b"data")
version_id=fs.info(path)["version_id"] # caches bucket/tmp/keyfs.rm_file(f"{path}?versionId={version_id}")
print(fs.exists(path)) # expected False, returns True from the cache
Environment
PyAthena master 775874c, S3FileSystem with version_aware=True.
Proposed fix (optional)
In invalidate_cache() (or in the version-aware delete paths), also invalidate the path without its versionId query. Add a test that deletes the only version of an object and checks exists().
Activity