Skip to content

Fix recursive chmod, empty key segments, and version paths in listings #1335

Open

Description

@laughingman7743

Part of #1327 (defects 8 to 11 of the audit report).

Summary

Listing and recursive operations mishandle four path shapes. Each was reproduced offline at 766d79d.

Defects

  1. chmod(path, acl, recursive=True) skips the object at the key when the key is also a prefix. With objects a and a/x, chmod("bucket/a", acl) updates a, and recursive=True updates only a/x: find(path, withdirs=False) lists a/... (s3.py:2206), and the method returns early on the assumption that a key prefix is not an object (s3.py:2232-2235). rm(recursive=True) does delete a.
  2. Keys with an empty segment (dir//x). A common prefix is named with c.prefix[:-1].rstrip("/") (s3.py:516-518, :585), so dir// becomes bucket/dir, the directory being listed, and _find_levels recurses into the same name (s3.py:857-860). ls("bucket/dir") returns ['bucket/dir']; find("bucket/dir", maxdepth=4, withdirs=True) returns bucket/dir five times; find("bucket/dir", maxdepth=4) returns [] and misses dir//x, which find without maxdepth finds. rm, copy and glob with a maxdepth miss such objects.
  3. ls/find of a version-qualified path. For bucket/dir?versionId=v1, _ls_dirs lists the prefix dir/, puts the version ID on the directory entries, and caches the listing under ("bucket/dir?versionId=v1", "/") (s3.py:497-527), which no invalidation reaches: after pipe_file("bucket/dir/b"), ls of the version path still returned the old listing without a request. A version path names one object, as info, expand_path and rm treat it.
  4. _split_version_paths is quadratic: [p for p in paths if p not in versions] is a list membership test (s3.py:963-964); 20,000 paths take 1.1 s. It is reached by cat, copy and get with lists of paths.

Fix direction

Include the object at the key in recursive chmod; name common prefixes by dropping exactly one delimiter and recurse on the raw prefix; resolve ls/find of a version path to that version without listing or caching; use a set in _split_version_paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions