Skip to content

Settle the response models, S3Object, and the listing cache #1341

Open

Description

@laughingman7743

Part of #1327 (design problems 4 and 5 and defects 22 and 23 of the audit report).

Summary

Settle the shape of the response models, of S3Object as fsspec's info entry, and of the listing cache.

Current state (766d79d)

  1. Two model styles in two modules: frozen dataclasses with from_response in s3_core.py (S3ObjectSummary, S3CommonPrefix, S3Bucket, pages, delete models) and property classes over raw dicts in s3_object.py (S3ObjectVersion, S3MultipartUpload, S3Owner, S3PutObject, S3MultipartUploadPart, S3CompleteMultipartUpload), with a circular import. S3ObjectVersion.name drops the version; checksum fields cover 11 algorithms on parts and 4 elsewhere.
  2. S3Object is a MutableMapping, not a dict: keys live in __dict__, so a key named keys, items, get, update or pop breaks dict(o) and o.keys() (defect 22); it has no repr and is not JSON-serializable; update() does not remap API names while __init__ does; directory and bucket entries carry None-valued keys that file entries omit. Building it costs 3.8 µs per listed key, against 1.1 µs for a dict subclass built from the typed page and 0.25 µs for a plain dict (about 3 s of CPU per million keys in find).
  3. S3Metadata is both the core's HeadObject result and the user-metadata mapping of fsspec's metadata(): an existing object without user metadata is falsy, and results of different objects compare equal to each other and to {}.
  4. S3Core results: three **params precedence rules (sent as given with TypeError on duplicates; arguments win; params win over the path in generate_presigned_url); copy_object, delete_object, replace_object_metadata return None and discard version and ETag; to_dict() of S3PutObject and S3CompleteMultipartUpload exposes _etag (returned by touch()).
  5. S3Path round trips: access-point ARN buckets cannot be parsed though the core accepts them; key="" re-parses as None; a listed key ending in a version-like query (data?versionId=1) parses as version 1 of data and cannot be addressed (defect 23).
  6. The listing cache is an ad-hoc key scheme (path, "", (path, "/"), (path, ""), (path, "lookups")) spread over about nine methods, and invalidate_cache re-enumerates every delimiter and query spelling. info() and ls(detail=True) return the cached S3Object by reference, so a caller's mutation changes the cache.

Proposal (for decision)

  1. S3Object as a dict subclass with attribute access, built directly from the typed page models (from_summary, from_version, from_prefix, root), which fixes defect 22, repr and JSON, and the per-entry cost.
  2. One module and one style for the response models; typed results for copy and delete; a typed HeadObject result with a user_metadata mapping, while fs.metadata() keeps returning the mapping.
  3. One **params rule across S3Core.
  4. An escape for keys that end in a version-like query, or a documented limit.
  5. A typed cache object over fs.dircache that owns the key scheme and invalidation, shared by both filesystems; return copies or read-only entries from info()/ls().

Risks: isinstance(entry, dict) becomes true and the pickled form changes; S3Core.head_object's return type changes; tests that inspect fs.dircache (#1328).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions