Repository navigation
Settle the response models, S3Object, and the listing cache #1341
Copy link
Copy link
Open
Labels
Description
Part of #1327 (design problems 4 and 5 and defects 22 and 23 of the audit report).
Summary
Settle the shape of the response models, of S3Object as fsspec's info entry, and of the listing cache.
Current state (766d79d)
- Two model styles in two modules: frozen dataclasses with
from_responseins3_core.py(S3ObjectSummary,S3CommonPrefix,S3Bucket, pages, delete models) and property classes over raw dicts ins3_object.py(S3ObjectVersion,S3MultipartUpload,S3Owner,S3PutObject,S3MultipartUploadPart,S3CompleteMultipartUpload), with a circular import.S3ObjectVersion.namedrops the version; checksum fields cover 11 algorithms on parts and 4 elsewhere. S3Objectis aMutableMapping, not adict: keys live in__dict__, so a key namedkeys,items,get,updateorpopbreaksdict(o)ando.keys()(defect 22); it has noreprand is not JSON-serializable;update()does not remap API names while__init__does; directory and bucket entries carryNone-valued keys that file entries omit. Building it costs 3.8 µs per listed key, against 1.1 µs for adictsubclass built from the typed page and 0.25 µs for a plain dict (about 3 s of CPU per million keys infind).S3Metadatais both the core's HeadObject result and the user-metadata mapping of fsspec'smetadata(): an existing object without user metadata is falsy, and results of different objects compare equal to each other and to{}.S3Coreresults: three**paramsprecedence rules (sent as given withTypeErroron duplicates; arguments win; params win over the path ingenerate_presigned_url);copy_object,delete_object,replace_object_metadatareturnNoneand discard version and ETag;to_dict()ofS3PutObjectandS3CompleteMultipartUploadexposes_etag(returned bytouch()).S3Pathround trips: access-point ARN buckets cannot be parsed though the core accepts them;key=""re-parses asNone; a listed key ending in a version-like query (data?versionId=1) parses as version1ofdataand cannot be addressed (defect 23).- The listing cache is an ad-hoc key scheme (path,
"",(path, "/"),(path, ""),(path, "lookups")) spread over about nine methods, andinvalidate_cachere-enumerates every delimiter and query spelling.info()andls(detail=True)return the cachedS3Objectby reference, so a caller's mutation changes the cache.
Proposal (for decision)
S3Objectas adictsubclass with attribute access, built directly from the typed page models (from_summary,from_version,from_prefix,root), which fixes defect 22,reprand JSON, and the per-entry cost.- One module and one style for the response models; typed results for copy and delete; a typed HeadObject result with a
user_metadatamapping, whilefs.metadata()keeps returning the mapping. - One
**paramsrule acrossS3Core. - An escape for keys that end in a version-like query, or a documented limit.
- A typed cache object over
fs.dircachethat owns the key scheme and invalidation, shared by both filesystems; return copies or read-only entries frominfo()/ls().
Risks: isinstance(entry, dict) becomes true and the pickled form changes; S3Core.head_object's return type changes; tests that inspect fs.dircache (#1328).
Activity