feat(file-server): add S3 ListObjectVersions and delete marker helpers

[yeziR4]

Resolves #690
Refs autonomys/auto-drive#789

Summary

Following the code review in autonomys/auto-drive#789 by @jim-counter and tracking issue #690 by @EmilFattakhov, this PR extracts the pure S3 versioning logic and delete marker helpers from apps/backend/src/core/s3/index.ts into the SDK's @autonomys/file-server package (src/s3/):

1. Delete Marker Helpers:

   • deleteMarkerVersionId(deletedAt: Date): Formats deterministic delete marker version IDs (dm-${deletedAt.getTime()}).
   • resolveDeleteMarkerResult(deletedAt): Formats DeleteObjectResult for DeleteObject responses with x-amz-delete-marker headers.

2. Types:

   • S3VersionEntry, S3DeleteMarkerEntry, ListObjectVersionsParams, ListObjectVersionsResult, S3VersionRow, DeleteObjectResult.

3. Pure Aggregation Engine:

   • buildListVersionsResult(rows: S3VersionRow[], maxKeys: number):
     • Aggregates version rows by key, marking the first (newest) row of active keys as isLatest: true.
     • Synthesises delete markers (isLatest: true) for soft-deleted (pointerDeletedAt) or owner-removed keys.
     • De-duplicates repeated writes sharing the same CID under a single key.
     • Preserves whole keys across page boundaries, cleanly computing isTruncated and nextKeyMarker.

Verification

• Added comprehensive unit tests in src/s3/__tests__/listObjectVersions.test.ts.
• All 93 tests in @autonomys/file-server pass (5/5 test suites green).
• Prettier and TypeScript builds pass without errors.

[jim-counter]

Review: auto-sdk #696 - S3 ListObjectVersions + delete marker helpers

Checked against the source: auto-drive#789 head f31e1c2, apps/backend/src/core/s3/index.ts
and apps/backend/src/app/controllers/s3/. Tests 93/93, tsc --noEmit clean.

Verdict

The core choice is right. buildListVersionsResult is the valuable pure kernel and the
transcription is faithful; S3VersionRow is the correct seam, since auto-drive already imports
S3ObjectListing into its repository layer. deleteMarkerVersionId deserves sharing because
ListObjectVersions and DeleteObject must agree on the format.

Two problems: the guardrails didn't come with it, and the helpers needed to use it were left
behind.

Blocking

1. Undocumented over-fetch precondition - silent data loss. isTruncated is set only on
seeing a row from a (maxKeys+1)-th key, so the caller must fetch maxKeys + 1 distinct keys.
auto-drive does and says so at the call site; the SDK docstring documents only row ordering. A
consumer fetching exactly maxKeys keys gets isTruncated: false and silently drops the rest
of the bucket. listObjects.ts already defends against this exact case with
computeListObjectsDbLimit + a dbLimit argument ("one extra empty page is harmless, but
silently dropping data is not"). No test covers it.

2. The two comments that mattered most were dropped. The #790 KNOWN LIMITATION (maxKeys
bounds keys, not Version+DeleteMarker entries - a deliberate spec deviation, no
version-id-marker resume) and the "display-only" caveat on dm- ids. The SDK now reads as if
dm-<epoch> is a resolvable versionId, when deleteObjectVersionHandler unconditionally 403s
and findVersionByCid can never match it. These matter more in a library than in the app.

3. pointerDeletedAt / ownerRemoved relaxed from required to optional. Both are required
upstream. Optional removes the only guard against a consumer forgetting to select the
delete/moderation state, and the failure mode is deleted or moderated content reported as a live
latest version. Unlike md5?, no rationale is given. Also undocumented: both are read from the
first row only and assumed constant across the key's group.

Missed

planListingEncoding / hasXmlIllegalChars / encodeS3Key (controllers/s3/utils.ts) -
the strongest miss, because it's downstream of this PR: auto-drive's handler calls
planListingEncoding on the exact result buildListVersionsResult returns. As shipped the SDK
gives you an aggregation engine and withholds the piece that stops its output producing invalid
XML or a 500. Pure, zero-dep, already unit-tested upstream.

Object Lock / versioning bodies (objectRetentionBody, objectLockConfigurationBody,
bucketVersioningBody, objectLegalHoldBody, OBJECT_LOCK_RETENTION_YEARS) - the other half of
"pure S3 versioning logic", encoding an invariant already violated once: bucket DefaultRetention
and per-object RetainUntilDate "were inconsistent before: a 100-year bucket default vs. a
year-9999 per-object date." Exactly what a shared constant prevents recurring.

Lower priority: parseMultipartParts (natural companion to the multipartETag the SDK already
owns, but needs a fast-xml-parser dep - fine to defer); parseBucketAndKey / parseCopySource.
sendXML and buildObjectLocation are framework-coupled and correctly left behind.

Smaller points

• resolveDeleteMarkerResult is the only new code here, not extracted code - a 4-line ternary
  upstream inlines at two sites. Defensible, but not a headline item, and DeleteObject doesn't
  belong in listObjectVersions.ts. Split into deleteMarker.ts.
• ListObjectVersionsParams is exported but consumed by nothing; ListObjectVersionsResult lacks
  the name/prefix/maxKeys its sibling ListObjectsResult carries, plus keyMarker. Adding
  finalizeListObjectVersions(params, rows, dbLimit) to mirror finalizeListObjects fixes this
  and blocking item 1 in one move.
• ETag fallback (md5 ?? cid) is baked into S3VersionEntry but left to the caller for
  S3ObjectListing. Extract objectETag(md5, cid) into etag.ts for both. Related: formatETag
  is documented as taking "a raw hex MD5" but is passed a CID.
• The S3VersionEntry.etag doc says "matching GET/HEAD/List". It matches List only - GET/HEAD do
  if (etag) res.set(...) and omit the header for legacy md5-null objects.

Test gaps

The 13 tests are well written but cover only happy paths. Missing: the truncation precondition
above; non-adjacent duplicate CIDs (A, B, A - the current test uses adjacent rows, which a naive
prev-cid check would also pass); both delete signals set at once; ownerRemoved on a multi-version
key; a truncated page ending on a deleted key. The maxKeys <= 0 test asserts an unresumable page
(isTruncated: true, nextKeyMarker: null) without saying so - listObjects.ts at least explains
that choice in a comment.

Order of work

Releases here are manual (workflow_dispatch + lerna fixed version), so merging doesn't cut a
release and several PRs can publish together. The unit that matters for auto-drive is therefore
the release, not the PR.

Must be in this PR - blocking 1-3, finalizeListObjectVersions, the delete-marker module
split, the test gaps. These are defects in what's being shipped, not added scope. All additive.

Must be in the same release - planListingEncoding / hasXmlIllegalChars / encodeS3Key.
The risk is auto-drive getting a version with the aggregation engine but not the encoder its own
handler calls on that engine's output. A separate PR landing before the same release solves that
equally well; author's choice which.

Separate PRs. Object Lock bodies (in scope by this PR's own description, small and pure -
worth taking here if offered, but not worth asking for while blocking items are open);
parseMultipartParts (adds a fast-xml-parser dependency - that decision deserves its own PR);
objectETag (refactors the existing listObjects path rather than adding to it);
parseBucketAndKey / parseCopySource (independent, no release coupling).