DD-2271 separate clients for upload and download

[janvanmansum]

Fixes DD-2271

Description of changes

• Split the S3AsyncClient in S3AccessIO in two separated clients: one for read and one for write actions, with multipart enabled on the write-client and disabled on the read-client. In many calls it does not really whether multipart is enabled, but for readability I stuck to the logic that read actions are performed by the read client and write actions by the write client.
• Fixed some problems found by AI code review:
  • potential concurrency problems with the static maps
  • handling of InterruptedException
  • replaced syserr with logging.

Notify

@DANS-KNAW/core-systems

[Copilot]

Pull request overview

This PR refactors S3AccessIO to use separate AWS S3 async clients for download (read) vs upload (write), enabling multipart configuration only for upload operations while leaving read operations on a non-multipart client.

Changes:

• Split the single S3AsyncClient into s3ReadClient and s3WriteClient, and routed S3 operations to the appropriate client.
• Introduced separate static client caches for upload vs download clients and updated TransferManager to use the upload client.
• Minor cleanups (e.g., isEmpty() usage, less printStackTrace usage).

Comments suppressed due to low confidence (2)

src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java:1054

• S3Presigner is obtained from a static cache (getPresigner/driverPresignerMap) but is closed in this method’s finally block later. Closing a cached/shared presigner can cause future upload URL generation to fail. Prefer either per-call presigner construction (and close) or cached presigner reuse without closing (or evict/recreate after closing).

        if (s3ReadClient == null || s3WriteClient == null) {
            throw new IOException("ERROR: s3 not initialised. ");
        }

src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java:1117

• This method closes s3Presigner at the end of multipart URL generation, but s3Presigner is sourced from a static cache (driverPresignerMap). Closing it here can break later presign calls for other requests/threads. Consider using a non-cached presigner for these operations, or keep it cached and remove all close() calls on the shared instance.

            // Use the existing s3 async client for the createMultipartUpload operation
            CompletableFuture<CreateMultipartUploadResponse> createMultipartUploadFuture = s3WriteClient.createMultipartUpload(createMultipartUploadRequestBuilder.build());
            CreateMultipartUploadResponse createMultipartUploadResponse = createMultipartUploadFuture.join();

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.