aws client config

[drernie]

When creating a client
I want quiltcore-java to use any nextflow config settings
So it works with my Security Policy

We have this added in nextflow config (in seqera):

aws {  client {    storageEncryption = 'AES256'  }}

[drernie]

AWS Client is created by nf-amazon using client.s3Config.

StorageEncryption is a predefined key, along with:

• private String storageClass
• private String storageEncryption
• private String storageKmsKeyId
• private Boolean debug
• private CannedAccessControlList s3Acl
• private Boolean pathStyleAccess
• private Boolean anonymous
• private Boolean requesterPays

[drernie]

This property is stored in the S3Client wrapper, and used by:

• S3OutputStream.java
• S3CopyStream.java

[drernie]

quiltcore-java writes to S3 in three places, all of which will presumably need to be modified to use storageEncryption:

A. Manifests

Using java.io.ByteArrayOutputStream

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        newManifest.serializeToOutputStream(out);

B. Individual files

Using S3PhysicalKey::putBytes

    public void putBytes(byte[] bytes) throws IOException {
        S3Client s3 = getClient();

        PutObjectRequest objectRequest = PutObjectRequest.builder()
            .bucket(bucket)
            .key(key)
            .build();

        s3.putObject(objectRequest, RequestBody.fromBytes(bytes));
    }

C. Package pushes

Using software.amazon.awssdk.transfer.s3.S3TransferManager

            S3TransferManager transferManager =
                S3TransferManager.builder()
                    .s3Client(s3)
                    .build()
...
                    .putObjectRequest(req -> req.bucket(destBucket).key(destPath))
                    .addTransferListener(LoggingTransferListener.create())
                    .source(sourcePath)
                    .build();

                logger.debug("Uploading file to bucket: {}, key: {}", destBucket, destPath);
                FileUpload uploadFile = transferManager.uploadFile(uploadFileRequest);
                futures.add(Map.entry(logicalKey, uploadFile.completionFuture()));

[drernie]

We probably also need one new API to register storageEncryption (and possibly more general attributes) somewhere, most likely the S3ClientStore.

Then we need to update nf-quilt to parse the aws.client attributes, and pass those to quiltcore-java.

Of course this would all be easier if we could reuse nf-amazon, but last time we tried that proved impossible because it is non-trivially integrated into nextflow proper. :-(

[kevinemoore]

@drernie Although it would be easier for users if nf-quilt could parse the nf-amazon configuration, couldn't we get by with simply allowing nf-quilt users to add the configuration setting in the configuration options for nf-quilt itself? Of course, that requires additional implementation as well, but it might be more straightforward since it wouldn't depend on any other plugins.

[drernie]

Parsing the configuration is the least difficult part. Anyway, that would be handled by nf-quilt, which requires a different ticket.

…

On Sun, Jan 26, 2025 at 18:05 Kevin Moore ***@***.***> wrote: @drernie <https://github.com/drernie> Although it would be easier for users if nf-quilt could parse the nf-amazon configuration, couldn't we get by with simply allowing nf-quilt users to add the configuration setting in the configuration options for nf-quilt itself? Of course, that requires additional implementation as well, but it might be more straightforward since it wouldn't depend on any other plugins. — Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE2TZHE26RGP2MZWZJFCL2MWH5TAVCNFSM6AAAAABV4VBRRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUG4ZDKMRUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>