From d95d0949308450104d45cea5bfe1085dc406ac05 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 8 Apr 2026 20:27:36 +0000 Subject: [PATCH 01/10] Add AWS SDK v2 dependencies alongside v1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add software.amazon.awssdk:2.37.5 (matching JDBC driver version): - BOM import in dependencyManagement - s3, s3-transfer-manager, netty-nio-client, auth, http-auth-aws - aws-crt excluded (cannot be shaded) Add shade relocation rules: - software.amazon.awssdk - software.amazon.eventstream - org.reactivestreams (new transitive dep from v2 async) Keep v1 (com.amazonaws) deps for now — will be removed after all files are migrated. Co-Authored-By: Claude Opus 4.6 --- pom.xml | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/pom.xml b/pom.xml index c0e5a9f84..e2ceae246 100644 --- a/pom.xml +++ b/pom.xml @@ -43,6 +43,7 @@ 4.4.16 true 1.11.4 + 2.37.5 1.79 1.9.13 1.18.0 @@ -108,6 +109,13 @@ pom import + + software.amazon.awssdk + bom + ${awssdk.version} + pom + import + com.google.code.gson gson @@ -996,6 +1004,45 @@ org.slf4j slf4j-api + + + software.amazon.awssdk + auth + + + software.amazon.awssdk + http-auth-aws + + + software.amazon.awssdk + aws-crt + + + + + software.amazon.awssdk + netty-nio-client + + + software.amazon.awssdk + s3 + + + software.amazon.awssdk + aws-crt + + + + + software.amazon.awssdk + s3-transfer-manager + + + software.amazon.awssdk + aws-crt + + + com.github.luben zstd-jni @@ -1640,6 +1687,18 @@ com.amazonaws ${shadeBase}.com.amazonaws + + software.amazon.awssdk + ${shadeBase}.software.amazon.awssdk + + + software.amazon.eventstream + ${shadeBase}.software.amazon.eventstream + + + org.reactivestreams + ${shadeBase}.org.reactivestreams + com.google.api.gax ${shadeBase}.com.google.api.gax From 091648a064004c99327997614ccfd2d11ce67176 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 8 Apr 2026 20:34:45 +0000 Subject: [PATCH 02/10] Migrate IcebergS3Client from AWS SDK v1 to v2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace all com.amazonaws imports with software.amazon.awssdk equivalents: IcebergS3Client: - AmazonS3 → S3AsyncClient with NettyNioAsyncHttpClient - TransferManager → S3TransferManager - BasicAWSCredentials/BasicSessionCredentials → AwsBasicCredentials/AwsSessionCredentials - ClientConfiguration → ProxyConfiguration + builder params - ObjectMetadata SSE → PutObjectRequest.builder().serverSideEncryption() - AmazonS3Exception → S3Exception with null-safe awsErrorDetails() check - RegionUtils.getRegion() → Region.of() - Multipart threshold set to 16MB (match JDBC) - Streams wrapped in BufferedInputStream (JDBC CipherInputStream fix) - SSLConnectionSocketFactory removed (v2 Netty handles SSL) IcebergS3ObjectMetadata: - Removed v1 ObjectMetadata wrapper - Now uses plain Map + fields (like IcebergCommonObjectMetadata) IcebergStorageClientFactory: - Removed ClientConfiguration creation - Passes maxConnections directly to IcebergS3Client Co-Authored-By: Claude Opus 4.6 --- .plans/AWS_SDK_V2_MIGRATION_PLAN.md | 268 ++++++++++++++ .../fileTransferAgent/IcebergS3Client.java | 333 +++++++++--------- .../IcebergS3ObjectMetadata.java | 54 +-- .../IcebergStorageClientFactory.java | 28 +- 4 files changed, 480 insertions(+), 203 deletions(-) create mode 100644 .plans/AWS_SDK_V2_MIGRATION_PLAN.md diff --git a/.plans/AWS_SDK_V2_MIGRATION_PLAN.md b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md new file mode 100644 index 000000000..e395d76da --- /dev/null +++ b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md @@ -0,0 +1,268 @@ +# Plan: Migrate from AWS SDK v1 to AWS SDK v2 + +## Background + +The ingest SDK uses AWS SDK v1 (`com.amazonaws:1.12.655`) for S3 operations, GCS +access (via S3-compatible API), and HTTP proxy configuration. AWS SDK v1 is +deprecated and will reach end-of-support in 2025. + +The JDBC driver (`snowflake-jdbc`) has **already completed** this migration to +AWS SDK v2 (`software.amazon.awssdk:2.37.5`). We can reference their +implementation and learn from their bug fixes. + +--- + +## Scope + +**19 files** import `com.amazonaws.*`, all in `fileTransferAgent/`: + +| Category | Files | v2 Equivalent | +|---|---|---| +| S3 client (non-Iceberg) | `SnowflakeS3Client` | `S3AsyncClient` + `S3TransferManager` | +| S3 client (Iceberg) | `IcebergS3Client` | `S3AsyncClient` + `S3TransferManager` | +| GCS via AWS SDK | `AwsSdkGCPSigner`, `GCSAccessStrategyAwsSdk` | v2 `Signer` interface | +| Proxy/HTTP config | `S3HttpUtil`, `JdbcHttpUtil` | `ProxyConfiguration` + `ClientOverrideConfiguration` | +| Metadata wrappers | `S3ObjectMetadata`, `IcebergS3ObjectMetadata` | Direct `HeadObjectResponse` | +| Factories | `StorageClientFactory`, `IcebergStorageClientFactory` | Updated builders | +| Iterators | `S3ObjectSummariesIterator`, `StorageObjectSummary*` | `ListObjectsV2Response` | +| HTTP interceptor | `HeaderCustomizerHttpRequestInterceptor` | `ExecutionInterceptor` | +| Route planner | `SnowflakeMutableProxyRoutePlanner` | `ProxyConfiguration` | +| Trust manager | `SFTrustManager` | Minimal changes | + +--- + +## Known Risks (from JDBC's migration experience) + +### RISK 1: CipherInputStream + mark/reset — CRITICAL + +**Problem:** `CipherInputStream` does not support `mark()/reset()`. AWS SDK v2's +`AsyncRequestBody.fromInputStream()` may retry failed uploads by resetting the +stream. If the stream is a `CipherInputStream`, this causes silent data corruption +(~1% of uploads produce unreadable files). + +**Applies to us?** **YES — CRITICAL.** Our `EncryptionProvider.encrypt()` returns +a `CipherInputStream` (line 191), which is passed directly to TransferManager. +Same pattern that caused JDBC's bug. + +**How JDBC fixed it (PR #2502):** In `SnowflakeS3Client.upload()`, they wrap +the stream with `BufferedInputStream` before passing to `AsyncRequestBody`: +```java +AsyncRequestBody.fromInputStream( + new BufferedInputStream(uploadStreamInfo.left), + contentLength, executorService) +``` +`BufferedInputStream` supports mark/reset, so SDK retries work correctly. +File: `snowflake-jdbc/internal/jdbc/cloud/storage/SnowflakeS3Client.java` + +**Fix:** Same — wrap `CipherInputStream` in `BufferedInputStream` at upload time. + +**Affected files:** `SnowflakeS3Client.upload()` (non-Iceberg 128-bit encryption +path only). `IcebergS3Client` is NOT affected (no client-side encryption). + +### RISK 2: Multipart threshold change — MEDIUM + +**Problem:** AWS SDK v1 TransferManager default multipart threshold was 16MB. +SDK v2's `S3AsyncClient` with `multipartEnabled(true)` uses 8MB. Files 8-16MB +switch from single-part to multipart, adding extra API round trips. + +**Applies to us?** **YES.** Blobs can be up to 1 GB and the 8-16MB range is +hit during normal operation. + +**How JDBC fixed it (PR #2526):** Explicitly set 16MB threshold on the +`S3AsyncClient` builder: +```java +.multipartConfiguration( + MultipartConfiguration.builder() + .thresholdInBytes(16L * 1024 * 1024) + .build()) +``` +File: `snowflake-jdbc/internal/jdbc/cloud/storage/SnowflakeS3Client.java` + +**Fix:** Same — set 16MB threshold on all `S3AsyncClient` builders. + +### RISK 3: S3Exception.awsErrorDetails() null — MEDIUM + +**Problem:** `S3Exception.awsErrorDetails()` can return `null` in v2. Code that +calls `.errorCode()` on it without a null check will NPE. + +**Applies to us?** **YES.** Our error handlers in `SnowflakeS3Client` and +`IcebergS3Client` check for `ExpiredToken` error code. + +**How JDBC fixed it (PR #2550):** Added null check in `S3ErrorHandler.java`: +```java +if (e.awsErrorDetails() != null + && EXPIRED_AWS_TOKEN_ERROR_CODE.equalsIgnoreCase( + e.awsErrorDetails().errorCode())) { +``` +Also used `constant.equalsIgnoreCase(variable)` pattern for null safety. +File: `snowflake-jdbc/internal/jdbc/cloud/storage/S3ErrorHandler.java` + +**Fix:** Same — null-check `awsErrorDetails()` in all error handlers. + +### RISK 4: Custom signer API change — HIGH + +**Problem:** v1 uses `SignerFactory.registerSigner()` + `AWS4Signer` extension. +v2 uses `SdkAdvancedClientOption.SIGNER` + `software.amazon.awssdk.core.signer.Signer`. + +**Applies to us?** **YES.** `GCSAccessStrategyAwsSdk` registers a custom +`AwsSdkGCPSigner` that extends v1's `AWS4Signer`. + +**How JDBC did it:** Rewrote `AwsSdkGCPSigner` to implement v2's `Signer` +interface. The v2 signer: +1. Strips the AWS `Authorization` header +2. Adds `Authorization: Bearer ` for GCS +3. Maps `x-amz-*` → `x-goog-*` headers +4. Injected via `SdkAdvancedClientOption.SIGNER` on the client builder +5. Uses `AnonymousCredentialsProvider` (signing handled by custom signer) +File: `snowflake-jdbc/internal/jdbc/cloud/storage/AwsSdkGCPSigner.java` + +**Fix:** Replicate JDBC's v2 `AwsSdkGCPSigner` implementation. + +### RISK 5: aws-crt shading — LOW + +**Problem:** `aws-crt` native library cannot be shaded. Must be excluded. + +**Applies to us?** **YES.** We shade dependencies. + +**How JDBC did it:** Excludes `aws-crt` from `s3`, `s3-transfer-manager`, and +`http-auth-aws` dependencies in `parent-pom.xml`. Also has a build comment: +"aws-crt cannot be shaded". +File: `snowflake-jdbc/parent-pom.xml` + +**Fix:** Same — exclude `aws-crt` from all v2 deps that pull it transitively. + +--- + +## Migration Steps + +### Step 1: Update dependencies + +Replace v1 deps with v2 in `pom.xml`: + +```xml + +com.amazonaws:aws-java-sdk-core:1.12.655 +com.amazonaws:aws-java-sdk-kms:1.12.655 +com.amazonaws:aws-java-sdk-s3:1.12.655 + + +software.amazon.awssdk:s3 +software.amazon.awssdk:s3-transfer-manager +software.amazon.awssdk:netty-nio-client +software.amazon.awssdk:auth +software.amazon.awssdk:regions + +software.amazon.awssdk:bom:2.37.5 (type=pom, scope=import) +``` + +Exclude `aws-crt` from all v2 deps. Update shade rules: +`com.amazonaws` → `software.amazon.awssdk`. + +### Step 2: Migrate IcebergS3Client (lowest risk) + +Start here because it has NO client-side encryption (avoids Risk 1). + +**Changes:** +- `AmazonS3ClientBuilder` → `S3AsyncClient.builder()` +- `BasicAWSCredentials`/`BasicSessionCredentials` → `AwsBasicCredentials`/`AwsSessionCredentials` +- `AWSStaticCredentialsProvider` → `StaticCredentialsProvider` +- `ClientConfiguration` → `NettyNioAsyncHttpClient.builder()` with `ProxyConfiguration` +- `TransferManager` → `S3TransferManager` +- `ObjectMetadata` → `PutObjectRequest.builder()` metadata +- `SSEAwsKeyManagementParams` → `ServerSideEncryption.AWS_KMS` + `ssekmsKeyId()` +- Set multipart threshold to 16MB (Risk 2) +- Null-check `awsErrorDetails()` in error handler (Risk 3) +- `Region`/`RegionUtils` → `software.amazon.awssdk.regions.Region` + +### Step 3: Migrate SnowflakeS3Client (high risk — encryption) + +Same as Step 2, plus: +- Wrap `CipherInputStream` in `BufferedInputStream` for uploads (Risk 1) +- `AmazonS3EncryptionClient` → removed (encryption is manual via EncryptionProvider) +- `CryptoConfiguration`/`EncryptionMaterials` → removed (not needed, encryption is JCE-based) +- Keep `EncryptionProvider.encrypt()` returning `CipherInputStream` but wrap at upload time + +### Step 4: Migrate GCS-via-AWS-SDK + +**Changes:** +- `AwsSdkGCPSigner` (extends `AWS4Signer`) → implement v2 `Signer` interface + - Strip AWS Authorization header, add `Bearer` token + - Map `x-amz-*` → `x-goog-*` headers + - Inject via `SdkAdvancedClientOption.SIGNER` +- `GCSAccessStrategyAwsSdk`: + - Use `S3AsyncClient` with GCS endpoint (`storage.googleapis.com`) + - `AnonymousCredentialsProvider` (signing handled by custom signer) + - `forcePathStyle(false)` for virtual-hosted style + +### Step 5: Migrate support classes + +- `S3HttpUtil` → use `ProxyConfiguration` from v2 +- `S3ObjectMetadata`/`IcebergS3ObjectMetadata` → wrap v2 response types +- `HeaderCustomizerHttpRequestInterceptor` → implement `ExecutionInterceptor` (v2) +- `S3ObjectSummariesIterator` → use `ListObjectsV2Response` +- `StorageClientFactory`/`IcebergStorageClientFactory` → update builders +- `SnowflakeMutableProxyRoutePlanner` → `ProxyConfiguration` (v2 handles this) +- `StorageObjectSummary`/`StorageObjectSummaryCollection` → adapt to v2 types +- `StorageProviderException` → catch `SdkException` (v2 base exception) + +### Step 6: Update shade plugin + +```xml + +com.amazonaws + +software.amazon.awssdk +${shadeBase}.software.amazon.awssdk +``` + +### Step 7: Update tests + +Update all test files that reference v1 types. Verify: +- Encryption round-trip (EncryptionProviderTest, GcmEncryptionProviderTest) +- S3 client tests +- GCS signer test (AwsSdkGCPSignerTest) +- Integration tests on all 3 clouds + +--- + +## PR Strategy + +| PR | Content | Risk Level | +|---|---|---| +| PR 1 | Add v2 deps alongside v1, update shade rules | Low | +| PR 2 | Migrate `IcebergS3Client` + `IcebergStorageClientFactory` | Medium | +| PR 3 | Migrate `SnowflakeS3Client` + `StorageClientFactory` (encryption path) | **High** | +| PR 4 | Migrate `AwsSdkGCPSigner` + `GCSAccessStrategyAwsSdk` | Medium | +| PR 5 | Migrate support classes (metadata, iterators, interceptor, proxy) | Low | +| PR 6 | Remove v1 deps, clean up shade rules | Low | + +--- + +## JDBC Reference Files + +Key JDBC files to reference during migration (at `snowflakedb/snowflake-jdbc` main branch): + +| JDBC File | Ingest Equivalent | +|---|---| +| `internal/jdbc/cloud/storage/SnowflakeS3Client.java` | `SnowflakeS3Client.java` | +| `internal/jdbc/cloud/storage/GCSAccessStrategyAwsSdk.java` | `GCSAccessStrategyAwsSdk.java` | +| `internal/jdbc/cloud/storage/AwsSdkGCPSigner.java` | `AwsSdkGCPSigner.java` | +| `internal/jdbc/cloud/storage/EncryptionProvider.java` | `EncryptionProvider.java` | +| `internal/jdbc/cloud/storage/S3ErrorHandler.java` | Error handling in S3 clients | +| `parent-pom.xml` (deps section) | `pom.xml` | + +--- + +## Verification + +- [ ] `mvn compiler:compile` passes +- [ ] All unit tests pass +- [ ] Integration tests pass on S3, Azure, GCS +- [ ] Encryption round-trip verified (upload encrypted → download → decrypt → matches original) +- [ ] Multipart uploads > 16MB work correctly +- [ ] Uploads < 16MB use single-part (not multipart) +- [ ] GCS uploads via S3-compatible API work +- [ ] SSE-KMS uploads (Iceberg path) work +- [ ] Proxy configuration works +- [ ] Shaded jar contains no `com.amazonaws` classes +- [ ] e2e-jar-test passes (shaded + unshaded) diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java index 79f5e1ea5..1db324774 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java @@ -4,53 +4,48 @@ import static net.snowflake.ingest.streaming.internal.fileTransferAgent.StorageClientUtil.createDefaultExecutorService; import static net.snowflake.ingest.streaming.internal.fileTransferAgent.StorageClientUtil.getRootCause; -import com.amazonaws.AmazonClientException; -import com.amazonaws.AmazonServiceException; -import com.amazonaws.ClientConfiguration; -import com.amazonaws.Protocol; -import com.amazonaws.auth.AWSCredentials; -import com.amazonaws.auth.AWSStaticCredentialsProvider; -import com.amazonaws.auth.BasicAWSCredentials; -import com.amazonaws.auth.BasicSessionCredentials; -import com.amazonaws.client.builder.AwsClientBuilder; -import com.amazonaws.client.builder.ExecutorFactory; -import com.amazonaws.regions.Region; -import com.amazonaws.regions.RegionUtils; -import com.amazonaws.services.s3.AmazonS3; -import com.amazonaws.services.s3.AmazonS3Builder; -import com.amazonaws.services.s3.AmazonS3Client; -import com.amazonaws.services.s3.model.AmazonS3Exception; -import com.amazonaws.services.s3.model.ObjectMetadata; -import com.amazonaws.services.s3.model.PutObjectRequest; -import com.amazonaws.services.s3.model.SSEAlgorithm; -import com.amazonaws.services.s3.model.SSEAwsKeyManagementParams; -import com.amazonaws.services.s3.transfer.TransferManager; -import com.amazonaws.services.s3.transfer.TransferManagerBuilder; -import com.amazonaws.services.s3.transfer.Upload; +import java.io.BufferedInputStream; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; import java.net.SocketTimeoutException; +import java.net.URI; import java.security.InvalidKeyException; -import java.security.KeyManagementException; -import java.security.NoSuchAlgorithmException; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Properties; +import java.util.Set; import java.util.concurrent.ExecutorService; import net.snowflake.ingest.streaming.internal.VolumeEncryptionMode; -import net.snowflake.ingest.utils.HttpUtil; import net.snowflake.ingest.utils.Logging; import net.snowflake.ingest.utils.SFPair; import net.snowflake.ingest.utils.SFSessionProperty; import net.snowflake.ingest.utils.Stopwatch; import org.apache.commons.io.IOUtils; import org.apache.http.HttpStatus; -import org.apache.http.conn.ssl.SSLConnectionSocketFactory; -import org.apache.http.conn.ssl.SSLInitializationException; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentials; +import software.amazon.awssdk.auth.credentials.AwsSessionCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.async.AsyncRequestBody; +import software.amazon.awssdk.core.exception.SdkException; +import software.amazon.awssdk.core.exception.SdkServiceException; +import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient; +import software.amazon.awssdk.http.nio.netty.ProxyConfiguration; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3AsyncClient; +import software.amazon.awssdk.services.s3.S3AsyncClientBuilder; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; +import software.amazon.awssdk.services.s3.model.S3Exception; +import software.amazon.awssdk.services.s3.model.ServerSideEncryption; +import software.amazon.awssdk.transfer.s3.S3TransferManager; +import software.amazon.awssdk.transfer.s3.model.CompletedUpload; +import software.amazon.awssdk.transfer.s3.model.Upload; +import software.amazon.awssdk.transfer.s3.model.UploadRequest; class IcebergS3Client implements IcebergStorageClient { private static final Logging logger = new Logging(IcebergS3Client.class); @@ -59,41 +54,23 @@ class IcebergS3Client implements IcebergStorageClient { // expired AWS token error code private static final String EXPIRED_AWS_TOKEN_ERROR_CODE = "ExpiredToken"; + // Multipart upload threshold: 16 MB + private static final long MULTIPART_THRESHOLD_BYTES = 16L * 1024 * 1024; + private boolean isUseS3RegionalUrl = false; - private ClientConfiguration clientConfig = null; + private int maxConnections = 0; + private ProxyConfiguration proxyConfig = null; private String stageRegion = null; private Properties proxyProperties = null; private String stageEndPoint = null; // FIPS endpoint, if needed private boolean isClientSideEncrypted = true; - private AmazonS3 amazonClient = null; + private S3AsyncClient amazonClient = null; private VolumeEncryptionMode volumeEncryptionMode = null; private String encryptionKmsKeyId = null; - // socket factory used by s3 client's http client. - private static SSLConnectionSocketFactory s3ConnectionSocketFactory = null; - - private static SSLConnectionSocketFactory getSSLConnectionSocketFactory() { - if (s3ConnectionSocketFactory == null) { - synchronized (IcebergS3Client.class) { - if (s3ConnectionSocketFactory == null) { - try { - // trust manager is set to null, which will use default ones - // instead of SFTrustManager (which enables ocsp checking) - s3ConnectionSocketFactory = - new IngestSSLConnectionSocketFactory(null, HttpUtil.isSocksProxyDisabled()); - } catch (KeyManagementException | NoSuchAlgorithmException e) { - throw new SSLInitializationException(e.getMessage(), e); - } - } - } - } - - return s3ConnectionSocketFactory; - } - public IcebergS3Client( Map stageCredentials, - ClientConfiguration clientConfig, + int maxConnections, Properties proxyProperties, String stageRegion, String stageEndPoint, @@ -107,7 +84,7 @@ public IcebergS3Client( this.encryptionKmsKeyId = encryptionKmsKeyId; setupSnowflakeS3Client( stageCredentials, - clientConfig, + maxConnections, proxyProperties, stageRegion, stageEndPoint, @@ -116,7 +93,7 @@ public IcebergS3Client( private void setupSnowflakeS3Client( Map stageCredentials, - ClientConfiguration clientConfig, + int maxConnections, Properties proxyProperties, String stageRegion, String stageEndPoint, @@ -125,7 +102,7 @@ private void setupSnowflakeS3Client( // Save the client creation parameters so that we can reuse them, // to reset the AWS client. We won't save the awsCredentials since // we will be refreshing that, every time we reset the AWS client - this.clientConfig = clientConfig; + this.maxConnections = maxConnections; this.stageRegion = stageRegion; this.proxyProperties = proxyProperties; this.stageEndPoint = stageEndPoint; // FIPS endpoint, if needed @@ -137,38 +114,46 @@ private void setupSnowflakeS3Client( String awsToken = (String) stageCredentials.get("AWS_TOKEN"); // initialize aws credentials - AWSCredentials awsCredentials = + AwsCredentials awsCredentials = (awsToken != null) - ? new BasicSessionCredentials(awsID, awsKey, awsToken) - : new BasicAWSCredentials(awsID, awsKey); - - clientConfig.withSignerOverride("AWSS3V4SignerType"); - clientConfig.getApacheHttpClientConfig().setSslSocketFactory(getSSLConnectionSocketFactory()); - setSessionlessProxyForS3(proxyProperties, clientConfig); - AmazonS3Builder amazonS3Builder = - AmazonS3Client.builder() - .withCredentials(new AWSStaticCredentialsProvider(awsCredentials)) - .withClientConfiguration(clientConfig); - - Region region = RegionUtils.getRegion(stageRegion); + ? AwsSessionCredentials.create(awsID, awsKey, awsToken) + : AwsBasicCredentials.create(awsID, awsKey); + + this.proxyConfig = buildProxyConfiguration(proxyProperties); + + NettyNioAsyncHttpClient.Builder httpClientBuilder = + NettyNioAsyncHttpClient.builder().maxConcurrency(maxConnections); + if (this.proxyConfig != null) { + httpClientBuilder.proxyConfiguration(this.proxyConfig); + } + + S3AsyncClientBuilder s3Builder = + S3AsyncClient.builder() + .credentialsProvider(StaticCredentialsProvider.create(awsCredentials)) + .httpClientBuilder(httpClientBuilder) + .forcePathStyle(false) + .multipartEnabled(true) + .multipartConfiguration( + software.amazon.awssdk.services.s3.multipart.MultipartConfiguration.builder() + .thresholdInBytes(MULTIPART_THRESHOLD_BYTES) + .build()); + + Region region = Region.of(stageRegion); if (this.stageEndPoint != null && this.stageEndPoint != "" && this.stageEndPoint != "null") { - amazonS3Builder.withEndpointConfiguration( - new AwsClientBuilder.EndpointConfiguration(this.stageEndPoint, region.getName())); + s3Builder.endpointOverride(URI.create(this.stageEndPoint)); + s3Builder.region(region); } else { - if (region != null) { + if (stageRegion != null) { if (this.isUseS3RegionalUrl) { - String domainSuffixForRegionalUrl = getDomainSuffixForRegionalUrl(region.getName()); - amazonS3Builder.withEndpointConfiguration( - new AwsClientBuilder.EndpointConfiguration( - "s3." + region.getName() + "." + domainSuffixForRegionalUrl, region.getName())); - } else { - amazonS3Builder.withRegion(region.getName()); + String domainSuffixForRegionalUrl = getDomainSuffixForRegionalUrl(stageRegion); + s3Builder.endpointOverride( + URI.create("https://s3." + stageRegion + "." + domainSuffixForRegionalUrl)); } + s3Builder.region(region); } } - // Explicitly force to use virtual address style - amazonS3Builder.withPathStyleAccessEnabled(false); - amazonClient = (AmazonS3) amazonS3Builder.build(); + + amazonClient = s3Builder.build(); } /* Adds digest metadata to the StorageObjectMetadata object */ @@ -241,24 +226,25 @@ public String upload( final long originalContentLength = meta.getContentLength(); final List toClose = new ArrayList<>(); + + IcebergS3ObjectMetadata s3Meta; + if (meta instanceof IcebergS3ObjectMetadata) { + s3Meta = (IcebergS3ObjectMetadata) meta; + } else { + throw new IllegalArgumentException("Unexpected metadata object type"); + } + SFPair uploadStreamInfo = createUploadStream( srcFile, uploadFromStream, inputStream, fileBackedOutputStream, - ((IcebergS3ObjectMetadata) meta).getS3ObjectMetadata(), + s3Meta, originalContentLength, toClose); - ObjectMetadata s3Meta; - if (meta instanceof IcebergS3ObjectMetadata) { - s3Meta = ((IcebergS3ObjectMetadata) meta).getS3ObjectMetadata(); - } else { - throw new IllegalArgumentException("Unexpected metadata object type"); - } - - TransferManager tx = null; + S3TransferManager tx = null; int retryCount = 0; Stopwatch stopwatch = new Stopwatch(); stopwatch.start(); @@ -267,37 +253,51 @@ public String upload( logger.logDebug( "Creating executor service for transfer" + "manager with {} threads", parallelism); + ExecutorService executor = + createDefaultExecutorService("s3-transfer-manager-uploader-", parallelism); + // upload files to s3 - tx = - TransferManagerBuilder.standard() - .withS3Client(amazonClient) - .withExecutorFactory( - new ExecutorFactory() { - @Override - public ExecutorService newExecutor() { - return createDefaultExecutorService( - "s3-transfer-manager-uploader-", parallelism); - } - }) - .build(); - - PutObjectRequest putRequest = - uploadStreamInfo.right - ? new PutObjectRequest( - remoteStorageLocation, destFileName, uploadStreamInfo.left, s3Meta) - : new PutObjectRequest(remoteStorageLocation, destFileName, srcFile); - putRequest.setMetadata(s3Meta); + tx = S3TransferManager.builder().s3Client(amazonClient).executor(executor).build(); + + // Build the PutObjectRequest + PutObjectRequest.Builder putBuilder = + PutObjectRequest.builder() + .bucket(remoteStorageLocation) + .key(destFileName) + .metadata(s3Meta.getRawUserMetadata()) + .contentLength(originalContentLength); if (this.volumeEncryptionMode != null) { this.volumeEncryptionMode.validateKmsKeyId(this.encryptionKmsKeyId); } if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - putRequest.setSSEAwsKeyManagementParams( - new SSEAwsKeyManagementParams(this.encryptionKmsKeyId)); + putBuilder.serverSideEncryption(ServerSideEncryption.AWS_KMS); + putBuilder.ssekmsKeyId(this.encryptionKmsKeyId); + } else if (s3Meta.getSSEAlgorithm() != null) { + // Apply SSE-S3 (AES256) if set in metadata + putBuilder.serverSideEncryption(ServerSideEncryption.AES256); } - final Upload myUpload = tx.upload(putRequest); - myUpload.waitForCompletion(); + PutObjectRequest putRequest = putBuilder.build(); + + // Wrap the stream with BufferedInputStream to work around CipherInputStream bug + InputStream uploadStream = uploadStreamInfo.left; + AsyncRequestBody requestBody; + if (uploadStreamInfo.right) { + requestBody = + AsyncRequestBody.fromInputStream( + new BufferedInputStream(uploadStream), originalContentLength, executor); + } else { + requestBody = AsyncRequestBody.fromFile(srcFile.toPath()); + } + + Upload myUpload = + tx.upload( + UploadRequest.builder() + .putObjectRequest(putRequest) + .requestBody(requestBody) + .build()); + CompletedUpload result = myUpload.completionFuture().join(); stopwatch.stop(); long uploadMillis = stopwatch.elapsedMillis(); @@ -320,9 +320,7 @@ public ExecutorService newExecutor() { uploadMillis, retryCount); } - // -------------------------NEW LOGIC HERE START ----------------------------------------- - return myUpload.waitForUploadResult().getETag(); - // -------------------------NEW LOGIC HERE END ----------------------------------------- + return result.response().eTag(); } catch (Exception ex) { handleS3Exception(ex, ++retryCount, "upload", this); @@ -346,7 +344,7 @@ public ExecutorService newExecutor() { toClose); } finally { if (tx != null) { - tx.shutdownNow(false); + tx.close(); } } } while (retryCount <= getMaxRetries()); @@ -372,10 +370,10 @@ static String getDomainSuffixForRegionalUrl(String regionName) { * @return true if it's a 400 or 404 status code */ public boolean isClientException400Or404(Exception ex) { - if (ex instanceof AmazonServiceException) { - AmazonServiceException asEx = (AmazonServiceException) (ex); - return asEx.getStatusCode() == HttpStatus.SC_NOT_FOUND - || asEx.getStatusCode() == HttpStatus.SC_BAD_REQUEST; + if (ex instanceof SdkServiceException) { + SdkServiceException ssEx = (SdkServiceException) ex; + return ssEx.statusCode() == HttpStatus.SC_NOT_FOUND + || ssEx.statusCode() == HttpStatus.SC_BAD_REQUEST; } return false; } @@ -385,7 +383,7 @@ private SFPair createUploadStream( boolean uploadFromStream, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, - ObjectMetadata meta, + IcebergS3ObjectMetadata meta, long originalContentLength, List toClose) throws SnowflakeSQLException { @@ -406,10 +404,10 @@ private SFPair createUploadStream( if (this.volumeEncryptionMode == null || VolumeEncryptionMode.NONE.equals(this.volumeEncryptionMode) || VolumeEncryptionMode.AWS_SSE_S3.equals(this.volumeEncryptionMode)) { - // Default to SSE-S3 encryption - meta.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); + // Default to SSE-S3 encryption (AES256) + meta.setSSEAlgorithm("AES256"); } else if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - meta.setSSEAlgorithm(SSEAlgorithm.KMS.getAlgorithm()); + meta.setSSEAlgorithm("aws:kms"); } else { throw new IllegalArgumentException( "Unexpected volume encryption mode: " @@ -467,30 +465,39 @@ private static void handleS3Exception( } // Don't retry if max retries has been reached or the error code is 404/400 - if (ex instanceof AmazonClientException) { - logger.logDebug("AmazonClientException: " + ex.getMessage()); + if (ex instanceof SdkException) { + logger.logDebug("SdkException: " + ex.getMessage()); if (retryCount > s3Client.getMaxRetries() || s3Client.isClientException400Or404(ex)) { String extendedRequestId = "none"; - if (ex instanceof AmazonS3Exception) { - AmazonS3Exception ex1 = (AmazonS3Exception) ex; - extendedRequestId = ex1.getExtendedRequestId(); + if (ex instanceof S3Exception) { + S3Exception s3ex = (S3Exception) ex; + extendedRequestId = s3ex.extendedRequestId() != null ? s3ex.extendedRequestId() : "none"; } - if (ex instanceof AmazonServiceException) { - AmazonServiceException ex1 = (AmazonServiceException) ex; + if (ex instanceof SdkServiceException) { + SdkServiceException ex1 = (SdkServiceException) ex; // The AWS credentials might have expired when server returns error 400 and // does not return the ExpiredToken error code. // If session is null we cannot renew the token so throw the exception + String errorCode = "Unknown"; + String errorType = "Unknown"; + if (ex instanceof S3Exception) { + S3Exception s3ex = (S3Exception) ex; + if (s3ex.awsErrorDetails() != null) { + errorCode = s3ex.awsErrorDetails().errorCode(); + errorType = s3ex.awsErrorDetails().errorMessage(); + } + } throw new SnowflakeSQLLoggedException( SqlState.SYSTEM_ERROR, ErrorCode.S3_OPERATION_ERROR.getMessageCode(), ex1, operation, - ex1.getErrorType().toString(), - ex1.getErrorCode(), + errorType, + errorCode, ex1.getMessage(), - ex1.getRequestId(), + ex1.requestId() != null ? ex1.requestId() : "none", extendedRequestId); } else { throw new SnowflakeSQLLoggedException( @@ -525,12 +532,14 @@ private static void handleS3Exception( // If the exception indicates that the AWS token has expired, // we need to refresh our S3 client with the new token - if (ex instanceof AmazonS3Exception) { - AmazonS3Exception s3ex = (AmazonS3Exception) ex; - if (s3ex.getErrorCode().equalsIgnoreCase(EXPIRED_AWS_TOKEN_ERROR_CODE)) { + if (ex instanceof S3Exception) { + S3Exception s3ex = (S3Exception) ex; + if (s3ex.awsErrorDetails() != null + && EXPIRED_AWS_TOKEN_ERROR_CODE.equalsIgnoreCase( + s3ex.awsErrorDetails().errorCode())) { // If session is null we cannot renew the token so throw the ExpiredToken exception throw new SnowflakeSQLException( - s3ex.getErrorCode(), + s3ex.awsErrorDetails().errorCode(), CLOUD_STORAGE_CREDENTIALS_EXPIRED, "S3 credentials have expired"); } @@ -563,14 +572,14 @@ private static void handleS3Exception( } /** - * Copied from JDBC Driver S3HttpUtil to avoid class definition conflicts with AWS SDK + * Builds a ProxyConfiguration from proxy properties, or returns null if no proxy is configured. * * @param proxyProperties Proxy Properties - * @param clientConfig AWS Client Configuration + * @return ProxyConfiguration or null * @throws SnowflakeSQLException Thrown on configuration parsing failures */ - private static void setSessionlessProxyForS3( - Properties proxyProperties, ClientConfiguration clientConfig) throws SnowflakeSQLException { + private static ProxyConfiguration buildProxyConfiguration(Properties proxyProperties) + throws SnowflakeSQLException { if (proxyProperties != null && proxyProperties.size() > 0 && proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()) != null) { @@ -597,33 +606,41 @@ private static void setSessionlessProxyForS3( proxyProperties.getProperty(SFSessionProperty.PROXY_PASSWORD.getPropertyKey()); String nonProxyHosts = proxyProperties.getProperty(SFSessionProperty.NON_PROXY_HOSTS.getPropertyKey()); - String proxyProtocol = - proxyProperties.getProperty(SFSessionProperty.PROXY_PROTOCOL.getPropertyKey()); - Protocol protocolEnum = - isNotEmpty(proxyProtocol) && proxyProtocol.equalsIgnoreCase("https") - ? Protocol.HTTPS - : Protocol.HTTP; - clientConfig.setProxyHost(proxyHost); - clientConfig.setProxyPort(proxyPort); - clientConfig.setNonProxyHosts(nonProxyHosts); - clientConfig.setProxyProtocol(protocolEnum); + + ProxyConfiguration.Builder proxyBuilder = + ProxyConfiguration.builder().host(proxyHost).port(proxyPort); + + if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { + proxyBuilder.username(proxyUser).password(proxyPassword); + } + + if (isNotEmpty(nonProxyHosts)) { + Set nonProxyHostSet = new HashSet<>(); + for (String host : nonProxyHosts.split("\\|")) { + if (!host.isEmpty()) { + nonProxyHostSet.add(host); + } + } + proxyBuilder.nonProxyHosts(nonProxyHostSet); + } + String logMessage = String.format( - "Set sessionless S3 proxy. Host: %s, port: %d, non-proxy hosts: %s, protocol: %s", - proxyHost, proxyPort, nonProxyHosts, proxyProtocol); + "Set sessionless S3 proxy. Host: %s, port: %d, non-proxy hosts: %s", + proxyHost, proxyPort, nonProxyHosts); if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { logMessage = String.format("%s, user: %s with password provided", logMessage, proxyUser); - clientConfig.setProxyUsername(proxyUser); - clientConfig.setProxyPassword(proxyPassword); } - logger.logDebug(logMessage); + + return proxyBuilder.build(); } else { logger.logDebug("Omitting sessionless S3 proxy setup as proxy is disabled"); } } else { logger.logDebug("Omitting sessionless S3 proxy setup"); } + return null; } private static boolean isNotEmpty(final String string) { diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java index 998ccdabd..e54b93fc0 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java @@ -4,63 +4,75 @@ package net.snowflake.ingest.streaming.internal.fileTransferAgent; -import com.amazonaws.services.s3.model.ObjectMetadata; import java.util.Map; +import java.util.TreeMap; /** - * s3 implementation of platform independent StorageObjectMetadata interface, wraps an S3 - * ObjectMetadata class + * S3 implementation of platform independent StorageObjectMetadata interface. Uses plain Map-based + * storage (AWS SDK v2 does not have an ObjectMetadata class; metadata is passed directly on the + * PutObjectRequest builder). * *

It only supports a limited set of metadata properties currently used by the JDBC client * * @author lgiakoumakis */ public class IcebergS3ObjectMetadata implements StorageObjectMetadata { - private ObjectMetadata objectMetadata; + private long contentLength; + private final Map userMetadata; + private String contentEncoding; - IcebergS3ObjectMetadata() { - objectMetadata = new ObjectMetadata(); - } + // SSE algorithm to apply (e.g. "AES256", "aws:kms") + private String sseAlgorithm; - // Construct from an AWS S3 ObjectMetadata object - IcebergS3ObjectMetadata(ObjectMetadata meta) { - objectMetadata = meta; + IcebergS3ObjectMetadata() { + userMetadata = new TreeMap<>(String.CASE_INSENSITIVE_ORDER); } @Override public Map getUserMetadata() { - return StorageClientUtil.createCaseInsensitiveMap(objectMetadata.getUserMetadata()); + return StorageClientUtil.createCaseInsensitiveMap(userMetadata); + } + + /** + * Returns the raw (mutable) user metadata map. Used internally by IcebergS3Client to build the + * PutObjectRequest. + */ + Map getRawUserMetadata() { + return userMetadata; } @Override public long getContentLength() { - return objectMetadata.getContentLength(); + return contentLength; } @Override public void setContentLength(long contentLength) { - objectMetadata.setContentLength(contentLength); + this.contentLength = contentLength; } @Override public void addUserMetadata(String key, String value) { - objectMetadata.addUserMetadata(key, value); + userMetadata.put(key, value); } @Override public void setContentEncoding(String encoding) { - objectMetadata.setContentEncoding(encoding); + this.contentEncoding = encoding; } @Override public String getContentEncoding() { - return objectMetadata.getContentEncoding(); + return contentEncoding; } - /** - * @return Returns the encapsulated AWS S3 metadata object - */ - ObjectMetadata getS3ObjectMetadata() { - return objectMetadata; + /** Sets the server-side encryption algorithm (e.g. "AES256", "aws:kms"). */ + void setSSEAlgorithm(String algorithm) { + this.sseAlgorithm = algorithm; + } + + /** Returns the server-side encryption algorithm, or null if not set. */ + String getSSEAlgorithm() { + return sseAlgorithm; } } diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java index a2ab6db32..81f881b09 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java @@ -1,10 +1,8 @@ package net.snowflake.ingest.streaming.internal.fileTransferAgent; -import com.amazonaws.ClientConfiguration; import java.util.Map; import java.util.Properties; import net.snowflake.ingest.streaming.internal.VolumeEncryptionMode; -import net.snowflake.ingest.utils.HttpUtil; import net.snowflake.ingest.utils.Logging; /** @@ -108,35 +106,17 @@ private IcebergStorageClient createS3Client( VolumeEncryptionMode volumeEncryptionMode, String encryptionKmsKeyId) throws SnowflakeSQLException { - final int S3_TRANSFER_MAX_RETRIES = 3; - IcebergS3Client s3Client; - ClientConfiguration clientConfig = new ClientConfiguration(); - clientConfig.setMaxConnections(parallel + 1); - clientConfig.setMaxErrorRetry(S3_TRANSFER_MAX_RETRIES); - clientConfig.setDisableSocketProxy(HttpUtil.isSocksProxyDisabled()); - - // If proxy is set via connection properties or JVM settings these will be overridden later. - // This is to prevent the aws client builder from reading proxy environment variables. - clientConfig.setProxyHost(""); - clientConfig.setProxyPort(0); - clientConfig.setProxyUsername(""); - clientConfig.setProxyPassword(""); - - logger.logDebug( - "S3 client configuration: maxConnection: {}, connectionTimeout: {}, " - + "socketTimeout: {}, maxErrorRetry: {}", - clientConfig.getMaxConnections(), - clientConfig.getConnectionTimeout(), - clientConfig.getSocketTimeout(), - clientConfig.getMaxErrorRetry()); + int maxConnections = parallel + 1; + + logger.logDebug("S3 client configuration: maxConnections: {}", maxConnections); try { s3Client = new IcebergS3Client( stageCredentials, - clientConfig, + maxConnections, proxyProperties, stageRegion, stageEndPoint, From 9cd1d3894226562826a8afd10b26d2bf9c794800 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 8 Apr 2026 23:43:34 +0000 Subject: [PATCH 03/10] Add eventstream to ADDITIONAL_LICENSES_MAP for license processing software.amazon.eventstream:eventstream (Apache 2.0) is a transitive dependency of AWS SDK v2 that doesn't bundle a license file in its JAR. Co-Authored-By: Claude Opus 4.6 --- scripts/process_licenses.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/process_licenses.py b/scripts/process_licenses.py index 718aaba88..a685d203c 100644 --- a/scripts/process_licenses.py +++ b/scripts/process_licenses.py @@ -80,6 +80,7 @@ "org.roaringbitmap:shims": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-core": APACHE_LICENSE, "software.amazon.ion:ion-java": APACHE_LICENSE, + "software.amazon.eventstream:eventstream": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-kms": APACHE_LICENSE, "com.amazonaws:jmespath-java": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-s3": APACHE_LICENSE, From 4a56304a05aa328ce3ced041a0556df9b15cfc48 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Mon, 13 Apr 2026 19:59:37 +0000 Subject: [PATCH 04/10] Fix IcebergS3Client to match JDBC's AWS SDK v2 patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add ClientConfiguration inner class (maxConnections, maxErrorRetry, connectionTimeout, socketTimeout) matching JDBC's SnowflakeS3Client - Configure Netty HTTP timeouts: connectionAcquisitionTimeout(60s), connectionTimeout, readTimeout, writeTimeout - Fix exception handler to check ex.getCause() for SdkException (CompletableFuture.join() wraps in CompletionException) - Add CompletionException handling for non-SDK async failures - Fix string comparison bug: != "" → .isEmpty(), != "null" → .equals() - Add getS3PutObjectRequest() to IcebergS3ObjectMetadata with CRC32 checksum (matching JDBC's S3ObjectMetadata) - Restore ClientConfiguration creation in IcebergStorageClientFactory with timeouts from JdbcHttpUtil - Fix aws-crt exclusion groupId: software.amazon.awssdk → software.amazon.awssdk.crt Co-Authored-By: Claude Opus 4.6 (1M context) --- pom.xml | 6 +- .../fileTransferAgent/IcebergS3Client.java | 89 ++++++++++++++----- .../IcebergS3ObjectMetadata.java | 14 +++ .../IcebergStorageClientFactory.java | 21 ++++- 4 files changed, 99 insertions(+), 31 deletions(-) diff --git a/pom.xml b/pom.xml index e2ceae246..1f6aaaf23 100644 --- a/pom.xml +++ b/pom.xml @@ -1014,7 +1014,7 @@ http-auth-aws - software.amazon.awssdk + software.amazon.awssdk.crt aws-crt @@ -1028,7 +1028,7 @@ s3 - software.amazon.awssdk + software.amazon.awssdk.crt aws-crt @@ -1038,7 +1038,7 @@ s3-transfer-manager - software.amazon.awssdk + software.amazon.awssdk.crt aws-crt diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java index 1db324774..e328ebd2f 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java @@ -13,12 +13,14 @@ import java.net.SocketTimeoutException; import java.net.URI; import java.security.InvalidKeyException; +import java.time.Duration; import java.util.ArrayList; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.Set; +import java.util.concurrent.CompletionException; import java.util.concurrent.ExecutorService; import net.snowflake.ingest.streaming.internal.VolumeEncryptionMode; import net.snowflake.ingest.utils.Logging; @@ -58,7 +60,7 @@ class IcebergS3Client implements IcebergStorageClient { private static final long MULTIPART_THRESHOLD_BYTES = 16L * 1024 * 1024; private boolean isUseS3RegionalUrl = false; - private int maxConnections = 0; + private ClientConfiguration clientConfig = null; private ProxyConfiguration proxyConfig = null; private String stageRegion = null; private Properties proxyProperties = null; @@ -70,7 +72,7 @@ class IcebergS3Client implements IcebergStorageClient { public IcebergS3Client( Map stageCredentials, - int maxConnections, + ClientConfiguration clientConfig, Properties proxyProperties, String stageRegion, String stageEndPoint, @@ -84,7 +86,7 @@ public IcebergS3Client( this.encryptionKmsKeyId = encryptionKmsKeyId; setupSnowflakeS3Client( stageCredentials, - maxConnections, + clientConfig, proxyProperties, stageRegion, stageEndPoint, @@ -93,7 +95,7 @@ public IcebergS3Client( private void setupSnowflakeS3Client( Map stageCredentials, - int maxConnections, + ClientConfiguration clientConfig, Properties proxyProperties, String stageRegion, String stageEndPoint, @@ -102,7 +104,7 @@ private void setupSnowflakeS3Client( // Save the client creation parameters so that we can reuse them, // to reset the AWS client. We won't save the awsCredentials since // we will be refreshing that, every time we reset the AWS client - this.maxConnections = maxConnections; + this.clientConfig = clientConfig; this.stageRegion = stageRegion; this.proxyProperties = proxyProperties; this.stageEndPoint = stageEndPoint; // FIPS endpoint, if needed @@ -122,7 +124,12 @@ private void setupSnowflakeS3Client( this.proxyConfig = buildProxyConfiguration(proxyProperties); NettyNioAsyncHttpClient.Builder httpClientBuilder = - NettyNioAsyncHttpClient.builder().maxConcurrency(maxConnections); + NettyNioAsyncHttpClient.builder() + .maxConcurrency(clientConfig.getMaxConnections()) + .connectionAcquisitionTimeout(Duration.ofSeconds(60)) + .connectionTimeout(Duration.ofMillis(clientConfig.getConnectionTimeout())) + .readTimeout(Duration.ofMillis(clientConfig.getSocketTimeout())) + .writeTimeout(Duration.ofMillis(clientConfig.getSocketTimeout())); if (this.proxyConfig != null) { httpClientBuilder.proxyConfiguration(this.proxyConfig); } @@ -139,7 +146,9 @@ private void setupSnowflakeS3Client( .build()); Region region = Region.of(stageRegion); - if (this.stageEndPoint != null && this.stageEndPoint != "" && this.stageEndPoint != "null") { + if (this.stageEndPoint != null + && !this.stageEndPoint.isEmpty() + && !"null".equals(this.stageEndPoint)) { s3Builder.endpointOverride(URI.create(this.stageEndPoint)); s3Builder.region(region); } else { @@ -259,13 +268,11 @@ public String upload( // upload files to s3 tx = S3TransferManager.builder().s3Client(amazonClient).executor(executor).build(); - // Build the PutObjectRequest + // Build the PutObjectRequest from metadata (includes CRC32 checksum) PutObjectRequest.Builder putBuilder = - PutObjectRequest.builder() + s3Meta.getS3PutObjectRequest().toBuilder() .bucket(remoteStorageLocation) - .key(destFileName) - .metadata(s3Meta.getRawUserMetadata()) - .contentLength(originalContentLength); + .key(destFileName); if (this.volumeEncryptionMode != null) { this.volumeEncryptionMode.validateKmsKeyId(this.encryptionKmsKeyId); @@ -369,7 +376,7 @@ static String getDomainSuffixForRegionalUrl(String regionName) { * @param ex exception * @return true if it's a 400 or 404 status code */ - public boolean isClientException400Or404(Exception ex) { + public boolean isClientException400Or404(Throwable ex) { if (ex instanceof SdkServiceException) { SdkServiceException ssEx = (SdkServiceException) ex; return ssEx.statusCode() == HttpStatus.SC_NOT_FOUND @@ -465,25 +472,27 @@ private static void handleS3Exception( } // Don't retry if max retries has been reached or the error code is 404/400 - if (ex instanceof SdkException) { + // Check ex.getCause() because CompletableFuture.join() wraps in CompletionException + Throwable cause = ex.getCause(); + if (cause instanceof SdkException) { logger.logDebug("SdkException: " + ex.getMessage()); - if (retryCount > s3Client.getMaxRetries() || s3Client.isClientException400Or404(ex)) { + if (retryCount > s3Client.getMaxRetries() || s3Client.isClientException400Or404(cause)) { String extendedRequestId = "none"; - if (ex instanceof S3Exception) { - S3Exception s3ex = (S3Exception) ex; + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; extendedRequestId = s3ex.extendedRequestId() != null ? s3ex.extendedRequestId() : "none"; } - if (ex instanceof SdkServiceException) { - SdkServiceException ex1 = (SdkServiceException) ex; + if (cause instanceof SdkServiceException) { + SdkServiceException ex1 = (SdkServiceException) cause; // The AWS credentials might have expired when server returns error 400 and // does not return the ExpiredToken error code. // If session is null we cannot renew the token so throw the exception String errorCode = "Unknown"; String errorType = "Unknown"; - if (ex instanceof S3Exception) { - S3Exception s3ex = (S3Exception) ex; + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; if (s3ex.awsErrorDetails() != null) { errorCode = s3ex.awsErrorDetails().errorCode(); errorType = s3ex.awsErrorDetails().errorMessage(); @@ -532,8 +541,8 @@ private static void handleS3Exception( // If the exception indicates that the AWS token has expired, // we need to refresh our S3 client with the new token - if (ex instanceof S3Exception) { - S3Exception s3ex = (S3Exception) ex; + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; if (s3ex.awsErrorDetails() != null && EXPIRED_AWS_TOKEN_ERROR_CODE.equalsIgnoreCase( s3ex.awsErrorDetails().errorCode())) { @@ -547,7 +556,8 @@ private static void handleS3Exception( } } else { if (ex instanceof InterruptedException - || getRootCause(ex) instanceof SocketTimeoutException) { + || getRootCause(ex) instanceof SocketTimeoutException + || ex instanceof CompletionException) { if (retryCount > s3Client.getMaxRetries()) { throw new SnowflakeSQLLoggedException( SqlState.SYSTEM_ERROR, @@ -646,4 +656,35 @@ private static ProxyConfiguration buildProxyConfiguration(Properties proxyProper private static boolean isNotEmpty(final String string) { return string != null && !string.isEmpty(); } + + public static class ClientConfiguration { + private final int maxConnections; + private final int maxErrorRetry; + private final int connectionTimeout; + private final int socketTimeout; + + public ClientConfiguration( + int maxConnections, int maxErrorRetry, int connectionTimeout, int socketTimeout) { + this.maxConnections = maxConnections; + this.maxErrorRetry = maxErrorRetry; + this.connectionTimeout = connectionTimeout; + this.socketTimeout = socketTimeout; + } + + public int getMaxConnections() { + return maxConnections; + } + + public int getMaxErrorRetry() { + return maxErrorRetry; + } + + public int getConnectionTimeout() { + return connectionTimeout; + } + + public int getSocketTimeout() { + return socketTimeout; + } + } } diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java index e54b93fc0..b8cea9503 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java @@ -6,6 +6,8 @@ import java.util.Map; import java.util.TreeMap; +import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; /** * S3 implementation of platform independent StorageObjectMetadata interface. Uses plain Map-based @@ -75,4 +77,16 @@ void setSSEAlgorithm(String algorithm) { String getSSEAlgorithm() { return sseAlgorithm; } + + /** + * @return Returns the encapsulated AWS S3 PutObjectRequest + */ + PutObjectRequest getS3PutObjectRequest() { + return PutObjectRequest.builder() + .metadata(userMetadata) + .contentLength(contentLength) + .contentEncoding(contentEncoding) + .checksumAlgorithm(ChecksumAlgorithm.CRC32) + .build(); + } } diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java index 81f881b09..1c3bafa73 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java @@ -106,17 +106,30 @@ private IcebergStorageClient createS3Client( VolumeEncryptionMode volumeEncryptionMode, String encryptionKmsKeyId) throws SnowflakeSQLException { - IcebergS3Client s3Client; + final int S3_TRANSFER_MAX_RETRIES = 3; - int maxConnections = parallel + 1; + IcebergS3Client s3Client; - logger.logDebug("S3 client configuration: maxConnections: {}", maxConnections); + IcebergS3Client.ClientConfiguration clientConfig = + new IcebergS3Client.ClientConfiguration( + parallel + 1, + S3_TRANSFER_MAX_RETRIES, + (int) JdbcHttpUtil.getConnectionTimeout().toMillis(), + (int) JdbcHttpUtil.getSocketTimeout().toMillis()); + + logger.logDebug( + "S3 client configuration: maxConnection: {}, connectionTimeout: {}, " + + "socketTimeout: {}, maxErrorRetry: {}", + clientConfig.getMaxConnections(), + clientConfig.getConnectionTimeout(), + clientConfig.getSocketTimeout(), + clientConfig.getMaxErrorRetry()); try { s3Client = new IcebergS3Client( stageCredentials, - maxConnections, + clientConfig, proxyProperties, stageRegion, stageEndPoint, From 36b5ffa246e32725066b4368a61cf43e9297323a Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Mon, 13 Apr 2026 20:20:19 +0000 Subject: [PATCH 05/10] Restore proxy protocol and env variable isolation in S3 proxy config - Read PROXY_PROTOCOL property and set .scheme() on ProxyConfiguration (HTTP vs HTTPS, matching JDBC's CloudStorageProxyFactory) - Add .useEnvironmentVariableValues(false) and .useSystemPropertyValues(false) to prevent Netty from reading HTTP_PROXY/HTTPS_PROXY env vars (v2 equivalent of the old v1 pattern of setting empty proxy host/port/user/password) - Add protocol to proxy log message Co-Authored-By: Claude Opus 4.6 (1M context) --- .../fileTransferAgent/IcebergS3Client.java | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java index e328ebd2f..da6757e5f 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java @@ -616,9 +616,18 @@ private static ProxyConfiguration buildProxyConfiguration(Properties proxyProper proxyProperties.getProperty(SFSessionProperty.PROXY_PASSWORD.getPropertyKey()); String nonProxyHosts = proxyProperties.getProperty(SFSessionProperty.NON_PROXY_HOSTS.getPropertyKey()); + String proxyProtocol = + proxyProperties.getProperty(SFSessionProperty.PROXY_PROTOCOL.getPropertyKey()); + String scheme = + isNotEmpty(proxyProtocol) && proxyProtocol.equalsIgnoreCase("https") ? "https" : "http"; ProxyConfiguration.Builder proxyBuilder = - ProxyConfiguration.builder().host(proxyHost).port(proxyPort); + ProxyConfiguration.builder() + .scheme(scheme) + .host(proxyHost) + .port(proxyPort) + .useEnvironmentVariableValues(false) + .useSystemPropertyValues(false); if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { proxyBuilder.username(proxyUser).password(proxyPassword); @@ -636,8 +645,8 @@ private static ProxyConfiguration buildProxyConfiguration(Properties proxyProper String logMessage = String.format( - "Set sessionless S3 proxy. Host: %s, port: %d, non-proxy hosts: %s", - proxyHost, proxyPort, nonProxyHosts); + "Set sessionless S3 proxy. Host: %s, port: %d, protocol: %s, non-proxy hosts: %s", + proxyHost, proxyPort, scheme, nonProxyHosts); if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { logMessage = String.format("%s, user: %s with password provided", logMessage, proxyUser); } From d41993e698113885bf88b1f468e36994b35640e4 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Mon, 13 Apr 2026 21:02:37 +0000 Subject: [PATCH 06/10] Use v2 SSE constants and update migration plan - Use ServerSideEncryption.fromValue() instead of hardcoded SSE check - Use ServerSideEncryption.AES256/AWS_KMS.toString() for algorithm strings - Update migration plan to reflect actual PR structure and lessons learned: Steps 1+2 merged, ClientConfiguration pattern, CRC32 checksum, CompletionException unwrapping, proxy protocol, execution.interceptors Co-Authored-By: Claude Opus 4.6 (1M context) --- .plans/AWS_SDK_V2_MIGRATION_PLAN.md | 95 +++++++++++-------- .../fileTransferAgent/IcebergS3Client.java | 11 +-- 2 files changed, 62 insertions(+), 44 deletions(-) diff --git a/.plans/AWS_SDK_V2_MIGRATION_PLAN.md b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md index e395d76da..9ef735334 100644 --- a/.plans/AWS_SDK_V2_MIGRATION_PLAN.md +++ b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md @@ -135,44 +135,51 @@ File: `snowflake-jdbc/parent-pom.xml` ## Migration Steps -### Step 1: Update dependencies +### Step 1+2: Add v2 deps + Migrate IcebergS3Client (PR #1149) — DONE -Replace v1 deps with v2 in `pom.xml`: +Add v2 deps **alongside** v1 (v1 still used by SnowflakeS3Client, GCS clients): ```xml - + com.amazonaws:aws-java-sdk-core:1.12.655 com.amazonaws:aws-java-sdk-kms:1.12.655 com.amazonaws:aws-java-sdk-s3:1.12.655 - + +software.amazon.awssdk:bom:2.37.5 (type=pom, scope=import) software.amazon.awssdk:s3 software.amazon.awssdk:s3-transfer-manager software.amazon.awssdk:netty-nio-client software.amazon.awssdk:auth -software.amazon.awssdk:regions - -software.amazon.awssdk:bom:2.37.5 (type=pom, scope=import) +software.amazon.awssdk:http-auth-aws ``` -Exclude `aws-crt` from all v2 deps. Update shade rules: -`com.amazonaws` → `software.amazon.awssdk`. - -### Step 2: Migrate IcebergS3Client (lowest risk) +Exclude `aws-crt` (groupId `software.amazon.awssdk.crt`) from `s3`, `s3-transfer-manager`, +`http-auth-aws`. Add shade rules for `software.amazon.awssdk`, `software.amazon.eventstream`, +`org.reactivestreams`. -Start here because it has NO client-side encryption (avoids Risk 1). - -**Changes:** +**IcebergS3Client changes (lowest risk — no client-side encryption):** - `AmazonS3ClientBuilder` → `S3AsyncClient.builder()` - `BasicAWSCredentials`/`BasicSessionCredentials` → `AwsBasicCredentials`/`AwsSessionCredentials` - `AWSStaticCredentialsProvider` → `StaticCredentialsProvider` -- `ClientConfiguration` → `NettyNioAsyncHttpClient.builder()` with `ProxyConfiguration` +- v1 `ClientConfiguration` → v2 `ClientConfiguration` inner class (matching JDBC's + `SnowflakeS3Client.ClientConfiguration`) with `maxConnections`, `maxErrorRetry`, + `connectionTimeout`, `socketTimeout` +- `NettyNioAsyncHttpClient` configured with `connectionAcquisitionTimeout(60s)`, + `connectionTimeout`, `readTimeout`, `writeTimeout` (matching JDBC) - `TransferManager` → `S3TransferManager` -- `ObjectMetadata` → `PutObjectRequest.builder()` metadata -- `SSEAwsKeyManagementParams` → `ServerSideEncryption.AWS_KMS` + `ssekmsKeyId()` +- `ObjectMetadata` → `IcebergS3ObjectMetadata` with `getS3PutObjectRequest()` builder + (matching JDBC's `S3ObjectMetadata`) including `ChecksumAlgorithm.CRC32` +- `SSEAwsKeyManagementParams` → `ServerSideEncryption.fromValue()` + `ssekmsKeyId()` - Set multipart threshold to 16MB (Risk 2) - Null-check `awsErrorDetails()` in error handler (Risk 3) +- Exception handler checks `ex.getCause()` for `SdkException` — `CompletableFuture.join()` + wraps in `CompletionException` (matching JDBC) +- `CompletionException` handling for non-SDK async failures - `Region`/`RegionUtils` → `software.amazon.awssdk.regions.Region` +- Proxy: `ProxyConfiguration` with `.scheme()` (proxy protocol), `.useEnvironmentVariableValues(false)`, + `.useSystemPropertyValues(false)` (matching JDBC's `CloudStorageProxyFactory`) +- `SSLConnectionSocketFactory` removed (v2 Netty handles TLS natively) ### Step 3: Migrate SnowflakeS3Client (high risk — encryption) @@ -205,15 +212,21 @@ Same as Step 2, plus: - `StorageObjectSummary`/`StorageObjectSummaryCollection` → adapt to v2 types - `StorageProviderException` → catch `SdkException` (v2 base exception) -### Step 6: Update shade plugin +### Step 6: Update shade plugin + remove v1 deps -```xml - -com.amazonaws - -software.amazon.awssdk -${shadeBase}.software.amazon.awssdk -``` +Shade rules for `software.amazon.awssdk`, `software.amazon.eventstream`, +`org.reactivestreams` already added in Step 1. Remaining work: + +- Remove v1 `com.amazonaws` deps and shade rules +- **Patch `execution.interceptors` after shading** — JDBC patches + `software/amazon/awssdk/global/handlers/execution.interceptors` file contents + because the shade plugin relocates the file path but not the class names inside + it. We need the same antrun `replace` task: + ```xml + + ``` ### Step 7: Update tests @@ -227,14 +240,13 @@ Update all test files that reference v1 types. Verify: ## PR Strategy -| PR | Content | Risk Level | -|---|---|---| -| PR 1 | Add v2 deps alongside v1, update shade rules | Low | -| PR 2 | Migrate `IcebergS3Client` + `IcebergStorageClientFactory` | Medium | -| PR 3 | Migrate `SnowflakeS3Client` + `StorageClientFactory` (encryption path) | **High** | -| PR 4 | Migrate `AwsSdkGCPSigner` + `GCSAccessStrategyAwsSdk` | Medium | -| PR 5 | Migrate support classes (metadata, iterators, interceptor, proxy) | Low | -| PR 6 | Remove v1 deps, clean up shade rules | Low | +| PR | Content | Risk Level | Status | +|---|---|---|---| +| PR 1 (#1149) | Add v2 deps + migrate `IcebergS3Client` | Medium | **Open** | +| PR 2 | Migrate `SnowflakeS3Client` + `StorageClientFactory` (encryption path) | **High** | | +| PR 3 | Migrate `AwsSdkGCPSigner` + `GCSAccessStrategyAwsSdk` | Medium | | +| PR 4 | Migrate support classes (metadata, iterators, interceptor, proxy) | Low | | +| PR 5 | Remove v1 deps, shade cleanup, `execution.interceptors` patching | Low | | --- @@ -255,14 +267,21 @@ Key JDBC files to reference during migration (at `snowflakedb/snowflake-jdbc` ma ## Verification -- [ ] `mvn compiler:compile` passes +### Step 1+2 (IcebergS3Client) +- [x] `mvn compiler:compile` passes +- [x] `aws-crt` properly excluded from dependency tree - [ ] All unit tests pass -- [ ] Integration tests pass on S3, Azure, GCS -- [ ] Encryption round-trip verified (upload encrypted → download → decrypt → matches original) +- [ ] Integration tests pass on S3 (Iceberg path) +- [ ] SSE-S3 uploads (Iceberg path) work +- [ ] SSE-KMS uploads (Iceberg path) work - [ ] Multipart uploads > 16MB work correctly - [ ] Uploads < 16MB use single-part (not multipart) -- [ ] GCS uploads via S3-compatible API work -- [ ] SSE-KMS uploads (Iceberg path) work - [ ] Proxy configuration works + +### Full migration (after all PRs) +- [ ] Integration tests pass on S3, Azure, GCS +- [ ] Encryption round-trip verified (upload encrypted → download → decrypt → matches original) +- [ ] GCS uploads via S3-compatible API work - [ ] Shaded jar contains no `com.amazonaws` classes +- [ ] `execution.interceptors` patched correctly in shaded jar - [ ] e2e-jar-test passes (shaded + unshaded) diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java index da6757e5f..678e3de27 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java @@ -277,12 +277,11 @@ public String upload( if (this.volumeEncryptionMode != null) { this.volumeEncryptionMode.validateKmsKeyId(this.encryptionKmsKeyId); } + if (s3Meta.getSSEAlgorithm() != null) { + putBuilder.serverSideEncryption(ServerSideEncryption.fromValue(s3Meta.getSSEAlgorithm())); + } if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - putBuilder.serverSideEncryption(ServerSideEncryption.AWS_KMS); putBuilder.ssekmsKeyId(this.encryptionKmsKeyId); - } else if (s3Meta.getSSEAlgorithm() != null) { - // Apply SSE-S3 (AES256) if set in metadata - putBuilder.serverSideEncryption(ServerSideEncryption.AES256); } PutObjectRequest putRequest = putBuilder.build(); @@ -412,9 +411,9 @@ private SFPair createUploadStream( || VolumeEncryptionMode.NONE.equals(this.volumeEncryptionMode) || VolumeEncryptionMode.AWS_SSE_S3.equals(this.volumeEncryptionMode)) { // Default to SSE-S3 encryption (AES256) - meta.setSSEAlgorithm("AES256"); + meta.setSSEAlgorithm(ServerSideEncryption.AES256.toString()); } else if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - meta.setSSEAlgorithm("aws:kms"); + meta.setSSEAlgorithm(ServerSideEncryption.AWS_KMS.toString()); } else { throw new IllegalArgumentException( "Unexpected volume encryption mode: " From 36d7cb6621625c002314cf61f46ab61363bbfaab Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Tue, 14 Apr 2026 21:52:06 +0000 Subject: [PATCH 07/10] Add MIT-0 license (reactive-streams) to allowed licenses The reactive-streams library (transitive dep of AWS SDK v2) uses MIT-0 which wasn't in the license whitelist, causing the license-maven-plugin check to fail. Co-Authored-By: Claude Opus 4.6 (1M context) --- pom.xml | 2 +- scripts/process_licenses.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 1f6aaaf23..f79ea44ed 100644 --- a/pom.xml +++ b/pom.xml @@ -1430,7 +1430,7 @@ |Apache 2 BSD 2-Clause License |The BSD License |BSD - The MIT License|MIT License|MIT license + The MIT License|MIT License|MIT license|MIT-0 3-Clause BSD License|BSD-3-Clause The Go License|Go License diff --git a/scripts/process_licenses.py b/scripts/process_licenses.py index a685d203c..3ae3ddb13 100644 --- a/scripts/process_licenses.py +++ b/scripts/process_licenses.py @@ -81,6 +81,7 @@ "com.amazonaws:aws-java-sdk-core": APACHE_LICENSE, "software.amazon.ion:ion-java": APACHE_LICENSE, "software.amazon.eventstream:eventstream": APACHE_LICENSE, + "org.reactivestreams:reactive-streams": MIT_LICENSE, "com.amazonaws:aws-java-sdk-kms": APACHE_LICENSE, "com.amazonaws:jmespath-java": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-s3": APACHE_LICENSE, From 4e0f0c605451d4c1c0088d2cfa5ee9a0f1de50b5 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 15 Apr 2026 01:05:10 +0000 Subject: [PATCH 08/10] Add netty-transport-classes-epoll to ADDITIONAL_LICENSES_MAP New transitive dependency from AWS SDK v2's netty-nio-client. The JAR doesn't ship a license file, causing process_licenses.py to fail. Co-Authored-By: Claude Opus 4.6 (1M context) --- scripts/process_licenses.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/process_licenses.py b/scripts/process_licenses.py index 3ae3ddb13..441abca47 100644 --- a/scripts/process_licenses.py +++ b/scripts/process_licenses.py @@ -63,6 +63,7 @@ "io.netty:netty-handler-proxy": APACHE_LICENSE, "io.netty:netty-resolver": APACHE_LICENSE, "io.netty:netty-transport": APACHE_LICENSE, + "io.netty:netty-transport-classes-epoll": APACHE_LICENSE, "io.netty:netty-transport-native-unix-common": APACHE_LICENSE, "com.google.re2j:re2j": GO_LICENSE_ALT, "com.google.protobuf:protobuf-java": BSD_3_CLAUSE_LICENSE, From d149a1268bda463fc570cf39b18ec125ab34e942 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 15 Apr 2026 17:40:17 +0000 Subject: [PATCH 09/10] Fix dependency analysis: declare used transitive deps, remove unused MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add aws-core, http-client-spi, regions, sdk-core as direct deps (were used undeclared transitives) - Remove http-auth-aws (unused — no direct imports) - Fixes maven-dependency-plugin:analyze-only CI check Co-Authored-By: Claude Opus 4.6 (1M context) --- pom.xml | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/pom.xml b/pom.xml index f79ea44ed..a3691b26a 100644 --- a/pom.xml +++ b/pom.xml @@ -1011,18 +1011,20 @@ software.amazon.awssdk - http-auth-aws - - - software.amazon.awssdk.crt - aws-crt - - + aws-core + + + software.amazon.awssdk + http-client-spi software.amazon.awssdk netty-nio-client + + software.amazon.awssdk + regions + software.amazon.awssdk s3 @@ -1043,6 +1045,10 @@ + + software.amazon.awssdk + sdk-core + com.github.luben zstd-jni From 5e3d4f204690972de2ab84982ce5e3eb01f38c48 Mon Sep 17 00:00:00 2001 From: Gibbs Geng Date: Wed, 15 Apr 2026 20:30:02 +0000 Subject: [PATCH 10/10] Exclude leaked VersionInfo.java from shaded jar AWS SDK v2 sdk-core includes a VersionInfo.java source template at the jar root. The shade plugin doesn't relocate .java files, so it leaked into the shaded jar as a bare class outside the snowflake namespace, failing the check-shaded-content CI check. Co-Authored-By: Claude Opus 4.6 (1M context) --- pom.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/pom.xml b/pom.xml index a3691b26a..dd23efb90 100644 --- a/pom.xml +++ b/pom.xml @@ -1801,6 +1801,7 @@ LICENSE NOTICE iceberg-build.properties + VersionInfo.java google/protobuf/**/*.proto THIRD-PARTY.txt