diff --git a/.plans/AWS_SDK_V2_MIGRATION_PLAN.md b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md new file mode 100644 index 000000000..9ef735334 --- /dev/null +++ b/.plans/AWS_SDK_V2_MIGRATION_PLAN.md @@ -0,0 +1,287 @@ +# Plan: Migrate from AWS SDK v1 to AWS SDK v2 + +## Background + +The ingest SDK uses AWS SDK v1 (`com.amazonaws:1.12.655`) for S3 operations, GCS +access (via S3-compatible API), and HTTP proxy configuration. AWS SDK v1 is +deprecated and will reach end-of-support in 2025. + +The JDBC driver (`snowflake-jdbc`) has **already completed** this migration to +AWS SDK v2 (`software.amazon.awssdk:2.37.5`). We can reference their +implementation and learn from their bug fixes. + +--- + +## Scope + +**19 files** import `com.amazonaws.*`, all in `fileTransferAgent/`: + +| Category | Files | v2 Equivalent | +|---|---|---| +| S3 client (non-Iceberg) | `SnowflakeS3Client` | `S3AsyncClient` + `S3TransferManager` | +| S3 client (Iceberg) | `IcebergS3Client` | `S3AsyncClient` + `S3TransferManager` | +| GCS via AWS SDK | `AwsSdkGCPSigner`, `GCSAccessStrategyAwsSdk` | v2 `Signer` interface | +| Proxy/HTTP config | `S3HttpUtil`, `JdbcHttpUtil` | `ProxyConfiguration` + `ClientOverrideConfiguration` | +| Metadata wrappers | `S3ObjectMetadata`, `IcebergS3ObjectMetadata` | Direct `HeadObjectResponse` | +| Factories | `StorageClientFactory`, `IcebergStorageClientFactory` | Updated builders | +| Iterators | `S3ObjectSummariesIterator`, `StorageObjectSummary*` | `ListObjectsV2Response` | +| HTTP interceptor | `HeaderCustomizerHttpRequestInterceptor` | `ExecutionInterceptor` | +| Route planner | `SnowflakeMutableProxyRoutePlanner` | `ProxyConfiguration` | +| Trust manager | `SFTrustManager` | Minimal changes | + +--- + +## Known Risks (from JDBC's migration experience) + +### RISK 1: CipherInputStream + mark/reset — CRITICAL + +**Problem:** `CipherInputStream` does not support `mark()/reset()`. AWS SDK v2's +`AsyncRequestBody.fromInputStream()` may retry failed uploads by resetting the +stream. If the stream is a `CipherInputStream`, this causes silent data corruption +(~1% of uploads produce unreadable files). + +**Applies to us?** **YES — CRITICAL.** Our `EncryptionProvider.encrypt()` returns +a `CipherInputStream` (line 191), which is passed directly to TransferManager. +Same pattern that caused JDBC's bug. + +**How JDBC fixed it (PR #2502):** In `SnowflakeS3Client.upload()`, they wrap +the stream with `BufferedInputStream` before passing to `AsyncRequestBody`: +```java +AsyncRequestBody.fromInputStream( + new BufferedInputStream(uploadStreamInfo.left), + contentLength, executorService) +``` +`BufferedInputStream` supports mark/reset, so SDK retries work correctly. +File: `snowflake-jdbc/internal/jdbc/cloud/storage/SnowflakeS3Client.java` + +**Fix:** Same — wrap `CipherInputStream` in `BufferedInputStream` at upload time. + +**Affected files:** `SnowflakeS3Client.upload()` (non-Iceberg 128-bit encryption +path only). `IcebergS3Client` is NOT affected (no client-side encryption). + +### RISK 2: Multipart threshold change — MEDIUM + +**Problem:** AWS SDK v1 TransferManager default multipart threshold was 16MB. +SDK v2's `S3AsyncClient` with `multipartEnabled(true)` uses 8MB. Files 8-16MB +switch from single-part to multipart, adding extra API round trips. + +**Applies to us?** **YES.** Blobs can be up to 1 GB and the 8-16MB range is +hit during normal operation. + +**How JDBC fixed it (PR #2526):** Explicitly set 16MB threshold on the +`S3AsyncClient` builder: +```java +.multipartConfiguration( + MultipartConfiguration.builder() + .thresholdInBytes(16L * 1024 * 1024) + .build()) +``` +File: `snowflake-jdbc/internal/jdbc/cloud/storage/SnowflakeS3Client.java` + +**Fix:** Same — set 16MB threshold on all `S3AsyncClient` builders. + +### RISK 3: S3Exception.awsErrorDetails() null — MEDIUM + +**Problem:** `S3Exception.awsErrorDetails()` can return `null` in v2. Code that +calls `.errorCode()` on it without a null check will NPE. + +**Applies to us?** **YES.** Our error handlers in `SnowflakeS3Client` and +`IcebergS3Client` check for `ExpiredToken` error code. + +**How JDBC fixed it (PR #2550):** Added null check in `S3ErrorHandler.java`: +```java +if (e.awsErrorDetails() != null + && EXPIRED_AWS_TOKEN_ERROR_CODE.equalsIgnoreCase( + e.awsErrorDetails().errorCode())) { +``` +Also used `constant.equalsIgnoreCase(variable)` pattern for null safety. +File: `snowflake-jdbc/internal/jdbc/cloud/storage/S3ErrorHandler.java` + +**Fix:** Same — null-check `awsErrorDetails()` in all error handlers. + +### RISK 4: Custom signer API change — HIGH + +**Problem:** v1 uses `SignerFactory.registerSigner()` + `AWS4Signer` extension. +v2 uses `SdkAdvancedClientOption.SIGNER` + `software.amazon.awssdk.core.signer.Signer`. + +**Applies to us?** **YES.** `GCSAccessStrategyAwsSdk` registers a custom +`AwsSdkGCPSigner` that extends v1's `AWS4Signer`. + +**How JDBC did it:** Rewrote `AwsSdkGCPSigner` to implement v2's `Signer` +interface. The v2 signer: +1. Strips the AWS `Authorization` header +2. Adds `Authorization: Bearer ` for GCS +3. Maps `x-amz-*` → `x-goog-*` headers +4. Injected via `SdkAdvancedClientOption.SIGNER` on the client builder +5. Uses `AnonymousCredentialsProvider` (signing handled by custom signer) +File: `snowflake-jdbc/internal/jdbc/cloud/storage/AwsSdkGCPSigner.java` + +**Fix:** Replicate JDBC's v2 `AwsSdkGCPSigner` implementation. + +### RISK 5: aws-crt shading — LOW + +**Problem:** `aws-crt` native library cannot be shaded. Must be excluded. + +**Applies to us?** **YES.** We shade dependencies. + +**How JDBC did it:** Excludes `aws-crt` from `s3`, `s3-transfer-manager`, and +`http-auth-aws` dependencies in `parent-pom.xml`. Also has a build comment: +"aws-crt cannot be shaded". +File: `snowflake-jdbc/parent-pom.xml` + +**Fix:** Same — exclude `aws-crt` from all v2 deps that pull it transitively. + +--- + +## Migration Steps + +### Step 1+2: Add v2 deps + Migrate IcebergS3Client (PR #1149) — DONE + +Add v2 deps **alongside** v1 (v1 still used by SnowflakeS3Client, GCS clients): + +```xml + +com.amazonaws:aws-java-sdk-core:1.12.655 +com.amazonaws:aws-java-sdk-kms:1.12.655 +com.amazonaws:aws-java-sdk-s3:1.12.655 + + +software.amazon.awssdk:bom:2.37.5 (type=pom, scope=import) +software.amazon.awssdk:s3 +software.amazon.awssdk:s3-transfer-manager +software.amazon.awssdk:netty-nio-client +software.amazon.awssdk:auth +software.amazon.awssdk:http-auth-aws +``` + +Exclude `aws-crt` (groupId `software.amazon.awssdk.crt`) from `s3`, `s3-transfer-manager`, +`http-auth-aws`. Add shade rules for `software.amazon.awssdk`, `software.amazon.eventstream`, +`org.reactivestreams`. + +**IcebergS3Client changes (lowest risk — no client-side encryption):** +- `AmazonS3ClientBuilder` → `S3AsyncClient.builder()` +- `BasicAWSCredentials`/`BasicSessionCredentials` → `AwsBasicCredentials`/`AwsSessionCredentials` +- `AWSStaticCredentialsProvider` → `StaticCredentialsProvider` +- v1 `ClientConfiguration` → v2 `ClientConfiguration` inner class (matching JDBC's + `SnowflakeS3Client.ClientConfiguration`) with `maxConnections`, `maxErrorRetry`, + `connectionTimeout`, `socketTimeout` +- `NettyNioAsyncHttpClient` configured with `connectionAcquisitionTimeout(60s)`, + `connectionTimeout`, `readTimeout`, `writeTimeout` (matching JDBC) +- `TransferManager` → `S3TransferManager` +- `ObjectMetadata` → `IcebergS3ObjectMetadata` with `getS3PutObjectRequest()` builder + (matching JDBC's `S3ObjectMetadata`) including `ChecksumAlgorithm.CRC32` +- `SSEAwsKeyManagementParams` → `ServerSideEncryption.fromValue()` + `ssekmsKeyId()` +- Set multipart threshold to 16MB (Risk 2) +- Null-check `awsErrorDetails()` in error handler (Risk 3) +- Exception handler checks `ex.getCause()` for `SdkException` — `CompletableFuture.join()` + wraps in `CompletionException` (matching JDBC) +- `CompletionException` handling for non-SDK async failures +- `Region`/`RegionUtils` → `software.amazon.awssdk.regions.Region` +- Proxy: `ProxyConfiguration` with `.scheme()` (proxy protocol), `.useEnvironmentVariableValues(false)`, + `.useSystemPropertyValues(false)` (matching JDBC's `CloudStorageProxyFactory`) +- `SSLConnectionSocketFactory` removed (v2 Netty handles TLS natively) + +### Step 3: Migrate SnowflakeS3Client (high risk — encryption) + +Same as Step 2, plus: +- Wrap `CipherInputStream` in `BufferedInputStream` for uploads (Risk 1) +- `AmazonS3EncryptionClient` → removed (encryption is manual via EncryptionProvider) +- `CryptoConfiguration`/`EncryptionMaterials` → removed (not needed, encryption is JCE-based) +- Keep `EncryptionProvider.encrypt()` returning `CipherInputStream` but wrap at upload time + +### Step 4: Migrate GCS-via-AWS-SDK + +**Changes:** +- `AwsSdkGCPSigner` (extends `AWS4Signer`) → implement v2 `Signer` interface + - Strip AWS Authorization header, add `Bearer` token + - Map `x-amz-*` → `x-goog-*` headers + - Inject via `SdkAdvancedClientOption.SIGNER` +- `GCSAccessStrategyAwsSdk`: + - Use `S3AsyncClient` with GCS endpoint (`storage.googleapis.com`) + - `AnonymousCredentialsProvider` (signing handled by custom signer) + - `forcePathStyle(false)` for virtual-hosted style + +### Step 5: Migrate support classes + +- `S3HttpUtil` → use `ProxyConfiguration` from v2 +- `S3ObjectMetadata`/`IcebergS3ObjectMetadata` → wrap v2 response types +- `HeaderCustomizerHttpRequestInterceptor` → implement `ExecutionInterceptor` (v2) +- `S3ObjectSummariesIterator` → use `ListObjectsV2Response` +- `StorageClientFactory`/`IcebergStorageClientFactory` → update builders +- `SnowflakeMutableProxyRoutePlanner` → `ProxyConfiguration` (v2 handles this) +- `StorageObjectSummary`/`StorageObjectSummaryCollection` → adapt to v2 types +- `StorageProviderException` → catch `SdkException` (v2 base exception) + +### Step 6: Update shade plugin + remove v1 deps + +Shade rules for `software.amazon.awssdk`, `software.amazon.eventstream`, +`org.reactivestreams` already added in Step 1. Remaining work: + +- Remove v1 `com.amazonaws` deps and shade rules +- **Patch `execution.interceptors` after shading** — JDBC patches + `software/amazon/awssdk/global/handlers/execution.interceptors` file contents + because the shade plugin relocates the file path but not the class names inside + it. We need the same antrun `replace` task: + ```xml + + ``` + +### Step 7: Update tests + +Update all test files that reference v1 types. Verify: +- Encryption round-trip (EncryptionProviderTest, GcmEncryptionProviderTest) +- S3 client tests +- GCS signer test (AwsSdkGCPSignerTest) +- Integration tests on all 3 clouds + +--- + +## PR Strategy + +| PR | Content | Risk Level | Status | +|---|---|---|---| +| PR 1 (#1149) | Add v2 deps + migrate `IcebergS3Client` | Medium | **Open** | +| PR 2 | Migrate `SnowflakeS3Client` + `StorageClientFactory` (encryption path) | **High** | | +| PR 3 | Migrate `AwsSdkGCPSigner` + `GCSAccessStrategyAwsSdk` | Medium | | +| PR 4 | Migrate support classes (metadata, iterators, interceptor, proxy) | Low | | +| PR 5 | Remove v1 deps, shade cleanup, `execution.interceptors` patching | Low | | + +--- + +## JDBC Reference Files + +Key JDBC files to reference during migration (at `snowflakedb/snowflake-jdbc` main branch): + +| JDBC File | Ingest Equivalent | +|---|---| +| `internal/jdbc/cloud/storage/SnowflakeS3Client.java` | `SnowflakeS3Client.java` | +| `internal/jdbc/cloud/storage/GCSAccessStrategyAwsSdk.java` | `GCSAccessStrategyAwsSdk.java` | +| `internal/jdbc/cloud/storage/AwsSdkGCPSigner.java` | `AwsSdkGCPSigner.java` | +| `internal/jdbc/cloud/storage/EncryptionProvider.java` | `EncryptionProvider.java` | +| `internal/jdbc/cloud/storage/S3ErrorHandler.java` | Error handling in S3 clients | +| `parent-pom.xml` (deps section) | `pom.xml` | + +--- + +## Verification + +### Step 1+2 (IcebergS3Client) +- [x] `mvn compiler:compile` passes +- [x] `aws-crt` properly excluded from dependency tree +- [ ] All unit tests pass +- [ ] Integration tests pass on S3 (Iceberg path) +- [ ] SSE-S3 uploads (Iceberg path) work +- [ ] SSE-KMS uploads (Iceberg path) work +- [ ] Multipart uploads > 16MB work correctly +- [ ] Uploads < 16MB use single-part (not multipart) +- [ ] Proxy configuration works + +### Full migration (after all PRs) +- [ ] Integration tests pass on S3, Azure, GCS +- [ ] Encryption round-trip verified (upload encrypted → download → decrypt → matches original) +- [ ] GCS uploads via S3-compatible API work +- [ ] Shaded jar contains no `com.amazonaws` classes +- [ ] `execution.interceptors` patched correctly in shaded jar +- [ ] e2e-jar-test passes (shaded + unshaded) diff --git a/pom.xml b/pom.xml index c0e5a9f84..dd23efb90 100644 --- a/pom.xml +++ b/pom.xml @@ -43,6 +43,7 @@ 4.4.16 true 1.11.4 + 2.37.5 1.79 1.9.13 1.18.0 @@ -108,6 +109,13 @@ pom import + + software.amazon.awssdk + bom + ${awssdk.version} + pom + import + com.google.code.gson gson @@ -996,6 +1004,51 @@ org.slf4j slf4j-api + + + software.amazon.awssdk + auth + + + software.amazon.awssdk + aws-core + + + software.amazon.awssdk + http-client-spi + + + software.amazon.awssdk + netty-nio-client + + + software.amazon.awssdk + regions + + + software.amazon.awssdk + s3 + + + software.amazon.awssdk.crt + aws-crt + + + + + software.amazon.awssdk + s3-transfer-manager + + + software.amazon.awssdk.crt + aws-crt + + + + + software.amazon.awssdk + sdk-core + com.github.luben zstd-jni @@ -1383,7 +1436,7 @@ |Apache 2 BSD 2-Clause License |The BSD License |BSD - The MIT License|MIT License|MIT license + The MIT License|MIT License|MIT license|MIT-0 3-Clause BSD License|BSD-3-Clause The Go License|Go License @@ -1640,6 +1693,18 @@ com.amazonaws ${shadeBase}.com.amazonaws + + software.amazon.awssdk + ${shadeBase}.software.amazon.awssdk + + + software.amazon.eventstream + ${shadeBase}.software.amazon.eventstream + + + org.reactivestreams + ${shadeBase}.org.reactivestreams + com.google.api.gax ${shadeBase}.com.google.api.gax @@ -1736,6 +1801,7 @@ LICENSE NOTICE iceberg-build.properties + VersionInfo.java google/protobuf/**/*.proto THIRD-PARTY.txt diff --git a/scripts/process_licenses.py b/scripts/process_licenses.py index 718aaba88..441abca47 100644 --- a/scripts/process_licenses.py +++ b/scripts/process_licenses.py @@ -63,6 +63,7 @@ "io.netty:netty-handler-proxy": APACHE_LICENSE, "io.netty:netty-resolver": APACHE_LICENSE, "io.netty:netty-transport": APACHE_LICENSE, + "io.netty:netty-transport-classes-epoll": APACHE_LICENSE, "io.netty:netty-transport-native-unix-common": APACHE_LICENSE, "com.google.re2j:re2j": GO_LICENSE_ALT, "com.google.protobuf:protobuf-java": BSD_3_CLAUSE_LICENSE, @@ -80,6 +81,8 @@ "org.roaringbitmap:shims": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-core": APACHE_LICENSE, "software.amazon.ion:ion-java": APACHE_LICENSE, + "software.amazon.eventstream:eventstream": APACHE_LICENSE, + "org.reactivestreams:reactive-streams": MIT_LICENSE, "com.amazonaws:aws-java-sdk-kms": APACHE_LICENSE, "com.amazonaws:jmespath-java": APACHE_LICENSE, "com.amazonaws:aws-java-sdk-s3": APACHE_LICENSE, diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java index 79f5e1ea5..678e3de27 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3Client.java @@ -4,53 +4,50 @@ import static net.snowflake.ingest.streaming.internal.fileTransferAgent.StorageClientUtil.createDefaultExecutorService; import static net.snowflake.ingest.streaming.internal.fileTransferAgent.StorageClientUtil.getRootCause; -import com.amazonaws.AmazonClientException; -import com.amazonaws.AmazonServiceException; -import com.amazonaws.ClientConfiguration; -import com.amazonaws.Protocol; -import com.amazonaws.auth.AWSCredentials; -import com.amazonaws.auth.AWSStaticCredentialsProvider; -import com.amazonaws.auth.BasicAWSCredentials; -import com.amazonaws.auth.BasicSessionCredentials; -import com.amazonaws.client.builder.AwsClientBuilder; -import com.amazonaws.client.builder.ExecutorFactory; -import com.amazonaws.regions.Region; -import com.amazonaws.regions.RegionUtils; -import com.amazonaws.services.s3.AmazonS3; -import com.amazonaws.services.s3.AmazonS3Builder; -import com.amazonaws.services.s3.AmazonS3Client; -import com.amazonaws.services.s3.model.AmazonS3Exception; -import com.amazonaws.services.s3.model.ObjectMetadata; -import com.amazonaws.services.s3.model.PutObjectRequest; -import com.amazonaws.services.s3.model.SSEAlgorithm; -import com.amazonaws.services.s3.model.SSEAwsKeyManagementParams; -import com.amazonaws.services.s3.transfer.TransferManager; -import com.amazonaws.services.s3.transfer.TransferManagerBuilder; -import com.amazonaws.services.s3.transfer.Upload; +import java.io.BufferedInputStream; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; import java.net.SocketTimeoutException; +import java.net.URI; import java.security.InvalidKeyException; -import java.security.KeyManagementException; -import java.security.NoSuchAlgorithmException; +import java.time.Duration; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Properties; +import java.util.Set; +import java.util.concurrent.CompletionException; import java.util.concurrent.ExecutorService; import net.snowflake.ingest.streaming.internal.VolumeEncryptionMode; -import net.snowflake.ingest.utils.HttpUtil; import net.snowflake.ingest.utils.Logging; import net.snowflake.ingest.utils.SFPair; import net.snowflake.ingest.utils.SFSessionProperty; import net.snowflake.ingest.utils.Stopwatch; import org.apache.commons.io.IOUtils; import org.apache.http.HttpStatus; -import org.apache.http.conn.ssl.SSLConnectionSocketFactory; -import org.apache.http.conn.ssl.SSLInitializationException; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentials; +import software.amazon.awssdk.auth.credentials.AwsSessionCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.async.AsyncRequestBody; +import software.amazon.awssdk.core.exception.SdkException; +import software.amazon.awssdk.core.exception.SdkServiceException; +import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient; +import software.amazon.awssdk.http.nio.netty.ProxyConfiguration; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3AsyncClient; +import software.amazon.awssdk.services.s3.S3AsyncClientBuilder; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; +import software.amazon.awssdk.services.s3.model.S3Exception; +import software.amazon.awssdk.services.s3.model.ServerSideEncryption; +import software.amazon.awssdk.transfer.s3.S3TransferManager; +import software.amazon.awssdk.transfer.s3.model.CompletedUpload; +import software.amazon.awssdk.transfer.s3.model.Upload; +import software.amazon.awssdk.transfer.s3.model.UploadRequest; class IcebergS3Client implements IcebergStorageClient { private static final Logging logger = new Logging(IcebergS3Client.class); @@ -59,38 +56,20 @@ class IcebergS3Client implements IcebergStorageClient { // expired AWS token error code private static final String EXPIRED_AWS_TOKEN_ERROR_CODE = "ExpiredToken"; + // Multipart upload threshold: 16 MB + private static final long MULTIPART_THRESHOLD_BYTES = 16L * 1024 * 1024; + private boolean isUseS3RegionalUrl = false; private ClientConfiguration clientConfig = null; + private ProxyConfiguration proxyConfig = null; private String stageRegion = null; private Properties proxyProperties = null; private String stageEndPoint = null; // FIPS endpoint, if needed private boolean isClientSideEncrypted = true; - private AmazonS3 amazonClient = null; + private S3AsyncClient amazonClient = null; private VolumeEncryptionMode volumeEncryptionMode = null; private String encryptionKmsKeyId = null; - // socket factory used by s3 client's http client. - private static SSLConnectionSocketFactory s3ConnectionSocketFactory = null; - - private static SSLConnectionSocketFactory getSSLConnectionSocketFactory() { - if (s3ConnectionSocketFactory == null) { - synchronized (IcebergS3Client.class) { - if (s3ConnectionSocketFactory == null) { - try { - // trust manager is set to null, which will use default ones - // instead of SFTrustManager (which enables ocsp checking) - s3ConnectionSocketFactory = - new IngestSSLConnectionSocketFactory(null, HttpUtil.isSocksProxyDisabled()); - } catch (KeyManagementException | NoSuchAlgorithmException e) { - throw new SSLInitializationException(e.getMessage(), e); - } - } - } - } - - return s3ConnectionSocketFactory; - } - public IcebergS3Client( Map stageCredentials, ClientConfiguration clientConfig, @@ -137,38 +116,53 @@ private void setupSnowflakeS3Client( String awsToken = (String) stageCredentials.get("AWS_TOKEN"); // initialize aws credentials - AWSCredentials awsCredentials = + AwsCredentials awsCredentials = (awsToken != null) - ? new BasicSessionCredentials(awsID, awsKey, awsToken) - : new BasicAWSCredentials(awsID, awsKey); - - clientConfig.withSignerOverride("AWSS3V4SignerType"); - clientConfig.getApacheHttpClientConfig().setSslSocketFactory(getSSLConnectionSocketFactory()); - setSessionlessProxyForS3(proxyProperties, clientConfig); - AmazonS3Builder amazonS3Builder = - AmazonS3Client.builder() - .withCredentials(new AWSStaticCredentialsProvider(awsCredentials)) - .withClientConfiguration(clientConfig); - - Region region = RegionUtils.getRegion(stageRegion); - if (this.stageEndPoint != null && this.stageEndPoint != "" && this.stageEndPoint != "null") { - amazonS3Builder.withEndpointConfiguration( - new AwsClientBuilder.EndpointConfiguration(this.stageEndPoint, region.getName())); + ? AwsSessionCredentials.create(awsID, awsKey, awsToken) + : AwsBasicCredentials.create(awsID, awsKey); + + this.proxyConfig = buildProxyConfiguration(proxyProperties); + + NettyNioAsyncHttpClient.Builder httpClientBuilder = + NettyNioAsyncHttpClient.builder() + .maxConcurrency(clientConfig.getMaxConnections()) + .connectionAcquisitionTimeout(Duration.ofSeconds(60)) + .connectionTimeout(Duration.ofMillis(clientConfig.getConnectionTimeout())) + .readTimeout(Duration.ofMillis(clientConfig.getSocketTimeout())) + .writeTimeout(Duration.ofMillis(clientConfig.getSocketTimeout())); + if (this.proxyConfig != null) { + httpClientBuilder.proxyConfiguration(this.proxyConfig); + } + + S3AsyncClientBuilder s3Builder = + S3AsyncClient.builder() + .credentialsProvider(StaticCredentialsProvider.create(awsCredentials)) + .httpClientBuilder(httpClientBuilder) + .forcePathStyle(false) + .multipartEnabled(true) + .multipartConfiguration( + software.amazon.awssdk.services.s3.multipart.MultipartConfiguration.builder() + .thresholdInBytes(MULTIPART_THRESHOLD_BYTES) + .build()); + + Region region = Region.of(stageRegion); + if (this.stageEndPoint != null + && !this.stageEndPoint.isEmpty() + && !"null".equals(this.stageEndPoint)) { + s3Builder.endpointOverride(URI.create(this.stageEndPoint)); + s3Builder.region(region); } else { - if (region != null) { + if (stageRegion != null) { if (this.isUseS3RegionalUrl) { - String domainSuffixForRegionalUrl = getDomainSuffixForRegionalUrl(region.getName()); - amazonS3Builder.withEndpointConfiguration( - new AwsClientBuilder.EndpointConfiguration( - "s3." + region.getName() + "." + domainSuffixForRegionalUrl, region.getName())); - } else { - amazonS3Builder.withRegion(region.getName()); + String domainSuffixForRegionalUrl = getDomainSuffixForRegionalUrl(stageRegion); + s3Builder.endpointOverride( + URI.create("https://s3." + stageRegion + "." + domainSuffixForRegionalUrl)); } + s3Builder.region(region); } } - // Explicitly force to use virtual address style - amazonS3Builder.withPathStyleAccessEnabled(false); - amazonClient = (AmazonS3) amazonS3Builder.build(); + + amazonClient = s3Builder.build(); } /* Adds digest metadata to the StorageObjectMetadata object */ @@ -241,24 +235,25 @@ public String upload( final long originalContentLength = meta.getContentLength(); final List toClose = new ArrayList<>(); + + IcebergS3ObjectMetadata s3Meta; + if (meta instanceof IcebergS3ObjectMetadata) { + s3Meta = (IcebergS3ObjectMetadata) meta; + } else { + throw new IllegalArgumentException("Unexpected metadata object type"); + } + SFPair uploadStreamInfo = createUploadStream( srcFile, uploadFromStream, inputStream, fileBackedOutputStream, - ((IcebergS3ObjectMetadata) meta).getS3ObjectMetadata(), + s3Meta, originalContentLength, toClose); - ObjectMetadata s3Meta; - if (meta instanceof IcebergS3ObjectMetadata) { - s3Meta = ((IcebergS3ObjectMetadata) meta).getS3ObjectMetadata(); - } else { - throw new IllegalArgumentException("Unexpected metadata object type"); - } - - TransferManager tx = null; + S3TransferManager tx = null; int retryCount = 0; Stopwatch stopwatch = new Stopwatch(); stopwatch.start(); @@ -267,37 +262,48 @@ public String upload( logger.logDebug( "Creating executor service for transfer" + "manager with {} threads", parallelism); + ExecutorService executor = + createDefaultExecutorService("s3-transfer-manager-uploader-", parallelism); + // upload files to s3 - tx = - TransferManagerBuilder.standard() - .withS3Client(amazonClient) - .withExecutorFactory( - new ExecutorFactory() { - @Override - public ExecutorService newExecutor() { - return createDefaultExecutorService( - "s3-transfer-manager-uploader-", parallelism); - } - }) - .build(); - - PutObjectRequest putRequest = - uploadStreamInfo.right - ? new PutObjectRequest( - remoteStorageLocation, destFileName, uploadStreamInfo.left, s3Meta) - : new PutObjectRequest(remoteStorageLocation, destFileName, srcFile); - putRequest.setMetadata(s3Meta); + tx = S3TransferManager.builder().s3Client(amazonClient).executor(executor).build(); + + // Build the PutObjectRequest from metadata (includes CRC32 checksum) + PutObjectRequest.Builder putBuilder = + s3Meta.getS3PutObjectRequest().toBuilder() + .bucket(remoteStorageLocation) + .key(destFileName); if (this.volumeEncryptionMode != null) { this.volumeEncryptionMode.validateKmsKeyId(this.encryptionKmsKeyId); } + if (s3Meta.getSSEAlgorithm() != null) { + putBuilder.serverSideEncryption(ServerSideEncryption.fromValue(s3Meta.getSSEAlgorithm())); + } if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - putRequest.setSSEAwsKeyManagementParams( - new SSEAwsKeyManagementParams(this.encryptionKmsKeyId)); + putBuilder.ssekmsKeyId(this.encryptionKmsKeyId); } - final Upload myUpload = tx.upload(putRequest); - myUpload.waitForCompletion(); + PutObjectRequest putRequest = putBuilder.build(); + + // Wrap the stream with BufferedInputStream to work around CipherInputStream bug + InputStream uploadStream = uploadStreamInfo.left; + AsyncRequestBody requestBody; + if (uploadStreamInfo.right) { + requestBody = + AsyncRequestBody.fromInputStream( + new BufferedInputStream(uploadStream), originalContentLength, executor); + } else { + requestBody = AsyncRequestBody.fromFile(srcFile.toPath()); + } + + Upload myUpload = + tx.upload( + UploadRequest.builder() + .putObjectRequest(putRequest) + .requestBody(requestBody) + .build()); + CompletedUpload result = myUpload.completionFuture().join(); stopwatch.stop(); long uploadMillis = stopwatch.elapsedMillis(); @@ -320,9 +326,7 @@ public ExecutorService newExecutor() { uploadMillis, retryCount); } - // -------------------------NEW LOGIC HERE START ----------------------------------------- - return myUpload.waitForUploadResult().getETag(); - // -------------------------NEW LOGIC HERE END ----------------------------------------- + return result.response().eTag(); } catch (Exception ex) { handleS3Exception(ex, ++retryCount, "upload", this); @@ -346,7 +350,7 @@ public ExecutorService newExecutor() { toClose); } finally { if (tx != null) { - tx.shutdownNow(false); + tx.close(); } } } while (retryCount <= getMaxRetries()); @@ -371,11 +375,11 @@ static String getDomainSuffixForRegionalUrl(String regionName) { * @param ex exception * @return true if it's a 400 or 404 status code */ - public boolean isClientException400Or404(Exception ex) { - if (ex instanceof AmazonServiceException) { - AmazonServiceException asEx = (AmazonServiceException) (ex); - return asEx.getStatusCode() == HttpStatus.SC_NOT_FOUND - || asEx.getStatusCode() == HttpStatus.SC_BAD_REQUEST; + public boolean isClientException400Or404(Throwable ex) { + if (ex instanceof SdkServiceException) { + SdkServiceException ssEx = (SdkServiceException) ex; + return ssEx.statusCode() == HttpStatus.SC_NOT_FOUND + || ssEx.statusCode() == HttpStatus.SC_BAD_REQUEST; } return false; } @@ -385,7 +389,7 @@ private SFPair createUploadStream( boolean uploadFromStream, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, - ObjectMetadata meta, + IcebergS3ObjectMetadata meta, long originalContentLength, List toClose) throws SnowflakeSQLException { @@ -406,10 +410,10 @@ private SFPair createUploadStream( if (this.volumeEncryptionMode == null || VolumeEncryptionMode.NONE.equals(this.volumeEncryptionMode) || VolumeEncryptionMode.AWS_SSE_S3.equals(this.volumeEncryptionMode)) { - // Default to SSE-S3 encryption - meta.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); + // Default to SSE-S3 encryption (AES256) + meta.setSSEAlgorithm(ServerSideEncryption.AES256.toString()); } else if (VolumeEncryptionMode.AWS_SSE_KMS.equals(this.volumeEncryptionMode)) { - meta.setSSEAlgorithm(SSEAlgorithm.KMS.getAlgorithm()); + meta.setSSEAlgorithm(ServerSideEncryption.AWS_KMS.toString()); } else { throw new IllegalArgumentException( "Unexpected volume encryption mode: " @@ -467,30 +471,41 @@ private static void handleS3Exception( } // Don't retry if max retries has been reached or the error code is 404/400 - if (ex instanceof AmazonClientException) { - logger.logDebug("AmazonClientException: " + ex.getMessage()); - if (retryCount > s3Client.getMaxRetries() || s3Client.isClientException400Or404(ex)) { + // Check ex.getCause() because CompletableFuture.join() wraps in CompletionException + Throwable cause = ex.getCause(); + if (cause instanceof SdkException) { + logger.logDebug("SdkException: " + ex.getMessage()); + if (retryCount > s3Client.getMaxRetries() || s3Client.isClientException400Or404(cause)) { String extendedRequestId = "none"; - if (ex instanceof AmazonS3Exception) { - AmazonS3Exception ex1 = (AmazonS3Exception) ex; - extendedRequestId = ex1.getExtendedRequestId(); + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; + extendedRequestId = s3ex.extendedRequestId() != null ? s3ex.extendedRequestId() : "none"; } - if (ex instanceof AmazonServiceException) { - AmazonServiceException ex1 = (AmazonServiceException) ex; + if (cause instanceof SdkServiceException) { + SdkServiceException ex1 = (SdkServiceException) cause; // The AWS credentials might have expired when server returns error 400 and // does not return the ExpiredToken error code. // If session is null we cannot renew the token so throw the exception + String errorCode = "Unknown"; + String errorType = "Unknown"; + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; + if (s3ex.awsErrorDetails() != null) { + errorCode = s3ex.awsErrorDetails().errorCode(); + errorType = s3ex.awsErrorDetails().errorMessage(); + } + } throw new SnowflakeSQLLoggedException( SqlState.SYSTEM_ERROR, ErrorCode.S3_OPERATION_ERROR.getMessageCode(), ex1, operation, - ex1.getErrorType().toString(), - ex1.getErrorCode(), + errorType, + errorCode, ex1.getMessage(), - ex1.getRequestId(), + ex1.requestId() != null ? ex1.requestId() : "none", extendedRequestId); } else { throw new SnowflakeSQLLoggedException( @@ -525,12 +540,14 @@ private static void handleS3Exception( // If the exception indicates that the AWS token has expired, // we need to refresh our S3 client with the new token - if (ex instanceof AmazonS3Exception) { - AmazonS3Exception s3ex = (AmazonS3Exception) ex; - if (s3ex.getErrorCode().equalsIgnoreCase(EXPIRED_AWS_TOKEN_ERROR_CODE)) { + if (cause instanceof S3Exception) { + S3Exception s3ex = (S3Exception) cause; + if (s3ex.awsErrorDetails() != null + && EXPIRED_AWS_TOKEN_ERROR_CODE.equalsIgnoreCase( + s3ex.awsErrorDetails().errorCode())) { // If session is null we cannot renew the token so throw the ExpiredToken exception throw new SnowflakeSQLException( - s3ex.getErrorCode(), + s3ex.awsErrorDetails().errorCode(), CLOUD_STORAGE_CREDENTIALS_EXPIRED, "S3 credentials have expired"); } @@ -538,7 +555,8 @@ private static void handleS3Exception( } } else { if (ex instanceof InterruptedException - || getRootCause(ex) instanceof SocketTimeoutException) { + || getRootCause(ex) instanceof SocketTimeoutException + || ex instanceof CompletionException) { if (retryCount > s3Client.getMaxRetries()) { throw new SnowflakeSQLLoggedException( SqlState.SYSTEM_ERROR, @@ -563,14 +581,14 @@ private static void handleS3Exception( } /** - * Copied from JDBC Driver S3HttpUtil to avoid class definition conflicts with AWS SDK + * Builds a ProxyConfiguration from proxy properties, or returns null if no proxy is configured. * * @param proxyProperties Proxy Properties - * @param clientConfig AWS Client Configuration + * @return ProxyConfiguration or null * @throws SnowflakeSQLException Thrown on configuration parsing failures */ - private static void setSessionlessProxyForS3( - Properties proxyProperties, ClientConfiguration clientConfig) throws SnowflakeSQLException { + private static ProxyConfiguration buildProxyConfiguration(Properties proxyProperties) + throws SnowflakeSQLException { if (proxyProperties != null && proxyProperties.size() > 0 && proxyProperties.getProperty(SFSessionProperty.USE_PROXY.getPropertyKey()) != null) { @@ -599,34 +617,82 @@ private static void setSessionlessProxyForS3( proxyProperties.getProperty(SFSessionProperty.NON_PROXY_HOSTS.getPropertyKey()); String proxyProtocol = proxyProperties.getProperty(SFSessionProperty.PROXY_PROTOCOL.getPropertyKey()); - Protocol protocolEnum = - isNotEmpty(proxyProtocol) && proxyProtocol.equalsIgnoreCase("https") - ? Protocol.HTTPS - : Protocol.HTTP; - clientConfig.setProxyHost(proxyHost); - clientConfig.setProxyPort(proxyPort); - clientConfig.setNonProxyHosts(nonProxyHosts); - clientConfig.setProxyProtocol(protocolEnum); + String scheme = + isNotEmpty(proxyProtocol) && proxyProtocol.equalsIgnoreCase("https") ? "https" : "http"; + + ProxyConfiguration.Builder proxyBuilder = + ProxyConfiguration.builder() + .scheme(scheme) + .host(proxyHost) + .port(proxyPort) + .useEnvironmentVariableValues(false) + .useSystemPropertyValues(false); + + if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { + proxyBuilder.username(proxyUser).password(proxyPassword); + } + + if (isNotEmpty(nonProxyHosts)) { + Set nonProxyHostSet = new HashSet<>(); + for (String host : nonProxyHosts.split("\\|")) { + if (!host.isEmpty()) { + nonProxyHostSet.add(host); + } + } + proxyBuilder.nonProxyHosts(nonProxyHostSet); + } + String logMessage = String.format( - "Set sessionless S3 proxy. Host: %s, port: %d, non-proxy hosts: %s, protocol: %s", - proxyHost, proxyPort, nonProxyHosts, proxyProtocol); + "Set sessionless S3 proxy. Host: %s, port: %d, protocol: %s, non-proxy hosts: %s", + proxyHost, proxyPort, scheme, nonProxyHosts); if (isNotEmpty(proxyUser) && isNotEmpty(proxyPassword)) { logMessage = String.format("%s, user: %s with password provided", logMessage, proxyUser); - clientConfig.setProxyUsername(proxyUser); - clientConfig.setProxyPassword(proxyPassword); } - logger.logDebug(logMessage); + + return proxyBuilder.build(); } else { logger.logDebug("Omitting sessionless S3 proxy setup as proxy is disabled"); } } else { logger.logDebug("Omitting sessionless S3 proxy setup"); } + return null; } private static boolean isNotEmpty(final String string) { return string != null && !string.isEmpty(); } + + public static class ClientConfiguration { + private final int maxConnections; + private final int maxErrorRetry; + private final int connectionTimeout; + private final int socketTimeout; + + public ClientConfiguration( + int maxConnections, int maxErrorRetry, int connectionTimeout, int socketTimeout) { + this.maxConnections = maxConnections; + this.maxErrorRetry = maxErrorRetry; + this.connectionTimeout = connectionTimeout; + this.socketTimeout = socketTimeout; + } + + public int getMaxConnections() { + return maxConnections; + } + + public int getMaxErrorRetry() { + return maxErrorRetry; + } + + public int getConnectionTimeout() { + return connectionTimeout; + } + + public int getSocketTimeout() { + return socketTimeout; + } + } } diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java index 998ccdabd..b8cea9503 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergS3ObjectMetadata.java @@ -4,63 +4,89 @@ package net.snowflake.ingest.streaming.internal.fileTransferAgent; -import com.amazonaws.services.s3.model.ObjectMetadata; import java.util.Map; +import java.util.TreeMap; +import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; /** - * s3 implementation of platform independent StorageObjectMetadata interface, wraps an S3 - * ObjectMetadata class + * S3 implementation of platform independent StorageObjectMetadata interface. Uses plain Map-based + * storage (AWS SDK v2 does not have an ObjectMetadata class; metadata is passed directly on the + * PutObjectRequest builder). * *

It only supports a limited set of metadata properties currently used by the JDBC client * * @author lgiakoumakis */ public class IcebergS3ObjectMetadata implements StorageObjectMetadata { - private ObjectMetadata objectMetadata; + private long contentLength; + private final Map userMetadata; + private String contentEncoding; - IcebergS3ObjectMetadata() { - objectMetadata = new ObjectMetadata(); - } + // SSE algorithm to apply (e.g. "AES256", "aws:kms") + private String sseAlgorithm; - // Construct from an AWS S3 ObjectMetadata object - IcebergS3ObjectMetadata(ObjectMetadata meta) { - objectMetadata = meta; + IcebergS3ObjectMetadata() { + userMetadata = new TreeMap<>(String.CASE_INSENSITIVE_ORDER); } @Override public Map getUserMetadata() { - return StorageClientUtil.createCaseInsensitiveMap(objectMetadata.getUserMetadata()); + return StorageClientUtil.createCaseInsensitiveMap(userMetadata); + } + + /** + * Returns the raw (mutable) user metadata map. Used internally by IcebergS3Client to build the + * PutObjectRequest. + */ + Map getRawUserMetadata() { + return userMetadata; } @Override public long getContentLength() { - return objectMetadata.getContentLength(); + return contentLength; } @Override public void setContentLength(long contentLength) { - objectMetadata.setContentLength(contentLength); + this.contentLength = contentLength; } @Override public void addUserMetadata(String key, String value) { - objectMetadata.addUserMetadata(key, value); + userMetadata.put(key, value); } @Override public void setContentEncoding(String encoding) { - objectMetadata.setContentEncoding(encoding); + this.contentEncoding = encoding; } @Override public String getContentEncoding() { - return objectMetadata.getContentEncoding(); + return contentEncoding; + } + + /** Sets the server-side encryption algorithm (e.g. "AES256", "aws:kms"). */ + void setSSEAlgorithm(String algorithm) { + this.sseAlgorithm = algorithm; + } + + /** Returns the server-side encryption algorithm, or null if not set. */ + String getSSEAlgorithm() { + return sseAlgorithm; } /** - * @return Returns the encapsulated AWS S3 metadata object + * @return Returns the encapsulated AWS S3 PutObjectRequest */ - ObjectMetadata getS3ObjectMetadata() { - return objectMetadata; + PutObjectRequest getS3PutObjectRequest() { + return PutObjectRequest.builder() + .metadata(userMetadata) + .contentLength(contentLength) + .contentEncoding(contentEncoding) + .checksumAlgorithm(ChecksumAlgorithm.CRC32) + .build(); } } diff --git a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java index a2ab6db32..1c3bafa73 100644 --- a/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java +++ b/src/main/java/net/snowflake/ingest/streaming/internal/fileTransferAgent/IcebergStorageClientFactory.java @@ -1,10 +1,8 @@ package net.snowflake.ingest.streaming.internal.fileTransferAgent; -import com.amazonaws.ClientConfiguration; import java.util.Map; import java.util.Properties; import net.snowflake.ingest.streaming.internal.VolumeEncryptionMode; -import net.snowflake.ingest.utils.HttpUtil; import net.snowflake.ingest.utils.Logging; /** @@ -112,17 +110,12 @@ private IcebergStorageClient createS3Client( IcebergS3Client s3Client; - ClientConfiguration clientConfig = new ClientConfiguration(); - clientConfig.setMaxConnections(parallel + 1); - clientConfig.setMaxErrorRetry(S3_TRANSFER_MAX_RETRIES); - clientConfig.setDisableSocketProxy(HttpUtil.isSocksProxyDisabled()); - - // If proxy is set via connection properties or JVM settings these will be overridden later. - // This is to prevent the aws client builder from reading proxy environment variables. - clientConfig.setProxyHost(""); - clientConfig.setProxyPort(0); - clientConfig.setProxyUsername(""); - clientConfig.setProxyPassword(""); + IcebergS3Client.ClientConfiguration clientConfig = + new IcebergS3Client.ClientConfiguration( + parallel + 1, + S3_TRANSFER_MAX_RETRIES, + (int) JdbcHttpUtil.getConnectionTimeout().toMillis(), + (int) JdbcHttpUtil.getSocketTimeout().toMillis()); logger.logDebug( "S3 client configuration: maxConnection: {}, connectionTimeout: {}, "