Repository navigation
feat: admit S3FileIO's AssumeRoleAwsClientFactory for native Iceberg writes - #7680
surawut-jirasaktavee wants to merge 1 commit into
Conversation
d57018d to
cd74716
Compare
comphead
left a comment
There was a problem hiding this comment.
The factory, ARN, region and credential-conflict handling line up with iceberg-java's AssumeRoleAwsClientFactory and the pinned iceberg-rust/opendal S3 parser. I found one gate/native mismatch in the flag check, one existing hole this PR makes visible, an undocumented STS/session-name difference, and a gap in test coverage of the new rule.
| NativeS3RegionProperties.filter(properties.get(_).exists(_ != region)) ++ | ||
| AssumeRoleConflictingCredentialProperties.filter(properties.contains) ++ | ||
| AssumeRoleConflictingCredentialFlags.filter( | ||
| properties.get(_).exists(java.lang.Boolean.parseBoolean)) |
There was a problem hiding this comment.
Boolean.parseBoolean only accepts true, but the pinned iceberg-rust reads these three flags with is_truthy (crates/storage/opendal/src/s3.rs and utils.rs at de0a6ac), which accepts true, t, 1 and on case-insensitively. So s3.allow-anonymous=1, s3.disable-config-load=on or s3.disable-ec2-metadata=t pass this check, iceberg-java reads them as false, and the native writer then skips signing or drops the env/IMDS credential sources that AssumeRole needs as its source. The write fails in the task instead of falling back. Please use the same truthiness as iceberg-rust here and add those spellings to the test, which only uses true.
There was a problem hiding this comment.
Fixed. s3.allow-anonymous, s3.disable-ec2-metadata and s3.disable-config-load now use the same rule as iceberg-rust's is_truthy (true, t, 1, on, case-insensitive), so any of those spellings falls back with the factory. The test now also covers 1, T and on
| .get(CometS3CredentialProviderClassProperty) | ||
| .exists(_.trim.nonEmpty) | ||
|
|
||
| val assumeRole = usesAssumeRoleClientFactory(properties) |
There was a problem hiding this comment.
Without client.factory, iceberg-java ignores client.assume-role.* (AwsClientFactories.from defaults to DefaultAwsClientFactory, and a HadoopFileIO write goes through S3A). But client.assume-role.arn, external-id and session-name are in SupportedS3FileIOProperties unconditionally, and the test at unsupported(assumeRole - "client.factory") == Seq("client.assume-role.region") shows that an ARN without the factory is still admitted when no region is set. The native writer then assumes the role while the JVM writer would have used the default credentials, so the write happens under a different identity. This predates the PR, but the PR description states the rule and this is the place to enforce it. Consider declining the ARN, external id and session name unless assumeRole holds and the FileIO is S3FileIO.
There was a problem hiding this comment.
Fixed here. client.assume-role.arn, external-id and session-name are no longer in SupportedS3FileIOProperties. They are admitted only together with client.factory=AssumeRoleAwsClientFactory, a non-blank arn and region, and S3FileIO, so an ARN without the factory now falls back. I updated the existing supported-keys test and added a case for the assume-role keys without the factory
| With `S3FileIO`, `client.factory` is admitted only as `org.apache.iceberg.aws.AssumeRoleAwsClientFactory` | ||
| with `client.assume-role.arn` and `client.assume-role.region` set, and that region becomes the write | ||
| region. The write falls back when the factory is combined with a different `client.region` or | ||
| `s3.region`, static S3 keys, a credential-provider class, or `s3.allow-anonymous`, | ||
| `s3.disable-ec2-metadata` or `s3.disable-config-load` set to `true`. |
There was a problem hiding this comment.
Two observable differences from the factory are not listed. First, iceberg-java names the session iceberg-aws-<uuid> when client.assume-role.session-name is unset (AssumeRoleAwsClientFactory.genSessionName), while opendal/reqsign sends reqsign (your CloudTrail output shows it). A trust policy with an sts:RoleSessionName condition that works for the JVM writer will deny the native one, and the query fails instead of falling back. Second, the native STS call goes to the regional endpoint of client.assume-role.region, whereas the factory's STS client takes its region from the default provider chain (sts() sets none). I expect this matters only with a restricted STS route, but I have not tested it. Please add both to the accepted divergences, or decline when a trust-policy-sensitive setup cannot be ruled out.
There was a problem hiding this comment.
Session name: when client.assume-role.session-name is unset, the native writer now gets iceberg-aws-<uuid>, the same form as AssumeRoleAwsClientFactory.genSessionName. iceberg-rust passes it to opendal as role_session_name, so a trust policy with an sts:RoleSessionName condition sees the same pattern from both writers. One difference remains: iceberg-java generates one name per factory instance, while the native writer generates one per write
| lazy val s3FileIO = IcebergReflection | ||
| .getFileIO(ctx.table) | ||
| .flatMap(io => IcebergReflection.resolveFileIOClass(io, location)) | ||
| .exists(cls => | ||
| IcebergReflection | ||
| .classNameInHierarchy(cls, Set(IcebergReflection.ClassNames.S3_FILE_IO))) | ||
| unsupportedSettingsReason( | ||
| "S3 FileIO", | ||
| unsupportedS3FileIOProperties( | ||
| ctx.fileIOProperties, | ||
| ctx.fileIOProperties.contains(S3FileIOClientFactoryProperty) && s3FileIO)) |
There was a problem hiding this comment.
The rule that decides s3FileIO is not covered. The new tests call unsupportedS3FileIOProperties(..., s3FileIO = true/false) directly, so the HadoopFileIO fallback test only proves a boolean parameter, not that a ResolvingFileIO resolving to HadoopFileIO, a failed resolveFileIOClass, or an S3FileIO subclass lands on the right side. gcsDataLocationRejection takes the resolved Option[Class[_]] for this reason. Consider the same shape here so a test can feed S3FileIO, HadoopFileIO and None. Also note classNameInHierarchy admits any subclass of S3FileIO, which can override client creation and ignore the factory. An exact class match would fail closed.
There was a problem hiding this comment.
Fixed. The decision is now isS3FileIO(resolvedFileIO: Option[Class[_]]), the same shape as gcsDataLocationRejection. It matches org.apache.iceberg.aws.s3.S3FileIO exactly instead of using classNameInHierarchy, so a subclass falls back. A new test checks S3FileIO, an S3FileIO subclass, HadoopFileIO and None
d461558 to
e3b6e05
Compare
…writes iceberg-java turns on its assume-role path only through client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory, and its S3 client then takes its region from client.assume-role.region, a key the pinned iceberg-rust does not read. Both keys made every such write fall back. Admit the factory together with the client.assume-role.* settings when the data location is written through S3FileIO itself and client.assume-role.arn and client.assume-role.region are set. Without the factory, iceberg-java ignores the assume-role settings, so they now fall back. Pass the region to the native writer as client.region and, when no session name is set, name the session iceberg-aws-<uuid> as the factory does. Still fall back when the factory is combined with settings the native writer would read and the factory's S3 client ignores: a different client.region or s3.region, static S3 keys, a Comet credential provider class, or s3.allow-anonymous, s3.disable-ec2-metadata or s3.disable-config-load set to a value iceberg-rust reads as true.
e3b6e05 to
e2df0d1
Compare
Which issue does this PR close?
Closes #7667.
Rationale for this change
The Iceberg docs' assume-role setup for Spark on EKS (
io-impl=S3FileIO,client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory,client.assume-role.arn,client.assume-role.region) makes every native Iceberg write fall back:The pinned iceberg-rust already reads
client.assume-role.arn,external-idandsession-name. The fallback comes from the two keys iceberg-java uses to turn that path on:client.factory, without which iceberg-java ignores everyclient.assume-role.*key, andclient.assume-role.region, which the pinned iceberg-rust does not read.What changes are included in this PR?
client.factoryandclient.assume-role.arn,external-id,session-nameandregionare admitted together, only when all of these hold:org.apache.iceberg.aws.AssumeRoleAwsClientFactory;client.assume-role.arnandclient.assume-role.regionare set and not blank;S3FileIOitself, including aResolvingFileIOthat resolves to it. A subclass ofS3FileIOcan ignore the factory, andHadoopFileIOignores it and writes through S3A, so both fall back.client.assume-role.arn,external-idandsession-namenow fall back. Before this PR they were admitted on their own, and the native writer assumed a role that iceberg-java ignores.client.regionwhen neitherclient.regionnors3.regionis set.client.assume-role.session-nameis not set, the native writer names the sessioniceberg-aws-<uuid>, as the factory does, instead of opendal'sreqsign.client.regionors3.regiondifferent from the assume-role region;s3.access-key-id,s3.secret-access-keyors3.session-token(opendal would use them as the source credentials for AssumeRole);s3.allow-anonymous,s3.disable-ec2-metadataors3.disable-config-loadset to a value iceberg-rust reads as true (true,t,1oron, in any case).client.assume-role.tags.*andclient.assume-role.timeout-sec, still fall back.How are these changes tested?
CometIcebergWriteDetectionSuitecover:external-idandsession-name;1,tandonspellings of the flags;S3FileIO, a subclass,HadoopFileIOand an unresolved class.sts:AssumeRole, the configuration above):CometIcebergWritewith no fallback, on a single pod and on 3 executors in client mode (12 write tasks, 0 failures). Reading back 200,000 rows gives the same count, sum and distinct count as written.reqsign) all succeeding against the configured role.Two follow-ups found during the end-to-end test, not changed here:
AWS_REGIONin the environment, reading the table goes tous-east-1and fails with status 400. Pods that setAWS_REGIONare not affected.