Skip to content

feat: admit S3FileIO's AssumeRoleAwsClientFactory for native Iceberg writes - #7680

Open
surawut-jirasaktavee wants to merge 1 commit into
apache:mainfrom
surawut-jirasaktavee:iceberg-write-assume-role-client-factory
Open

surawut-jirasaktavee wants to merge 1 commit into
apache:mainfrom
surawut-jirasaktavee:iceberg-write-assume-role-client-factory

Conversation

@surawut-jirasaktavee

@surawut-jirasaktavee surawut-jirasaktavee commented Nov 18, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7667.

Rationale for this change

The Iceberg docs' assume-role setup for Spark on EKS (io-impl=S3FileIO, client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory, client.assume-role.arn, client.assume-role.region) makes every native Iceberg write fall back:

IcebergWrite [COMET: unsupported S3 FileIO settings: client.assume-role.region, client.factory]

The pinned iceberg-rust already reads client.assume-role.arn, external-id and session-name. The fallback comes from the two keys iceberg-java uses to turn that path on: client.factory, without which iceberg-java ignores every client.assume-role.* key, and client.assume-role.region, which the pinned iceberg-rust does not read.

What changes are included in this PR?

  • client.factory and client.assume-role.arn, external-id, session-name and region are admitted together, only when all of these hold:
    • the factory is exactly org.apache.iceberg.aws.AssumeRoleAwsClientFactory;
    • client.assume-role.arn and client.assume-role.region are set and not blank;
    • the data location is written through S3FileIO itself, including a ResolvingFileIO that resolves to it. A subclass of S3FileIO can ignore the factory, and HadoopFileIO ignores it and writes through S3A, so both fall back.
  • Without the factory, client.assume-role.arn, external-id and session-name now fall back. Before this PR they were admitted on their own, and the native writer assumed a role that iceberg-java ignores.
  • The assume-role region is passed to the native writer as client.region when neither client.region nor s3.region is set.
  • When client.assume-role.session-name is not set, the native writer names the session iceberg-aws-<uuid>, as the factory does, instead of opendal's reqsign.
  • With the factory, the write still falls back on settings the native writer would read but the factory's S3 client ignores, and the fallback reason names the key:
    • a client.region or s3.region different from the assume-role region;
    • s3.access-key-id, s3.secret-access-key or s3.session-token (opendal would use them as the source credentials for AssumeRole);
    • a Comet credential provider class;
    • s3.allow-anonymous, s3.disable-ec2-metadata or s3.disable-config-load set to a value iceberg-rust reads as true (true, t, 1 or on, in any case).
  • Other assume-role keys the native writer cannot honour, such as client.assume-role.tags.* and client.assume-role.timeout-sec, still fall back.
  • The Iceberg user guide describes the rule and one remaining difference: the native writer calls STS in the assume-role region, while the factory's STS client takes its region from the default AWS region chain.

How are these changes tested?

  • New unit tests in CometIcebergWriteDetectionSuite cover:
    • the admitted configuration, with and without external-id and session-name;
    • the assume-role keys falling back without the factory;
    • each conflicting key, including the 1, t and on spellings of the flags;
    • other factory classes, and a missing or blank arn or region;
    • the region and session-name mapping;
    • which resolved FileIO class takes the factory: S3FileIO, a subclass, HadoopFileIO and an unresolved class.
  • The suite passes (96 tests).
  • End to end on EKS, run before the review changes (Spark 4.1, Iceberg 1.11, Glue catalog, Pod Identity holding only sts:AssumeRole, the configuration above):
    • Comet 1.2.0 falls back with the message above.
    • With this branch the plan uses CometIcebergWrite with no fallback, on a single pod and on 3 executors in client mode (12 write tasks, 0 failures). Reading back 200,000 rows gives the same count, sum and distinct count as written.
    • CloudTrail shows the native AssumeRole calls (session name reqsign) all succeeding against the configured role.

Two follow-ups found during the end-to-end test, not changed here:

  • The native Iceberg scan does not get the assume-role region. Without AWS_REGION in the environment, reading the table goes to us-east-1 and fails with status 400. Pods that set AWS_REGION are not affected.
  • The native path makes one STS AssumeRole call per data file (280 calls for 140 files written and read back), where iceberg-java reuses one session per client. Large writes could hit STS throttling.

@surawut-jirasaktavee
surawut-jirasaktavee force-pushed the iceberg-write-assume-role-client-factory branch 2 times, most recently from d57018d to cd74716 Compare November 18, 2026 17:46

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The factory, ARN, region and credential-conflict handling line up with iceberg-java's AssumeRoleAwsClientFactory and the pinned iceberg-rust/opendal S3 parser. I found one gate/native mismatch in the flag check, one existing hole this PR makes visible, an undocumented STS/session-name difference, and a gap in test coverage of the new rule.

NativeS3RegionProperties.filter(properties.get(_).exists(_ != region)) ++
AssumeRoleConflictingCredentialProperties.filter(properties.contains) ++
AssumeRoleConflictingCredentialFlags.filter(
properties.get(_).exists(java.lang.Boolean.parseBoolean))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boolean.parseBoolean only accepts true, but the pinned iceberg-rust reads these three flags with is_truthy (crates/storage/opendal/src/s3.rs and utils.rs at de0a6ac), which accepts true, t, 1 and on case-insensitively. So s3.allow-anonymous=1, s3.disable-config-load=on or s3.disable-ec2-metadata=t pass this check, iceberg-java reads them as false, and the native writer then skips signing or drops the env/IMDS credential sources that AssumeRole needs as its source. The write fails in the task instead of falling back. Please use the same truthiness as iceberg-rust here and add those spellings to the test, which only uses true.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. s3.allow-anonymous, s3.disable-ec2-metadata and s3.disable-config-load now use the same rule as iceberg-rust's is_truthy (true, t, 1, on, case-insensitive), so any of those spellings falls back with the factory. The test now also covers 1, T and on

.get(CometS3CredentialProviderClassProperty)
.exists(_.trim.nonEmpty)

val assumeRole = usesAssumeRoleClientFactory(properties)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without client.factory, iceberg-java ignores client.assume-role.* (AwsClientFactories.from defaults to DefaultAwsClientFactory, and a HadoopFileIO write goes through S3A). But client.assume-role.arn, external-id and session-name are in SupportedS3FileIOProperties unconditionally, and the test at unsupported(assumeRole - "client.factory") == Seq("client.assume-role.region") shows that an ARN without the factory is still admitted when no region is set. The native writer then assumes the role while the JVM writer would have used the default credentials, so the write happens under a different identity. This predates the PR, but the PR description states the rule and this is the place to enforce it. Consider declining the ARN, external id and session name unless assumeRole holds and the FileIO is S3FileIO.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed here. client.assume-role.arn, external-id and session-name are no longer in SupportedS3FileIOProperties. They are admitted only together with client.factory=AssumeRoleAwsClientFactory, a non-blank arn and region, and S3FileIO, so an ARN without the factory now falls back. I updated the existing supported-keys test and added a case for the assume-role keys without the factory

Comment on lines +574 to +578
With `S3FileIO`, `client.factory` is admitted only as `org.apache.iceberg.aws.AssumeRoleAwsClientFactory`
with `client.assume-role.arn` and `client.assume-role.region` set, and that region becomes the write
region. The write falls back when the factory is combined with a different `client.region` or
`s3.region`, static S3 keys, a credential-provider class, or `s3.allow-anonymous`,
`s3.disable-ec2-metadata` or `s3.disable-config-load` set to `true`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two observable differences from the factory are not listed. First, iceberg-java names the session iceberg-aws-<uuid> when client.assume-role.session-name is unset (AssumeRoleAwsClientFactory.genSessionName), while opendal/reqsign sends reqsign (your CloudTrail output shows it). A trust policy with an sts:RoleSessionName condition that works for the JVM writer will deny the native one, and the query fails instead of falling back. Second, the native STS call goes to the regional endpoint of client.assume-role.region, whereas the factory's STS client takes its region from the default provider chain (sts() sets none). I expect this matters only with a restricted STS route, but I have not tested it. Please add both to the accepted divergences, or decline when a trust-policy-sensitive setup cannot be ruled out.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Session name: when client.assume-role.session-name is unset, the native writer now gets iceberg-aws-<uuid>, the same form as AssumeRoleAwsClientFactory.genSessionName. iceberg-rust passes it to opendal as role_session_name, so a trust policy with an sts:RoleSessionName condition sees the same pattern from both writers. One difference remains: iceberg-java generates one name per factory instance, while the native writer generates one per write

Comment on lines +1152 to +1162
lazy val s3FileIO = IcebergReflection
.getFileIO(ctx.table)
.flatMap(io => IcebergReflection.resolveFileIOClass(io, location))
.exists(cls =>
IcebergReflection
.classNameInHierarchy(cls, Set(IcebergReflection.ClassNames.S3_FILE_IO)))
unsupportedSettingsReason(
"S3 FileIO",
unsupportedS3FileIOProperties(
ctx.fileIOProperties,
ctx.fileIOProperties.contains(S3FileIOClientFactoryProperty) && s3FileIO))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule that decides s3FileIO is not covered. The new tests call unsupportedS3FileIOProperties(..., s3FileIO = true/false) directly, so the HadoopFileIO fallback test only proves a boolean parameter, not that a ResolvingFileIO resolving to HadoopFileIO, a failed resolveFileIOClass, or an S3FileIO subclass lands on the right side. gcsDataLocationRejection takes the resolved Option[Class[_]] for this reason. Consider the same shape here so a test can feed S3FileIO, HadoopFileIO and None. Also note classNameInHierarchy admits any subclass of S3FileIO, which can override client creation and ignore the factory. An exact class match would fail closed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The decision is now isS3FileIO(resolvedFileIO: Option[Class[_]]), the same shape as gcsDataLocationRejection. It matches org.apache.iceberg.aws.s3.S3FileIO exactly instead of using classNameInHierarchy, so a subclass falls back. A new test checks S3FileIO, an S3FileIO subclass, HadoopFileIO and None

@surawut-jirasaktavee
surawut-jirasaktavee force-pushed the iceberg-write-assume-role-client-factory branch 2 times, most recently from d461558 to e3b6e05 Compare November 20, 2026 04:58
…writes

iceberg-java turns on its assume-role path only through
client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory, and its S3
client then takes its region from client.assume-role.region, a key the pinned
iceberg-rust does not read. Both keys made every such write fall back.

Admit the factory together with the client.assume-role.* settings when the data
location is written through S3FileIO itself and client.assume-role.arn and
client.assume-role.region are set. Without the factory, iceberg-java ignores
the assume-role settings, so they now fall back. Pass the region to the native
writer as client.region and, when no session name is set, name the session
iceberg-aws-<uuid> as the factory does. Still fall back when the factory is
combined with settings the native writer would read and the factory's S3
client ignores: a different client.region or s3.region, static S3 keys, a
Comet credential provider class, or s3.allow-anonymous,
s3.disable-ec2-metadata or s3.disable-config-load set to a value iceberg-rust
reads as true.
@surawut-jirasaktavee
surawut-jirasaktavee force-pushed the iceberg-write-assume-role-client-factory branch from e3b6e05 to e2df0d1 Compare November 20, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Iceberg writes: support S3FileIO's AssumeRoleAwsClientFactory (client.factory + client.assume-role.region)

2 participants