You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
No refreshing Iceberg credential provider on Spark 3.x builds #7389
Is your feature request related to a problem or challenge?
#7363 adds IcebergVendedS3CredentialProvider in the spark-4.x source set only, because it links against AWS SDK v2. Iceberg 1.8.1, which the Spark 3.5 build uses, already has the same refresh logic in AwsClientProperties, and Comet loads Iceberg by reflection. On Spark 3.x, a long native Iceberg scan against a REST catalog with short-lived credentials still fails with ExpiredToken.
Describe the solution you'd like
Ship the provider in Spark 3.x builds too. Iceberg's provider is SDK v2 there as well, so the class would need its own body in the spark-3.x source set, or a loader-neutral way to read the SDK v2 credentials.
Additional context
The docs added in #7363 state that Spark 3.x builds don't include the provider.
Activity