You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
S3 credential providers are retained until JVM shutdown when a catalog re-vends credentials #7390
Comet keys provider instances by (FQCN, dispatchKey, catalogProperties). When an Iceberg REST catalog vends new credentials for the same table, the FileIO properties change, so Comet creates a new provider instance under a new key. The old instance stays cached, and close() runs only from the JVM shutdown hook.
Reported in review of #7363: three credential snapshots against one refresh endpoint and the same dispatch key left three providers retained. Each Iceberg VendedCredentialsProvider owns an HTTP client and an auth session, so a long-lived executor that reloads tables (Spark Connect, Thrift Server) keeps accumulating them.
Expected behavior
The dispatcher retires a provider once a newer instance supersedes it for the same (FQCN, dispatchKey) and its active readers have finished, then calls close().
Additional context
This affects every provider, not only IcebergVendedS3CredentialProvider.
Activity