Skip to content

S3 credential providers are retained until JVM shutdown when a catalog re-vends credentials #7390

Closed

Description

@laxmanmamididd

Describe the bug

Comet keys provider instances by (FQCN, dispatchKey, catalogProperties). When an Iceberg REST catalog vends new credentials for the same table, the FileIO properties change, so Comet creates a new provider instance under a new key. The old instance stays cached, and close() runs only from the JVM shutdown hook.

Reported in review of #7363: three credential snapshots against one refresh endpoint and the same dispatch key left three providers retained. Each Iceberg VendedCredentialsProvider owns an HTTP client and an auth session, so a long-lived executor that reloads tables (Spark Connect, Thrift Server) keeps accumulating them.

Expected behavior

The dispatcher retires a provider once a newer instance supersedes it for the same (FQCN, dispatchKey) and its active readers have finished, then calls close().

Additional context

This affects every provider, not only IcebergVendedS3CredentialProvider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Icebergarea:scanParquet scan / data readingbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions