Skip to content

Support refreshing vended storage credentials for REST catalog tables #2931

Open

Description

@zakariya-s

Is your feature request related to a problem or challenge?

REST catalogs can vend short-lived storage credentials for accessing table data. Iceberg Java supports refreshing these credentials through cloud-specific refresh endpoints before they expire.

iceberg-rust does not currently refresh per-prefix credentials. Consequently, long-running reads, writes, and maintenance operations can fail once the initial S3 or GCS credentials expire.

Refresh support must preserve per-prefix credential isolation, table-scoped catalog authentication, and support tables referencing multiple storage prefixes or clouds.

This issue tracks the work required across the core storage API, OpenDAL backends, and REST catalog. The initial reference implementation was developed and reviewed in #2932.

Describe the solution you'd like

Add refreshable storage credential support across the REST catalog, FileIO, and storage backends:

  • Introduce a backend-independent credential-provider interface for FileIO
  • Use the standard AWS and GCS refresh endpoint properties returned by REST catalogs
  • Cache credentials per cloud and select the longest matching storage prefix

Tasks

  • Core credential-provider API, with OpenDAL S3 and GCS support (feat(io)!: [1/N] support refreshable storage credentials for S3 and GCS #2976)

    • StorageCredentialProvider, StorageCredential, and FileIO support, including serialization.
    • Adapt vended credentials into reqsign's AWS and Google providers, reloading them before expiry.
    • Keep bulk deletes within one credential scope.
  • OpenDAL ADLS support (depends on feat(io)!: [1/N] support refreshable storage credentials for S3 and GCS #2976)

    • Bump OpenDAL to 0.59 and adapt SAS tokens into reqsign's Azure provider.
    • Support account-specific adls.sas-token.<account> properties.
  • REST credential-refresh provider and catalog wiring (depends on feat(io)!: [1/N] support refreshable storage credentials for S3 and GCS #2976)

    • Fetch and parse credentials from catalog-provided refresh endpoints, with table-scoped authentication.
    • Cache credentials by cloud and select the longest matching prefix.
    • Prefetch before expiry with jitter, and back off on failure.
    • Build the provider from table properties and attach it to the table's FileIO.
  • Independent fixes

    • gcs.no-auth enables anonymous access only for a true value.
    • The GCS backend accepts gcs:// paths, which the resolving storage already routes to it.
    • Redact secret-bearing storage and catalog configuration in Debug output.

Willingness to contribute

I can contribute to this feature independently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions