Skip to content

bergr rest … table … fails for REST catalogs other than AWS Glue's (no storage factory) #41

Closed

Description

@mdub

bergr rest can list namespaces and tables in any REST catalog, but every table command fails unless the catalog is AWS Glue's:

$ bergr rest --auth oauth2 http://localhost:8181/api/catalog --warehouse quickstart_catalog table bergr_demo.events schemas
ERROR: could not load table 'bergr_demo.events'
...
Caused by:
    Unexpected => StorageFactory must be provided for REST catalog table operations. Use `with_storage_factory` to configure it.

That run was against the Apache Polaris quickstart, while testing #40, but the cause isn't specific to Polaris or to authentication. Since iceberg 0.11, RestCatalog needs a storage factory before it can load a table. aws_rest_catalog (the Glue path) passes one, but rest_catalog (src/rest.rs) never has, on main or in #40.

Possible fix

  • Give the plain REST catalog a storage factory too. iceberg-storage-opendal 0.11 has OpenDalResolvingStorageFactory, which picks the storage implementation from each path's scheme (s3://, file://, and others, if their features are enabled), so bergr rest needn't know the table's storage up front.
  • That alone gets a table's metadata loading, but storage still needs configuring and credentials:
    • Vended credentials: send X-Iceberg-Access-Delegation: vended-credentials (iceberg-catalog-rest's header.* properties), and the catalog returns short-lived storage credentials and settings with each table. Polaris, Unity Catalog and others support this, and it's how most clients use these catalogs.
    • Ambient credentials: fall back to the user's AWS configuration for s3:// paths, as bergr glue does.
    • Explicit settings, such as an S3 endpoint and path-style access for a local object store (the Polaris quickstart uses one at http://localhost:9000).

Which of these bergr should support, and how a user chooses (flag, environment variable, or automatic), is open. #38 listed vended credentials as out of scope for authentication, and suggested its own flag.

To reproduce

Run the Polaris quickstart docker-compose, create a namespace and a table with curl (the setup container prints the commands), then:

BERGR_REST_OAUTH2_CREDENTIAL=<client-id>:<client-secret> BERGR_REST_OAUTH2_SCOPE=PRINCIPAL_ROLE:ALL \
  bergr rest --auth oauth2 http://localhost:8181/api/catalog --warehouse quickstart_catalog \
  table <namespace>.<table> schemas

(--auth needs #40. Before it, use any REST catalog that doesn't need authentication.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions