Repository navigation
fix: read table files from non-Glue REST catalogs, using vended credentials - #42
Merged
Merged
Conversation
…ntials
Every `bergr rest … table …` command failed for REST catalogs other
than AWS Glue's ("StorageFactory must be provided for REST catalog
table operations"). Since iceberg 0.11, `RestCatalog` needs a storage
factory to load a table, and only the Glue path supplied one.
The plain REST catalog now gets an `OpenDalResolvingStorageFactory`,
which picks storage by each path's scheme, and reads table files with
the storage settings the catalog returns with each table.
It also asks the catalog for vended credentials, by sending
`X-Iceberg-Access-Delegation: vended-credentials`, as PyIceberg does by
default. When the catalog vends nothing, opendal's default credential
chain falls back to the user's own (for S3, AWS environment variables
or profile); when both exist, vended credentials win.
`--access-delegation none` (or BERGR_REST_ACCESS_DELEGATION=none) sends
the spec's `none` value instead, for catalogs that shouldn't be asked.
The option takes the spec's own values; `remote-signing` isn't offered,
since iceberg-rust doesn't implement it. The Glue path is unchanged.
Closes #41
Collaborator
Author
|
Glue regression check: I ran
|
This was referenced Nov 11, 2026
mdub
marked this pull request as ready for review
November 11, 2026 04:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bergr rest … table …now works for REST catalogs other than AWS Glue's. It asks the catalog for vended credentials by default, and falls back to your own storage credentials when the catalog doesn't vend any. Closes #41.Before, every
tablecommand failed with "StorageFactory must be provided for REST catalog table operations". Since iceberg 0.11,RestCatalogneeds a storage factory to load a table, and only the Glue path supplied one.What changed:
OpenDalResolvingStorageFactory, which picks storage by each path's scheme, and reads table files with the storage settings the catalog returns with each table (s3.endpoint, path-style access, region, and credentials).X-Iceberg-Access-Delegation: vended-credentials, as PyIceberg does by default.--access-delegation none(orBERGR_REST_ACCESS_DELEGATION=none) sends the spec'snonevalue instead. The option uses the spec's own values;remote-signingisn't offered, since iceberg-rust doesn't implement it.Testing, against the Apache Polaris quickstart, with two snapshots written by PyIceberg:
schemas,snapshots,snapshot current info,filesandfiles --verifyall work.--access-delegation none, AWS credentials from environment variables or a profile file work, and wrong ones fail with the object store's 401.Things to be aware of:
noneand no credentials anywhere, opendal probes the EC2 metadata service before failing, which adds about 1.5 s. bergr's own AWS credential chain skips that probe; avoiding it here would need a custom credential loader, which didn't seem worth it.