Skip to content

fix: read table files from non-Glue REST catalogs, using vended credentials - #42

Merged
mdub merged 1 commit into
mainfrom
mdub-20261111-rest-vended-credentials
Nov 11, 2026
Merged

mdub merged 1 commit into
mainfrom
mdub-20261111-rest-vended-credentials

Conversation

@mdub

@mdub mdub commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

bergr rest … table … now works for REST catalogs other than AWS Glue's. It asks the catalog for vended credentials by default, and falls back to your own storage credentials when the catalog doesn't vend any. Closes #41.

Before, every table command failed with "StorageFactory must be provided for REST catalog table operations". Since iceberg 0.11, RestCatalog needs a storage factory to load a table, and only the Glue path supplied one.

What changed:

  • The plain REST catalog gets an OpenDalResolvingStorageFactory, which picks storage by each path's scheme, and reads table files with the storage settings the catalog returns with each table (s3.endpoint, path-style access, region, and credentials).
  • bergr sends X-Iceberg-Access-Delegation: vended-credentials, as PyIceberg does by default.
  • --access-delegation none (or BERGR_REST_ACCESS_DELEGATION=none) sends the spec's none value instead. The option uses the spec's own values; remote-signing isn't offered, since iceberg-rust doesn't implement it.
  • The Glue path is unchanged: it still reads table files with your AWS credentials.

Testing, against the Apache Polaris quickstart, with two snapshots written by PyIceberg:

  • With the defaults and no S3 configuration, schemas, snapshots, snapshot current info, files and files --verify all work.
  • Vended credentials win over (bogus) AWS credentials in the environment.
  • With --access-delegation none, AWS credentials from environment variables or a profile file work, and wrong ones fail with the object store's 401.
  • New mock-server tests check the header for both values, and that a table's FileIO gets the vended keys.

Things to be aware of:

  • With none and no credentials anywhere, opendal probes the EC2 metadata service before failing, which adds about 1.5 s. bergr's own AWS credential chain skips that probe; avoiding it here would need a custom credential loader, which didn't seem worth it.
  • Not supported yet: explicit storage settings for catalogs that don't return them, credential refresh (bergr's runs are short), and GCS or Azure storage (which need opendal features bergr doesn't enable).

…ntials

Every `bergr rest … table …` command failed for REST catalogs other
than AWS Glue's ("StorageFactory must be provided for REST catalog
table operations"). Since iceberg 0.11, `RestCatalog` needs a storage
factory to load a table, and only the Glue path supplied one.

The plain REST catalog now gets an `OpenDalResolvingStorageFactory`,
which picks storage by each path's scheme, and reads table files with
the storage settings the catalog returns with each table.

It also asks the catalog for vended credentials, by sending
`X-Iceberg-Access-Delegation: vended-credentials`, as PyIceberg does by
default. When the catalog vends nothing, opendal's default credential
chain falls back to the user's own (for S3, AWS environment variables
or profile); when both exist, vended credentials win.

`--access-delegation none` (or BERGR_REST_ACCESS_DELEGATION=none) sends
the spec's `none` value instead, for catalogs that shouldn't be asked.
The option takes the spec's own values; `remote-signing` isn't offered,
since iceberg-rust doesn't implement it. The Glue path is unchanged.

Closes #41
@mdub

mdub commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Glue regression check: I ran main and this branch against a real AWS Glue Iceberg REST catalog, on a table with 29 snapshots and 444 data files. schemas, snapshots, snapshot current info, files and files --verify all succeed with both builds, and their output matches.

files output is byte-identical. The others match once sorted: bergr prints schemas, snapshots and snapshot-summary keys in an order that varies from run to run, on main too. That's pre-existing, and I'll raise it separately.

@mdub
mdub marked this pull request as ready for review November 11, 2026 04:08
@mdub
mdub merged commit 62581f4 into main Nov 11, 2026
4 checks passed
@mdub
mdub deleted the mdub-20261111-rest-vended-credentials branch November 11, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bergr rest … table … fails for REST catalogs other than AWS Glue's (no storage factory)

1 participant