Repository navigation
Read Glue REST tables with Glue's vended credentials - #47
Merged
Merged
Conversation
AWS Glue's Iceberg REST endpoint vends short-lived S3 credentials with every table, through Lake Formation, but bergr read Glue tables' files with the user's own AWS credentials: its custom credential loader replaces opendal's whole credential chain, including keys in the table's config. So tables granted only through Lake Formation couldn't be read. - Give the Glue REST catalog a storage factory that leaves out the custom loader when a table's config has vended keys, and keeps bergr's own credential chain as the fallback. - List files for `files --verify` with an S3 client built from the vended keys, when there are any. - Fall back to per-file existence checks when listing fails: vended keys can't always list (Polaris's can't), and listing is only an optimisation. Fixes #44
mdub
marked this pull request as ready for review
November 15, 2026 22:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bergr restagainst AWS Glue's Iceberg REST endpoint now reads table files with the short-lived S3 credentials that Glue vends for each table, instead of your own AWS credentials. So tables you can read through Lake Formation, but not directly in S3, now work. Fixes #44.Why it didn't work
Glue's
loadTableresponse always includess3.access-key-id,s3.secret-access-keyands3.session-tokenin the table'sconfig, whether or not the request asks for them. bergr ignored these keys. Its Glue path gives opendal's S3 storage a custom credential loader, and opendal'scredential_provider_chainreplaces the whole credential chain with that loader, keys in the table's config included.Changes
VendedFirstS3StorageFactory(src/aws.rs) wrapsOpenDalStorageFactory::S3, and the Glue REST catalog now uses it. It decides per table: when the table's config has vended keys, it builds the storage without the custom loader, so opendal uses them; otherwise it uses bergr's own credential chain as before.StorageFactoryis atypetagtrait, so the wrapper derives serde, skipping the process-local loader, andtypetagbecomes a direct dependency. It was already in the lockfile.files --verifylists data files with an S3 client built from the vended keys (vended_s3_client), when there are any. The client reads region, endpoint and path-style access from the table's properties, as opendal does.create_existence_checkernow falls back to per-file checks when the bulkListObjectsV2fails, and logs a warning. Listing is only an optimisation, and vended keys can't always list: Polaris's quickstart keys can read objects but getAccessDeniedonListObjectsV2. Without the fallback, the change above would have broken--verifyagainst Polaris.The Glue path still doesn't send
X-Iceberg-Access-Delegation, since Glue vends either way. Plainbergr glue(the Glue API) is unchanged.Testing
cargo fmt,cargo clippy --all-targets -- -D warningsandcargo test(73 tests) pass.main:filesandfiles --verifyfail withInvalidAccessKeyId.--verifystill uses the bulk listing, so the vended keys allowListObjectsV2.filesand--verifywork as before.files --verifyfalls back to per-file checks after the denied listing, and finds every file.Not tested: a principal that can only reach the data through Lake Formation. The bogus-key test exercises the same path.