You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use AWS Glue's vended credentials for bergr rest against Glue #44
bergr rest against AWS Glue's Iceberg REST endpoint reads table files with your own AWS credentials, even though Glue returns short-lived storage credentials with every table. So it needs direct S3 read permission on the table's location, and can't read tables that you're granted through Lake Formation alone.
What happens now
Glue's loadTable response includes s3.access-key-id, s3.secret-access-key and s3.session-token in config, whether or not the request sends X-Iceberg-Access-Delegation: vended-credentials. These appear to be vended through Lake Formation, which is why Glue's REST endpoint needs lakeformation:GetDataAccess.
bergr's Glue path (aws_rest_catalog in src/aws.rs) gives opendal's S3 storage a custom credential loader, which supplies your AWS credentials. A custom loader overrides static keys from the table's config, so the vended keys are never used. I checked that against another catalog: with working vended keys and bogus keys from the loader, reads fail.
snapshot … files --verify on Glue also lists files with a separate S3 client, using your credentials.
Other REST catalogs, since #42, ask for vended credentials and use them in preference to your own.
Proposal
On the Glue path, read table files with the keys Glue vends, falling back to your AWS credentials when there are none:
Stop giving the REST catalog's storage a custom credential loader, as the plain REST path does. opendal's default chain still finds your AWS credentials when the table's config has no keys, though it then also probes the EC2 metadata service. Alternatively, write a loader that prefers keys from config.
Make files --verify use the table's FileIO, or an S3 client built from the vended keys, rather than one built from your credentials.
Possibly send X-Iceberg-Access-Delegation: vended-credentials too, honouring --access-delegation, for consistency with other catalogs, even though Glue doesn't seem to need it.
Testing needs a Glue table that you can read through Lake Formation but not directly in S3. Before the change, files should fail; after it, it should work.
bergr glue uses the Glue API rather than the REST endpoint, so it isn't affected.
Activity