Skip to content

Use AWS Glue's vended credentials for bergr rest against Glue #44

Closed

Description

@mdub

bergr rest against AWS Glue's Iceberg REST endpoint reads table files with your own AWS credentials, even though Glue returns short-lived storage credentials with every table. So it needs direct S3 read permission on the table's location, and can't read tables that you're granted through Lake Formation alone.

What happens now

  • Glue's loadTable response includes s3.access-key-id, s3.secret-access-key and s3.session-token in config, whether or not the request sends X-Iceberg-Access-Delegation: vended-credentials. These appear to be vended through Lake Formation, which is why Glue's REST endpoint needs lakeformation:GetDataAccess.
  • bergr's Glue path (aws_rest_catalog in src/aws.rs) gives opendal's S3 storage a custom credential loader, which supplies your AWS credentials. A custom loader overrides static keys from the table's config, so the vended keys are never used. I checked that against another catalog: with working vended keys and bogus keys from the loader, reads fail.
  • snapshot … files --verify on Glue also lists files with a separate S3 client, using your credentials.

Other REST catalogs, since #42, ask for vended credentials and use them in preference to your own.

Proposal

On the Glue path, read table files with the keys Glue vends, falling back to your AWS credentials when there are none:

  • Stop giving the REST catalog's storage a custom credential loader, as the plain REST path does. opendal's default chain still finds your AWS credentials when the table's config has no keys, though it then also probes the EC2 metadata service. Alternatively, write a loader that prefers keys from config.
  • Make files --verify use the table's FileIO, or an S3 client built from the vended keys, rather than one built from your credentials.
  • Possibly send X-Iceberg-Access-Delegation: vended-credentials too, honouring --access-delegation, for consistency with other catalogs, even though Glue doesn't seem to need it.

Testing needs a Glue table that you can read through Lake Formation but not directly in S3. Before the change, files should fail; after it, it should work.

bergr glue uses the Glue API rather than the REST endpoint, so it isn't affected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions