Repository navigation
feat: support AWS Glue's Iceberg REST endpoint (SigV4) - #37
Merged
Merged
Conversation
`bergr rest` failed against any `https://` catalog URI with "invalid URL, scheme is not http". iceberg-catalog-rest depends on reqwest 0.12 with default features off and no TLS backend, and nothing else in our dependency tree enabled one. Depend on reqwest 0.12 directly with `rustls-tls-native-roots`, so cargo's feature unification gives iceberg's client TLS. Native roots, rather than the bundled webpki roots, so it works behind TLS-inspecting proxies that install their own root certificate.
AWS Glue's Iceberg REST endpoint requires every request to be signed with AWS SigV4, so `bergr rest` couldn't talk to it. When the REST catalog URI is a `glue.<region>.amazonaws.com` endpoint, `bergr rest` now: - signs each request with SigV4, via an `AuthManager` (the pluggable auth added in iceberg-catalog-rest 0.11), using the same AWS credentials as the `glue` command; - gives the catalog bergr's S3 storage factory, which the REST catalog needs before it can load tables; - uses the region from the URI for AWS calls, so it doesn't depend on the profile's default region. This is a stopgap. iceberg-rust has SigV4 support for REST catalogs in progress (apache/iceberg-rust#1236), but the released 0.11 has only the pluggable auth, not a signer. Our signer is deliberately minimal, and should be replaced by upstream's `SigV4AuthManager` (apache/iceberg-rust#3092) once it ships. Closes #3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bergr restcan now talk to AWS Glue's Iceberg REST endpoint, which requires every request to be signed with AWS SigV4. Closes #3.This is a stopgap. SigV4 support for REST catalogs is under way upstream (apache/iceberg-rust#1236), but iceberg-rust 0.11 only ships the pluggable auth traits (#2838), not a signer. So
src/sigv4.rsis our own minimalAuthManager, built onaws-sigv4. Unlike upstream's signer, it doesn't reproduce Iceberg Java's signing quirks. Once upstream'sSigV4AuthManager(#3092) is released, we should delete ours and use that.Two commits:
fix: enable TLS for REST catalog requests:bergr resthas never worked with anhttps://URI ("invalid URL, scheme is not http"). iceberg-catalog-rest depends on reqwest 0.12 with no TLS backend, and nothing else in our tree enabled one. We now depend on reqwest 0.12 directly withrustls-tls-native-roots. This isn't specific to Glue, but Glue needs it.feat: sign requests to AWS Glue's REST catalog with SigV4: when the URI isglue.<region>.amazonaws.com,bergr rest:bergr glue;Other REST URIs behave as before.
I tested it against a real Glue catalog. Namespaces, tables, snapshots and
snapshot current files --verifyall matchbergr glue, including when the profile's default region differs from the URI's.Things to be aware of:
glue:GetCatalog, andlakeformation:GetDataAccessto load tables. The README says so.metadata.json. The current schema matched. That's Glue's behaviour, not bergr's, but it meansbergr rest … schemasandbergr glue … schemascan disagree on old schema versions.s3tables.<region>.amazonaws.com) isn't supported yet. It should only need a different signing name, but I couldn't test it.