Skip to content

feat: support AWS Glue's Iceberg REST endpoint (SigV4) - #37

Merged
mdub merged 2 commits into
mainfrom
mdub-20261110-glue-rest-sigv4
Nov 10, 2026
Merged

mdub merged 2 commits into
mainfrom
mdub-20261110-glue-rest-sigv4

Conversation

@mdub

@mdub mdub commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

bergr rest can now talk to AWS Glue's Iceberg REST endpoint, which requires every request to be signed with AWS SigV4. Closes #3.

bergr rest https://glue.us-west-2.amazonaws.com/iceberg --warehouse <account-id> namespaces

This is a stopgap. SigV4 support for REST catalogs is under way upstream (apache/iceberg-rust#1236), but iceberg-rust 0.11 only ships the pluggable auth traits (#2838), not a signer. So src/sigv4.rs is our own minimal AuthManager, built on aws-sigv4. Unlike upstream's signer, it doesn't reproduce Iceberg Java's signing quirks. Once upstream's SigV4AuthManager (#3092) is released, we should delete ours and use that.

Two commits:

  • fix: enable TLS for REST catalog requests: bergr rest has never worked with an https:// URI ("invalid URL, scheme is not http"). iceberg-catalog-rest depends on reqwest 0.12 with no TLS backend, and nothing else in our tree enabled one. We now depend on reqwest 0.12 directly with rustls-tls-native-roots. This isn't specific to Glue, but Glue needs it.

  • feat: sign requests to AWS Glue's REST catalog with SigV4: when the URI is glue.<region>.amazonaws.com, bergr rest:

    • signs requests using the same AWS credentials as bergr glue;
    • gives the catalog our S3 storage factory, which the REST catalog needs in order to load tables;
    • uses the URI's region for AWS calls.

    Other REST URIs behave as before.

I tested it against a real Glue catalog. Namespaces, tables, snapshots and snapshot current files --verify all match bergr glue, including when the profile's default region differs from the URI's.

Things to be aware of:

  • Glue's REST endpoint needs more IAM permissions than the Glue API: glue:GetCatalog, and lakeformation:GetDataAccess to load tables. The README says so.
  • For the table I tested, Glue's REST endpoint returned older schemas with some later-added columns missing, compared with the table's metadata.json. The current schema matched. That's Glue's behaviour, not bergr's, but it means bergr rest … schemas and bergr glue … schemas can disagree on old schema versions.
  • S3 Tables' REST endpoint (s3tables.<region>.amazonaws.com) isn't supported yet. It should only need a different signing name, but I couldn't test it.

mdub added 2 commits November 10, 2026 12:47
`bergr rest` failed against any `https://` catalog URI with "invalid URL,
scheme is not http". iceberg-catalog-rest depends on reqwest 0.12 with
default features off and no TLS backend, and nothing else in our
dependency tree enabled one.

Depend on reqwest 0.12 directly with `rustls-tls-native-roots`, so cargo's
feature unification gives iceberg's client TLS. Native roots, rather than
the bundled webpki roots, so it works behind TLS-inspecting proxies that
install their own root certificate.
AWS Glue's Iceberg REST endpoint requires every request to be signed
with AWS SigV4, so `bergr rest` couldn't talk to it.

When the REST catalog URI is a `glue.<region>.amazonaws.com` endpoint,
`bergr rest` now:

- signs each request with SigV4, via an `AuthManager` (the pluggable auth
  added in iceberg-catalog-rest 0.11), using the same AWS credentials as
  the `glue` command;
- gives the catalog bergr's S3 storage factory, which the REST catalog
  needs before it can load tables;
- uses the region from the URI for AWS calls, so it doesn't depend on the
  profile's default region.

This is a stopgap. iceberg-rust has SigV4 support for REST catalogs in
progress (apache/iceberg-rust#1236), but the released 0.11 has only the
pluggable auth, not a signer. Our signer is deliberately minimal, and
should be replaced by upstream's `SigV4AuthManager`
(apache/iceberg-rust#3092) once it ships.

Closes #3
@mdub
mdub merged commit 86de208 into main Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for AWS Glue's implementation of Iceberg REST Catalog API

1 participant