Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
97c60b1
WIP: Add support for configuring Iceberg REST catalog
sbernauer Oct 6, 2026
3c6dd88
fix: Accidential token/credentials swap
sbernauer Oct 8, 2026
36b1189
chore: nix stuff
sbernauer Oct 8, 2026
9b93545
fix: Store as v1alpha2
sbernauer Oct 8, 2026
d80e9e5
chore: Use v1alpha2 everywhere
sbernauer Oct 8, 2026
5af8d9e
user & pass -> clientId & clientSecret
sbernauer Oct 8, 2026
8d8c750
test: Add variant
sbernauer Oct 8, 2026
2baa4f4
regenerate charts
sbernauer Oct 8, 2026
3373167
Merge branch 'main' into feat/iceberg-rest-catalog-client
sbernauer Oct 8, 2026
b3e3e54
Rename security -> authentication
sbernauer Oct 9, 2026
a4e1d83
credentialSecretName -> credentialsSecretName for consistency
sbernauer Oct 9, 2026
a84519d
Merge branch 'main' into feat/iceberg-rest-catalog-client
sbernauer Oct 9, 2026
96844a5
test: Fix test
sbernauer Oct 9, 2026
eec3c71
fix: Try to get roundtrips to work
sbernauer Oct 9, 2026
a65c8bd
semicolon
sbernauer Oct 12, 2026
c153ce8
fmt
sbernauer Oct 12, 2026
1290db7
regenerate charts
sbernauer Oct 12, 2026
be0311a
Merge remote-tracking branch 'origin/main' into feat/iceberg-rest-cat…
sbernauer Oct 16, 2026
324bbd4
Merge remote-tracking branch 'origin/main' into feat/iceberg-rest-cat…
sbernauer Oct 19, 2026
c93b7f5
chore: Update op-rs feature branch
sbernauer Oct 19, 2026
1b9f0a8
refactor: Flatten credentials options
sbernauer Oct 20, 2026
a4c7ead
Merge branch 'main' into feat/iceberg-rest-catalog-client
sbernauer Oct 20, 2026
bfec62a
regenerate nix
sbernauer Oct 20, 2026
9ec4755
credentialsSecretName -> clientCredentialsSecretName
sbernauer Oct 20, 2026
96a27b0
regenerate CRD
sbernauer Oct 20, 2026
1a93e62
Merge branch 'main' into feat/iceberg-rest-catalog-client
sbernauer Oct 27, 2026
1b7c94c
bump op-rs
sbernauer Oct 30, 2026
7e6a42e
Merge remote-tracking branch 'origin/main' into feat/iceberg-rest-cat…
sbernauer Oct 30, 2026
c950aef
regenerate nix
sbernauer Oct 30, 2026
49a4a67
docs: Update docs
sbernauer Oct 30, 2026
c10d014
chore: Use v1alpha2 everywhere
sbernauer Oct 30, 2026
e38564d
test: Adopt tests
sbernauer Oct 30, 2026
d5a2f28
changelog
sbernauer Oct 30, 2026
c0e5113
docs: A few CRD docs improvements
sbernauer Oct 30, 2026
9f86a8b
refactor: Add helper function
sbernauer Oct 30, 2026
d9f11d5
test: v1alpha1 -> v1alpha2
sbernauer Oct 30, 2026
e9aedb9
docs: Clarify on teh oauth2 serverUri
sbernauer Oct 30, 2026
eedbc35
chore: Remodel URLs to hostname, port, path and TLS client details
sbernauer Oct 30, 2026
76f2f83
chore: Default TLS to webPki
sbernauer Oct 30, 2026
a08e9cb
feat: Remodel CRD to take scopes list
sbernauer Oct 30, 2026
7148bd1
test: Simplify test
sbernauer Oct 30, 2026
a18857d
docs: Improve docs
sbernauer Nov 2, 2026
df77f11
test: Add unit test for Iceberg REST catalog connection
sbernauer Nov 2, 2026
312b0d1
bump op-rs
sbernauer Nov 3, 2026
8dff9ad
chore: Allow Apache-2.0 WITH LLVM-exception license - #661
sbernauer Nov 3, 2026
e653279
Use op-rs 0.112.0
sbernauer Nov 3, 2026
d030733
Merge remote-tracking branch 'origin/main' into feat/iceberg-rest-cat…
sbernauer Nov 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ All notable changes to this project will be documented in this file.
- Add `/ready` endpoint to the operator Deployment, which reports the CRD installation status ([#949]).
- Coordinators now have a default affinity to the OPA Pods when OPA authorization is configured ([#924]).
- Add support for 483 ([#956]).
- Support configuring an Iceberg REST catalog in the Iceberg connector, optionally authenticated
using OAuth2 client credentials or a bearer token ([#950]).

### Changed

Expand Down Expand Up @@ -48,6 +50,11 @@ All notable changes to this project will be documented in this file.
- Support Vector 0.58.0 ([#953], [#959], [#960]).
- test: Bump hive to 4.2.1 ([#961]).
- The operator Deployment runs with a restricted security context and a read-only root filesystem, so it can be deployed into namespaces that enforce the `restricted` Pod Security Standard ([#963]).
- Add `TrinoCatalog` version `v1alpha2`, which is now the storage version. `v1alpha1` objects are
converted automatically by the conversion webhook, which requires the operator to maintain its
CRDs (the default). In `v1alpha2`, the optional `spec.connector.iceberg.metastore` is replaced
by the required `spec.connector.iceberg.catalog`, which is one of `rest`, `hiveMetastore` or
`userProvided` ([#950]).

### Fixed

Expand Down Expand Up @@ -82,6 +89,7 @@ All notable changes to this project will be documented in this file.
[#944]: https://github.com/stackabletech/trino-operator/pull/944
[#945]: https://github.com/stackabletech/trino-operator/pull/945
[#949]: https://github.com/stackabletech/trino-operator/pull/949
[#950]: https://github.com/stackabletech/trino-operator/pull/950
[#953]: https://github.com/stackabletech/trino-operator/pull/953
[#956]: https://github.com/stackabletech/trino-operator/pull/956
[#957]: https://github.com/stackabletech/trino-operator/pull/957
Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.nix

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion deploy/helm/trino-operator/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ annotations:
displayName: TrinoCluster
description: A Trino cluster stacklet. This resource is managed by the Stackable operator for Trino.
- kind: TrinoCatalog
version: v1alpha1
version: v1alpha2
name: trinocatalogs.trino.stackable.tech
displayName: TrinoCatalog
description: The TrinoCatalog resource can be used to define catalogs in Kubernetes objects.
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ stringData:
accessKey: minio-access-key
secretKey: minio-secret-key
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: tpch
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/trino/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
:github: https://github.com/stackabletech/trino-operator/
:crd: {crd-ref}{crd-ref-release}&operator=trino-operator
:crd-trinocluster: {crd-ref}/trino.stackable.tech/v1alpha1/trinocluster{crd-ref-release}
:crd-trinocatalog: {crd-ref}/trino.stackable.tech/v1alpha1/trinocatalog{crd-ref-release}
:crd-trinocatalog: {crd-ref}/trino.stackable.tech/v1alpha2/trinocatalog{crd-ref-release}
:feature-tracker: https://features.stackable.tech/unified
:k8s-crs: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ It works like the `/dev/null` device on Unix-like operating systems for data wri

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it will appear in Trino
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ It depends on a Hive metastore being present and makes use of the same metastore

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it will appear in Trino
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/trino/pages/usage-guide/catalogs/generic.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This is how you can e.g. use the {trino-psql-connector}[PostgreSQL connector]:

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it appears in Trino
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Consult the {trino-google-sheets-connector}[official documentation] on how to us

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: gsheet
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/trino/pages/usage-guide/catalogs/hive.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Deploy a Hive Stacklet with the xref:hive:index.adoc[].

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive-catalog # <1>
Expand Down Expand Up @@ -143,7 +143,7 @@ To use the unmanaged Hive metastore we define a TrinoCatalog object in the same

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: clouderahive
Expand Down
102 changes: 96 additions & 6 deletions docs/modules/trino/pages/usage-guide/catalogs/iceberg.adoc
Original file line number Diff line number Diff line change
@@ -1,15 +1,104 @@
= Apache Iceberg

Apache Iceberg is a format for huge analytic tables designed to address some of the scaling issues with traditional Hive tables.
Iceberg depends on a Hive metastore being present and makes use of the same metastore ConfigMap used by the Hive connector.
Iceberg stores table metadata in a metadata catalog, which is configured in `spec.connector.iceberg.catalog`.
The following catalogs are supported:

You can deploy a Hive Metastore Stacklet with the xref:hive:index.adoc[].
* `rest`: An https://trino.io/docs/current/object-storage/metastores.html#rest-catalog[Iceberg REST catalog]
* `hiveMetastore`: A Hive metastore, using the same metastore ConfigMap as the xref:usage-guide/catalogs/hive.adoc[] connector
* `userProvided`: The operator doesn't configure a catalog, you need to configure it yourself using `configOverrides`

== Example Iceberg REST catalog configuration

== Example Iceberg catalog configuration
[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: iceberg # <1>
labels:
trino: simple-trino
spec:
connector:
iceberg:
catalog:
rest:
hostname: iceberg-rest-catalog.example.com # <2>
port: 9001 # <3>
path: /catalog # <4>
tls: # <5>
verification:
server:
caCert:
secretClass: tls
authentication: # <6>
oAuth2:
server: # <7>
hostname: idp.example.com
path: /oauth2/token
clientCredentials: # <8>
secretName: trino-iceberg-client # <9>
scopes: # <10>
- lakehouse
s3:
inline:
host: test-minio
port: 9000
accessStyle: Path
credentials:
secretClass: minio-credentials
---
apiVersion: v1
kind: Secret
metadata:
name: trino-iceberg-client
stringData:
clientId: trino
clientSecret: my-client-secret
----
<1> The name of the catalog as it will appear in Trino
<2> The hostname of the REST catalog server
<3> Optional, defaults to 443 if TLS is used and to 80 otherwise
<4> Optional, defaults to `/`
<5> Optional, uses the same TLS configuration as xref:usage-guide/s3.adoc#tls[S3 connections].
If not specified, TLS is used and the server certificate is verified against the web PKI (`webPki`).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs say webPki verifies against the web PKI, but in practice it also trusts every SecretClass CA used by any other connection of the TrinoCluster. All connections share one JVM truststore (config/jvm.rs:83, seeded with the system bundle in https://github.com/stackabletech/trino-operator/blob/main/rust/operator-binary/src/controller/build/command.rs#L54, extended per connection in config/tls.rs:30. See the FIXME in https://github.com/stackabletech/trino-operator/blob/main/rust/operator-binary/src/controller/build/command.rs#L52).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely correct! We have that problem at many places already: Lot's of products don't support configuring the ca cert for explicit (S3, LDAP, whatnot) connections. So we don't have any other choice as to add the certs to the JVM truststore.
Ideally we should mention this globally for every affected product, as this applies not only here, but also to a bunch of other places e.g. in Trino. Not sure how to best proceed here, as you are right, the docs are wrong and trino-specific. E.g. on S3Connection there might be some product that do it correctly, here it's only for Trino. The docs describe how it should work, but we currently(!) have the Trino limitations - future Trino versions might support it. The entire platform TlsClientVerification struct is build on this assumption, so from my perspective we have a platform-global problem, this is just continuing status quo.

Set `tls: null` to connect without TLS. Disabling the TLS verification (`verification.none`) is not supported.
`tls: null` is kept when creating or replacing a TrinoCatalog, but `kubectl apply` and server-side apply remove it from an existing TrinoCatalog, which brings back the `webPki` default.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests revealed that kubectl apply replaces null with WebPKI on create. It's a little misleading, kubectl create would behave the way apply is described.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it's a common config mechanism we use and more complicated than though, are you ok with removing the sentence regarding how to set it to null?

Use `kubectl replace` or recreate the TrinoCatalog to switch an existing catalog to a connection without TLS.
<6> Optional, defaults to no authentication
<7> The token endpoint of the OAuth2 server, see <<oauth2-server>>.
It takes the same `hostname`, `port`, `path` and `tls` fields as the REST catalog, so it uses TLS with `webPki` verification by default.
<8> The client credentials are exchanged for a token using the OAuth2 client credentials flow.
Alternatively, use `token.secretName` to pass a Secret containing a static bearer token in the `token` key.
<9> Name of a Secret containing the `clientId` and `clientSecret` keys
<10> Optional, the scopes to request from the OAuth2 server.
If not specified, the `catalog` scope is requested. Specifying scopes replaces this default.

The operator only supports a subset of the https://trino.io/docs/current/object-storage/metastores.html#rest-catalog[REST catalog properties].
Use `configOverrides` to set additional ones.

[#oauth2-server]
=== OAuth2 server

`oAuth2.server` points to the token endpoint of the OAuth2 server.
What it is used for depends on the credential:

* `clientCredentials`: Trino fetches its token from this endpoint on startup and refreshes it there before it expires.
* `token`: Trino uses the token as is.
If the token is a JWT with an `exp` claim, Trino tries to refresh it before it expires by exchanging it at this endpoint (OAuth2 token exchange).
If the refresh fails, Trino logs a warning and keeps using the current token.
A token that isn't a JWT is never refreshed, so the endpoint is never called.

The OAuth2 server is required even for tokens that are never refreshed.
Without it, the Iceberg client used by Trino would fall back to the deprecated `/v1/oauth/tokens` endpoint of the REST catalog, which is planned to be removed (see https://github.com/apache/iceberg/issues/10537[apache/iceberg#10537]).

== Example Iceberg Hive metastore configuration

You can deploy a Hive Metastore Stacklet with the xref:hive:index.adoc[].

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: iceberg # <1>
Expand All @@ -19,8 +108,9 @@ spec:
connector:
# Specify iceberg here when defining a iceberg catalog
iceberg:
metastore:
configMap: simple-hive # <2>
catalog:
hiveMetastore:
configMap: simple-hive # <2>
s3:
inline:
host: test-minio
Expand Down
12 changes: 7 additions & 5 deletions docs/modules/trino/pages/usage-guide/catalogs/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ You can create a catalog using the `TrinoCatalog` object as follows.

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: hive-catalog
Expand All @@ -33,7 +33,7 @@ spec:
credentials:
secretClass: minio-credentials
---
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: iceberg
Expand All @@ -42,8 +42,9 @@ metadata:
spec:
connector:
iceberg:
metastore:
configMap: simple-hive
catalog:
hiveMetastore:
configMap: simple-hive
s3:
inline:
host: test-minio
Expand All @@ -70,7 +71,7 @@ This example illustrates how to use config overrides and config removals

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
spec:
# Add some properties
Expand Down Expand Up @@ -118,6 +119,7 @@ The Trino catalog will be called `my_postgres`:

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
name: my-postgres
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The https://trino.io/docs/current/connector/postgresql.html[PostgreSQL connector

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it will appear in Trino
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/trino/pages/usage-guide/catalogs/tpcds.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This is not a stored data set and is instead a virtual data set generated at que

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it will appear in Trino
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/trino/pages/usage-guide/catalogs/tpch.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This is not a stored data set and is instead a virtual data set generated at que

[source,yaml]
----
apiVersion: trino.stackable.tech/v1alpha1
apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
# The name of the catalog as it will appear in Trino
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/trino/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
* xref:trino:reference/index.adoc[]
** xref:trino:reference/crds.adoc[]
*** {crd-ref}/trino.stackable.tech/v1alpha1/trinocluster{crd-ref-release}[TrinoCluster {external-link-icon}^]
*** {crd-ref}/trino.stackable.tech/v1alpha1/trinocatalog{crd-ref-release}[TrinoCatalog {external-link-icon}^]
*** {crd-ref}/trino.stackable.tech/v1alpha2/trinocatalog{crd-ref-release}[TrinoCatalog {external-link-icon}^]
** xref:trino:reference/security.adoc[]
** xref:trino:reference/commandline-parameters.adoc[]
** xref:trino:reference/environment-variables.adoc[]
Loading
Loading