tracking: Test demo upgrades on nightly versions for 26.7.0

@adwk67

This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

hourglass: In progress / waiting
red_circle: In progress with errors
orange_circle: Completed with minor issues or anomalies
green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

List of stacks

Similar to the demos above we need to test all stacks that don't have a demo.
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the stack to indicate a particular stack was picked up for testing.

Tip

Some of the stacks have a tutorial to follow.

We decided an upgrade test is not needed

~~monitoring~~
~~observability~~
~~openldap~~
~~tutorial-openldap~~

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

# For demo testing, , install demo (stable operators) for the stable release (26.3).
stackablectl demo install <DEMO_NAME>

# For stack testing, install stack via
stackablectl stack install <STACK_NAME>

# --- IMPORTANT ---
# Run through the stable demo/stack instructions (refer to the list above).

# Install nightly version of operators
stackablectl release upgrade dev

# Bump every stacklet to the version that used by the nightly demo.
# Typically, this is the latest supported version.
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

# Check that the operator logs look good (no unexpected errors etc.)

# Run through the (still) nightly demo/stack instructions again.

HDFS rolling upgrade instructions

For demos that use HDFS, also test a rolling upgrade.
Also see https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/

# In some cases (eg. in the end-to-end-security demo), HDFS is kerberized.
# This requires these commands to be run first.
# Pick "nn/..."
klist -kt /stackable/kerberos/keytab
kinit -kt /stackable/kerberos/keytab <PRINCIPAL_FROM_ABOVE>

# These commands need to be executed in a superuser environment (eg. the namenode pod)
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query  # repeat until "Proceed with rolling upgrade"

# Now patch the productVersion in your regular shell
kubectl patch hdfs/<CLUSTER_NAME> --type=merge \
  --patch '{"spec": {"image": {"productVersion": "<NEW_VERSION>"}}}'

# Finalize the upgrade in the namenode pod
# Needs another kinit because the container was restarted
hdfs dfsadmin -rollingUpgrade finalize

kubectl patch hdfs/<CLUSTER_NAME> --subresource=status --type=merge \
  --patch '{"status": {"deployedProductVersion": "<NEW_VERSION>"}}'

🟢 logging

🟢 Install stable
🟢 Upgrade to nightly

@adwk67

🟢 signal-processing

🟢 Install stable
Initially, the create-nifi-ingestion-job failed because it timed out when trying to connect to NiFi. NiFi seems to fail to connect to the Postgres database: JDBC driver class 'org.postgresql.Driver' not found. No JDBC driver classes found in the provided resources. @adwk67 discovered the issue and it is fixed by:
- fix(signal-processing): fetch flow definition from release-26.3 #427
- fix: Processing URL and namespace replacement #428
🟢 Upgrade to nightly

⌛ trino-iceberg

🟢 Install stable
🟢 Upgrade to nightly
- Obviously had to address the CRD change from feat: Implement generic database connection hive-operator#674

⌛ trino-taxi-data

🟢 Install stable
🟢 Upgrade to nightly
- 🟢 Obviously had to address the CRD change from feat: Implement generic database connection hive-operator#674
- 🟢 And same for Superset: feat: Add generic database connection mechanism superset-operator#722. We found a bug related to SECRET_KEY rotation, see https://stackable-workspace.slack.com/archives/C0AV49795HQ/p1783672917029649 = > Fixed by fix: Correctly migrate SECRET_KEY from 26.3 setups superset-operator#754

🟢 spark-k8s-anomaly-detection-taxi-data

🟢 Baseline: Fine
🟢 Upgrade: Currently a problems:
- 🟢 spark-k8s-operator due to now-namespaced sparkapptemplate. Revert "fix: make application templates namespaced (#694)" spark-k8s-operator#719
- 🟢 hive CRD breaking change (Missing field metadataDatabase).
- 🟢 superset CRD breking change (Missing field metadataDatabase, also missing credentialSecretName, and a new SECRET_KEY is being generated instead of using the established one: 26.3 -> 0.0.0-dev SECRET_KEY rotation breaks stacklet superset-operator#753)

Upgrade commands

kubectl patch opaclusters/opa --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"1.16.2"}]'
kubectl patch hiveclusters/hive --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"4.2.0"}]'
kubectl patch hiveclusters/hive-iceberg --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"4.2.0"}]'
kubectl patch sparkapplications/spark-ad --type='json' -p='[{"op": "replace", "path": "/spec/sparkImage/productVersion", "value":"4.1.2"}]'
kubectl patch trinoclusters/trino --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"481"}]'
kubectl patch supersetclusters/superset --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"6.1.0"}]'

original issues

Hive operator nightly:

2026-07-10T07:28:11.361462Z  INFO reconciling object{object.ref=HiveCluster.v1alpha1.hive.stackable.tech/hive-iceberg.default object.reason=related object updated: StatefulSet.v1.apps/hive-iceberg-metastore-default.default}: stackable_hive_operator::controller: Starting reconcile
2026-07-10T07:28:11.361657Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="hivecluster.hive.stackable.tech" error=reconciler for object HiveCluster.v1alpha1.hive.stackable.tech/hive-iceberg.default failed error.sources=[HiveCluster object is invalid, Missing field metadataDatabase]

Spark operator nightly:

Error: failed to run webhook server

Caused by:
    0: failed to update certificate
    1: conversion webhook error
    2: failed to patch CRD "sparkapptemplates.spark.stackable.tech"
    3: ApiError: CustomResourceDefinition.apiextensions.k8s.io "sparkapptemplates.spark.stackable.tech" is invalid: spec.scope: Invalid value: "Namespaced": field is immutable: Invalid (Status { status: Some(Failure), code: 422, message: "CustomResourceDefinition.apiextensions.k8s.io \"sparkapptemplates.spark.stackable.tech\" is invalid: spec.scope: Invalid value: \"Namespaced\": field is immutable", metadata: Some(ListMeta { continue_: None, remaining_item_count: None, resource_version: None, self_link: None, shard_info: None }), reason: "Invalid", details: Some(StatusDetails { name: "sparkapptemplates.spark.stackable.tech", group: "apiextensions.k8s.io", kind: "CustomResourceDefinition", uid: "", causes: [StatusCause { reason: "FieldValueInvalid", message: "Invalid value: \"Namespaced\": field is immutable", field: "spec.scope" }], retry_after_seconds: 0 }) })
    4: CustomResourceDefinition.apiextensions.k8s.io "sparkapptemplates.spark.stackable.tech" is invalid: spec.scope: Invalid value: "Namespaced": field is immutable: Invalid
stream closed: EOF for stackable-operators/spark-k8s-operator-deployment-575544b9c4-rr487 (spark-k8s-operator)

🟢 airflow-scheduled-job

🟢 Baseline: Fine
🟢 Upgrade: Fine, other than:
- (🟠spark-k8s-operator due to now-namespaced sparkapptemplate)
- had to patch the airflow cluster to add the missing and required-in-nightly fields credentialsSecretName and metadataDatabase

🟢 jupyterhub-keycloak

🟢 Install stable
🟢 Upgrade to nightly
- notebook only completed successfully after I deleted old data from MinIO

🟢 end-to-end-security

🟢 Install stable
🟢 Upgrade to nightly (rolling upgrade worked after kinit)

🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

🟢 Install stable
🟢 Upgrade to nightly (had to delete the Sparkcluster STS)
🟢 Jupyterlab deployment needs image update: chore(jupyterhub-pyspark-hdfs): bump image used for Jupyterlab #430
🟢 HDFS rolling upgrade

🟢 hbase-hdfs-load-cycling-data

🟢 Install stable
🟢 Upgrade to nightly
- rest server could not be patched on operator upgrade so restarted with kubectl delete sts hbase-restserver-default
- rest server could could not be patched to 2.6.6 so restarted with kubectl delete sts hbase-restserver-default
- hdfs rolling upgrade OK

🟢 nifi-kafka-druid-earthquake-data

🟢 nifi-kafka-druid-water-level-data

🔴 argo-cd-git-ops

Note

Should be fixed by #438

🔴 Install stable
- Sealed secrets are not being decrypted. Demo doesn't fully come up because of that.
❌ Upgrade to nightly

🟢 data-lakehouse-iceberg-trino-spark
-> Had to do the usual CRD changes, all documented
-> Had to delete the SparkAppTemplate

Upgrade testing completed

Navigation Menu

Repository navigation

tracking: Test demo upgrades on nightly versions for 26.7.0 #425

List of demos

List of stacks

Stable to Nightly Upgrade Testing Instructions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tracking: Test demo upgrades on nightly versions for 26.7.0 #425

Description

List of demos

List of stacks

Stable to Nightly Upgrade Testing Instructions

Activity

Techassi commented on Jul 10, 2026

Techassi commented on Jul 10, 2026

sbernauer commented on Jul 10, 2026

sbernauer commented on Jul 10, 2026

NickLarsenNZ commented on Jul 10, 2026

adwk67 commented on Jul 10, 2026

Techassi commented on Jul 10, 2026

Techassi commented on Jul 10, 2026

sweb commented on Jul 10, 2026

adwk67 commented on Jul 10, 2026

sbernauer commented on Jul 13, 2026

sbernauer commented on Jul 13, 2026

Techassi commented on Jul 13, 2026

sbernauer commented on Jul 14, 2026

sbernauer commented on Jul 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions