Repository navigation
chore(tracking): Test demo upgrades on nightly versions for 25.11.0 #326
Description
Part of stackabletech/issues#759
This is testing that upgrades from the stable release to the nightly release of the operators and products do not negatively impact the products.
Note
Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:
:green_circle: **airflow-scheduled-job**
The CRD had been updated and I needed to change the following in the manifest:
...The following emojis can be used to indicate the status:
hourglass: In progress / waitingred_circle: In progress with errorsorange_circle: Completed with minor issues or anomaliesgreen_circle: Completed without issues or errors
List of demos
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your GitHub handle next to the demo to indicate a particular demo was picked up for testing.
- airflow-scheduled-job @adwk67
- data-lakehouse-iceberg-trino-spark @labrenbe
- end-to-end-security @Techassi
- hbase-hdfs-load-cycling-data @adwk67
- jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @adwk67
- jupyterhub-keycloak @adwk67
- logging @Techassi
- nifi-kafka-druid-earthquake-data @razvan
- nifi-kafka-druid-water-level-data @razvan
- signal-processing @Techassi
- spark-k8s-anomaly-detection-taxi-data @adwk67
- trino-iceberg @labrenbe
- trino-taxi-data @labrenbe
List of stacks
Unused stack upgrade testing is not needed. Demos already test product upgrades.
Therefore, the unused stack tests will be only done from scratch. See: #327
Stable to Nightly Upgrade Testing Instructions
These instructions are for deploying and completing the stable demo, and then upgrading operators, CRDs, and products to the nightly versions well as upgrading the operators and CRDS.
Tip
Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.
# Make sure stackablectl 1.2.0 is installed
stackablectl --version
# Install demo (stable operators) for the stable release (25.7).
stackablectl demo install <DEMO_NAME>
# --- IMPORTANT ---
# Run through the stable demo instructions (refer to the list above).
# Get a list of installed operators
stackablectl operator installed --output=plain
# Uninstall operators for the stable release (25.7)
stackablectl release uninstall 25.7
# Update CRDs to nightly version (on main)
# Repeat this for every operator used by the demo (use the list from the earlier step before deleting the operators)
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/...-operator/main/deploy/helm/...-operator/crds/crds.yaml
# N.B. for the secret-operator this should not be necessary as the latest version will be deployed by the operator
# Install nightly version of operators (use the list from the earlier step before deleting the operators)
stackablectl operator install commons ...
# Optionally update the product versions in the CRDs (to the latest non-experimental version for the new release), e.g.:
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed
Activity
adwk67 commentedon Oct 30, 2025
🟢 jupyterhub-keycloak
Techassi commentedon Oct 30, 2025
🟢 logging
adwk67 commentedon Oct 30, 2025
🟢 airflow-scheduled-job
the secret operator CRDs have to be deleted to avoid errors when rolling out the nightly operator. This is expected when moving from 25.7 to 25.11 as thespec.conversion.strategyvalues are different (NoneandWebhook). Should be ok thereafter.Techassi commentedon Oct 30, 2025
🟢 signal-processing
razvan commentedon Oct 30, 2025
Update
Retry with the this version of the secret operator stackabletech/secret-operator#653
🟢 nifi-kafka-druid-earthquake-data
Was:
🔴 nifi-kafka-druid-earthquake-data
secretclassesandtruststoresCRDsWarning FailedBinding 24m (x6 over 24m) ephemeral_volume ephemeral volume tls-mount: PVC default/druid-broker-default-0-tls-mount was not created for pod default/druid-broker-defaul t-0 (pod is not owner)Techassi commentedon Oct 31, 2025
🟢 end-to-end-security
WAS
🔴 end-to-end-security
SQL Error [84148225]: Query failed (#20251031_152856_00001_zgwra): Error accessing metadata file for table customer_analytics.customerERROR [Metastore-Handler-Pool: Thread-223] server.TThreadPoolServer: Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException: Socket is closed by peer.stackablectl release upgrade devcommand is broken because of a 404. See https://stackable-workspace.slack.com/archives/C0985QFRVGC/p1762172924746249 for details => Created fix: Don't crash duringrelease upgradefor SDP 25.11 stackablectl#418Details
Trino queries failed with
HDFS config still is
The hdfs-operator logs
Solution:
Afterwards I got
That's IMHO unrelated, but worth it's own research
adwk67 commentedon Oct 31, 2025
🟢 hbase-hdfs-load-cycling-data
Also: the secret-operator CRDs had to be deleted manually before upgrading the operators.adwk67 commentedon Oct 31, 2025
🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data
The SecretClass and TrustStore CRDs need to be deleted.adwk67 commentedon Oct 31, 2025
🟢 spark-k8s-anomaly-detection-taxi-data
Secret-op logs this out after the bump to nightly, not sure if it is a problem:
(Update: as discussed in the daily 03.11, this is to be expected)
labrenbe commentedon Oct 31, 2025
🟢 trino-iceberg
labrenbe commentedon Oct 31, 2025
🟢 trino-taxi-data
razvan commentedon Nov 2, 2025
🟢 nifi-kafka-druid-water-level-data
labrenbe commentedon Nov 3, 2025
🟢 data-lakehouse-iceberg-trino-spark
mainTechassi commentedon Nov 4, 2025
Done, closing.