[INFRA-799] feat: ship RabbitMQ 4.2.9 instead of end-of-life 3.13.6

charts/plane-ce/Chart.yaml

@@ -5,7 +5,7 @@ description: Meet Plane. An open-source software development tool to manage issu
 
 type: application
 
-version: 1.8.0
+version: 1.9.0
 appVersion: "1.4.1"
 
 home: https://plane.so

charts/plane-ce/README.md

@@ -21,6 +21,73 @@
     helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace
     ```
 
+## Upgrading RabbitMQ from 3.13 to 4.2
+
+This chart now ships `rabbitmq:4.2.9-management-alpine`. RabbitMQ 3.x is end-of-life and no longer receives security updates.
+
+> [!IMPORTANT]
+> **Upgrade the application before, or together with, this chart — not after.** RabbitMQ 4.2 refuses any AMQP connection that negotiates
+> a `frame_max` below 8192, and the `amqplib` client used by `silo`, `live` and `flux` defaulted to 4096 until 0.10.6. On a Plane build
+> older than the one carrying that bump, moving the broker to 4.2.9 silently kills those consumers: the broker logs
+> `failed to negotiate connection parameters: negotiated frame_max = 4096 is lower than the minimum allowed value (8192)`, integrations
+> and exports stop, and **every pod still reports `Ready`**. Celery and the pika-based consumers are unaffected, which makes the failure
+> easy to miss. Verified end to end — `plane-exports`, `silo-api` and `silo-integrations` all dropped to 0 consumers.
+>
+> If you must stage them, pin `rabbitmq.image` to `rabbitmq:3.13.6-management-alpine` when taking this chart, upgrade
+> `planeVersion`, then remove the pin.
+>
+> **The chart enforces this.** `rabbitmq.minPlaneVersion` is the lowest `planeVersion` known to work with a 4.x
+> broker; if you set a lower one while the chart is deploying a 4.x broker, rendering fails with an explanation rather than
+> deploying a broken stack. Branch, preview and prerelease tags are allowed through — they cannot be compared here. Set it to
+> an empty string to bypass the check.
+
+**If you are upgrading an existing install with `rabbitmq.local_setup: true`, do this first.** The chart upgrade restarts the broker StatefulSet against the same volume, and RabbitMQ requires all stable feature flags to be enabled *before* a major upgrade — otherwise the 4.2 node refuses to start and your queues are unreachable until you roll back.
+
+```bash
+# 0. Snapshot the RabbitMQ volume. This upgrade cannot be reversed and
+#    `helm rollback` will not undo it -- see the notes below.
+
+# 1. While still on 3.13, enable every stable feature flag.
+kubectl -n <namespace> exec <release>-rabbitmq-wl-0 -- rabbitmqctl enable_feature_flag all
+
+# 2. Confirm nothing stable is left disabled. On 3.13 `khepri_db` is an
+#    EXPERIMENTAL flag, so step 1 does not touch it -- and it must stay
+#    disabled here: a 3.13 node with Khepri enabled cannot be upgraded to
+#    4.x at all and needs a blue-green migration instead.
+kubectl -n <namespace> exec <release>-rabbitmq-wl-0 -- rabbitmqctl list_feature_flags
+
+# 3. Now run the chart upgrade, then confirm the broker came back.
+kubectl -n <namespace> exec <release>-rabbitmq-wl-0 -- rabbitmqctl status | grep 'RabbitMQ version'
+```
+
+Notes:
+
+- **Step 1 will normally report nothing to do.** A node whose data directory was created by 3.13 enables every stable flag at birth (measured on a fresh 3.13.7: all stable flags enabled, only experimental `khepri_db` disabled), which is the case for any install this chart created. Disabled stable flags come from a volume carried across older series. Run it to confirm, not to change anything. On 3.13 the command cannot turn Khepri on, because `khepri_db` is *experimental* there — that is **not** true on 4.2, where the same flag is stable (see below).
+- **Do not jump straight to 4.3.** RabbitMQ does not support a direct 3.13 → 4.3 upgrade ([version upgradability](https://www.rabbitmq.com/docs/upgrade#rabbitmq-version-upgradability)); 4.2 is the supported hop, and a later chart release will move to 4.3. Two further things break on 4.3 but not on 4.2: Celery's control/event queues (fixed in the application by `CELERY_CONTROL_QUEUE_EXCLUSIVE` / `CELERY_EVENT_QUEUE_EXCLUSIVE`), and `x-consumer-timeout` on classic queues.
+- **This upgrade is one-way. Snapshot the volume first.** Once a 4.2 node has started on a data directory, 3.13 cannot start on it again — and this is **not** conditional on enabling `khepri_db`. Measured: after a 3.13.7 → 4.2.9 upgrade the node still reported `khepri_db` *disabled* and was still using Mnesia, with no Khepri directories on disk, and 3.13.7 still refused to come back up. The failure mode is the awkward one — the container does **not** exit, it sits in `Waiting for Khepri projections` indefinitely, so there is no CrashLoopBackOff to alert on, only a pod that never becomes Ready. A `helm rollback` will not save you, because the data directory is what changed. Your only way back is a **pre-upgrade volume snapshot**, or a blue-green migration onto a fresh volume.
+- **The broker's readiness probe checks that it is actually serving.** It runs `rabbitmq-diagnostics check_running` (the app is booted, not merely listening) followed by `check_port_connectivity`, every 15s with a `failureThreshold` of 3 — so a broker that boots but never finishes leaves the pod NotReady within ~45s and is taken out of the Service. Note that `rabbitmq-diagnostics ping` is **not** sufficient here: it passes against a node stuck mid-boot, because the runtime is alive even though the broker is not.
+- **No queue changes are required for the broker this chart deploys.** Existing queues keep their arguments and are re-declared as-is by the application; durable messages survive the restart. Verified end to end on a 3.13.6 → 4.2.9 in-place upgrade with pre-existing queues. This holds because the chart runs a **single-node** broker, where classic queue mirroring — removed in 4.0 — cannot be in effect. Plane sets no `ha-mode` policy itself.
+- **Using an external broker?** If `rabbitmq.local_setup: false` and you point `external_rabbitmq_url` at a managed or clustered broker (Amazon MQ, CloudAMQP, your own cluster), this chart does not manage its version — upgrade it on the provider side, following the same feature-flag prerequisite. Check for mirrored classic queues first, because 4.0 removed mirroring and any still-mirrored queue silently loses its replicas:
+
+  ```bash
+  rabbitmq-queues check_if_cluster_has_classic_queue_mirroring_policy
+  ```
+
+  If that reports a policy, migrate those queues to quorum queues (or do a blue-green migration) before upgrading.
+
+### Reaching 4.3 later: the 4.2 feature flags
+
+A node upgraded in place to 4.2 leaves the flags 4.2 introduced **disabled** — measured on a 3.13.7 → 4.2.9 upgrade: `khepri_db`, `rabbitmq_4.0.0`, `rabbitmq_4.1.0`, `rabbitmq_4.2.0` and `rabbit_exchange_type_local_random` were all disabled afterwards, and the node kept using Mnesia. RabbitMQ 4.3 requires them, so they have to be enabled before that hop.
+
+**Do this as a deliberate step once 4.2 has been running cleanly — not as part of the upgrade.** On 4.2 `khepri_db` is *stable*, so `enable_feature_flag all` enables it, which converts the metadata store from Mnesia to Khepri. That conversion cannot be undone within 4.x either, so take a fresh volume snapshot before running it. (The route back to 3.13 is already gone by this point — see the one-way note above.)
+
+```bash
+kubectl -n <namespace> exec <release>-rabbitmq-wl-0 -- rabbitmqctl enable_feature_flag all
+kubectl -n <namespace> exec <release>-rabbitmq-wl-0 -- rabbitmqctl list_feature_flags
+```
+
+Until you run this the broker is fully supported on 4.2 and Plane works normally; the only thing you cannot do is upgrade to 4.3.
+
 ## Installing Plane
 
 1. Open Terminal or any other command-line app that has access to Kubernetes tools on your local system.
@@ -210,7 +277,7 @@ The default value is `"traefik"`. If you previously relied on the implicit defau
 | Setting                        |              Default              | Required | Description                                                                                                                                                                                                                                                                                                                                |
 | ------------------------------ | :-------------------------------: | :------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | rabbitmq.local_setup           |               true                |          | Plane uses `rabbitmq` as message queuing system. This can be hosted within kubernetes as part of helm chart deployment or can be used as hosted service remotely (e.g. aws mq or similar services). Set this to `true` when you choose to setup stateful deployment of `rabbitmq`. Mark it as `false` when using a remotely hosted service |
-| rabbitmq.image                 | rabbitmq:3.13.6-management-alpine |          | Using this key, user must provide the docker image name to setup the stateful deployment of `rabbitmq`. (must be set when `rabbitmq.local_setup=true`)                                                                                                                                                                                     |
+| rabbitmq.image                 | rabbitmq:4.2.9-management-alpine |          | Using this key, user must provide the docker image name to setup the stateful deployment of `rabbitmq`. (must be set when `rabbitmq.local_setup=true`)                                                                                                                                                                                     |
 | rabbitmq.pullPolicy            |           IfNotPresent            |          | Using this key, user can set the pull policy for the stateful deployment of `rabbitmq`. (must be set when `rabbitmq.local_setup=true`)                                                                                                                                                                                                     |
 | rabbitmq.servicePort           |               5672                |          | This key sets the default port number to be used while setting up stateful deployment of `rabbitmq`.                                                                                                                                                                                                                                       |
 | rabbitmq.managementPort        |               15672               |          | This key sets the default management port number to be used while setting up stateful deployment of `rabbitmq`.                                                                                                                                                                                                                            |

charts/plane-ce/questions.yml

@@ -407,7 +407,7 @@ questions:
   - variable: rabbitmq.image
     label: "Docker Image"
     type: string
-    default: "rabbitmq:3.13.6-management-alpine"
+    default: "rabbitmq:4.2.9-management-alpine"
     show_if: "rabbitmq.local_setup=true"
   - variable: rabbitmq.pullPolicy
     label: "Image Pull Policy"

charts/plane-ce/templates/_helpers.tpl

@@ -260,3 +260,51 @@ Caller must indent to the correct depth (env list items).
 {{- end }}
 {{- end }}
 {{- end -}}
+
+{{/*
+Refuse to deploy a 4.x broker under an application build whose amqplib cannot
+reach it. RabbitMQ >= 4.1 rejects any connection negotiating a frame_max below
+8192 and amqplib defaulted to 4096 until 0.10.6, so an older Plane build loses
+its silo/live consumers the moment the broker moves -- while every pod still
+reports Ready. Measured on 4.2.9 and 4.3.5.
+
+Only fires when this chart manages the broker, the pinned tag is 4.x, and
+planeVersion is an exact release. Branch, preview and prerelease tags are
+allowed through: they are unresolvable here and blocking them would be wrong
+more often than right. Set the floor to "" to bypass.
+*/}}
+{{- define "plane.validateBrokerClientCompat" -}}
+{{- $rmq := .rabbitmq -}}
+{{- $floor := $rmq.minPlaneVersion | default "" -}}
+{{- if $floor -}}
+  {{- if not (regexMatch "^v?[0-9]+(\\.[0-9]+){0,2}$" $floor) -}}
+    {{- fail (printf "\n\nrabbitmq.minPlaneVersion must be a version like \"3.2.0\", or \"\" to disable\nthe check. Got %q, which cannot be compared and would break every render.\n" $floor) -}}
+  {{- end -}}
+  {{- $tag := splitList ":" ($rmq.image | default "") | last -}}
+  {{- $major := regexFind "^[0-9]+" $tag -}}
+  {{- if and $major (ge (atoi $major) 4) -}}
+    {{- $pv := trimPrefix "v" (.planeVersion | default "") -}}
+    {{- if regexMatch "^[0-9]+\\.[0-9]+\\.[0-9]+$" $pv -}}
+      {{- if not (semverCompare (printf ">= %s" $floor) $pv) -}}
+        {{- $msg := list
+            (printf "RabbitMQ %s requires Plane >= %s, but planeVersion is %s." $tag $floor $pv)
+            ""
+            "RabbitMQ 4.1+ rejects connections negotiating frame_max below 8192, and the"
+            "amqplib client in that Plane build still negotiates 4096. Moving the broker"
+            "without moving the application drops the plane-exports, silo-api and"
+            "silo-integrations consumers to zero -- integrations, imports and exports"
+            "stop, while every pod continues to report Ready."
+            ""
+            "Pick one:"
+            (printf "  1. Set planeVersion to %s or newer (upgrade both together)." $floor)
+            "  2. Keep the broker on 3.13 for now: pin the rabbitmq image to"
+            "     rabbitmq:3.13.6-management-alpine, upgrade planeVersion later,"
+            "     then remove the pin."
+            "  3. Bypass this check: set rabbitmq.minPlaneVersion to an empty string."
+            | join "\n" -}}
+        {{- fail (printf "\n\n%s\n" $msg) -}}
+      {{- end -}}
+    {{- end -}}
+  {{- end -}}
+{{- end -}}
+{{- end -}}

charts/plane-ce/templates/workloads/rabbitmq.stateful.yaml

@@ -1,4 +1,5 @@
 {{- if .Values.rabbitmq.local_setup }}
+{{- include "plane.validateBrokerClientCompat" (dict "rabbitmq" .Values.rabbitmq "planeVersion" .Values.planeVersion) }}
 
 apiVersion: v1
 kind: Service
@@ -57,6 +58,16 @@ spec:
         - mountPath: /var/lib/rabbitmq
           name: pvc-{{ .Release.Name }}-rabbitmq-vol
           subPath: ''
+        readinessProbe:
+          exec:
+            command:
+              - sh
+              - -c
+              - rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_port_connectivity
+          initialDelaySeconds: 10
+          periodSeconds: 15
+          timeoutSeconds: 10
+          failureThreshold: 3
       {{- include "plane.podScheduling" .Values.rabbitmq }}
       serviceAccount: {{ include "plane.serviceAccountName" . }}
       serviceAccountName: {{ include "plane.serviceAccountName" . }}

charts/plane-ce/values.yaml

@@ -91,7 +91,17 @@ postgres:
 
 rabbitmq:
   local_setup: true
-  image: rabbitmq:3.13.6-management-alpine
+  image: rabbitmq:4.2.9-management-alpine
+  # Lowest planeVersion whose amqplib can reach a 4.x broker. RabbitMQ 4.1+
+  # rejects a negotiated frame_max below 8192; amqplib sent 4096 until 0.10.6,
+  # so an older app silently loses its silo/live consumers when the broker
+  # moves. Rendering fails with an explanation instead. Set to '' to bypass.
+  # MUST be set at release time to the FIRST Plane release whose images carry
+  # the amqplib bump -- normally the same value as Chart.yaml appVersion for
+  # that release. It then stays pinned there; do not raise it in later chart
+  # releases, or you will block app versions that are actually fine. A leading
+  # 'v' is accepted.
+  minPlaneVersion: ''
   pullPolicy: IfNotPresent
   servicePort: 5672
   managementPort: 15672

charts/plane-enterprise/Chart.yaml

@@ -5,7 +5,7 @@ description: Meet Plane. An Enterprise software development tool to manage issue
 
 type: application
 
-version: 3.5.8
+version: 3.6.0
 appVersion: "3.1.4"
 
 home: https://plane.so/