Skip to content

[silo] env knob to suppress Celery Producer initializer when silo is producer-only / not Celery #9100

@grkml

Description

@grkml

Summary

Silo's "Celery Producer" initializer re-declares an empty celery queue on the vhost it connects to on every AMQP reconnect, even when nothing on that vhost consumes from celery. In self-hosted topologies that isolate silo to its own RabbitMQ vhost (one vhost per Plane service), this leaves a vestigial empty queue that:

  • can't be cleaned up at the broker (silo re-declares on every reconnect),
  • pollutes rabbitmqctl export_definitions outputs consumed by backup/restore tooling, and
  • registers against generic QueueHasNoConsumers-style alerts as soon as the alert scope tightens.

Requesting an environment variable (e.g. SILO_DECLARE_CELERY=false or similar) on the silo container that skips the Celery Producer queue-declaration step when silo is producer-only / not actually feeding a Celery worker pool on its own vhost.

Reproduce

  1. Deploy Plane with one RabbitMQ vhost per service (/plane-api, /plane-silo, /plane-pi, /plane-flux); point silo at /plane-silo.
  2. rabbitmqctl delete_queue celery -p /plane-silo while silo is connected.
  3. Wait for silo's next AMQP reconnect (or roll the silo Deployment).
  4. rabbitmqctl list_queues -p /plane-silo shows celery re-declared with 0 messages, 0 consumers.

Silo's logs around the re-declare are:

Initializing Celery Producer.. ♨️
Message Queue celery connected successfully 🐇🐇🐰

Why the queue is vestigial in this topology

Celery workers live behind /plane-api's celery queue (consumed by the worker pool). /plane-silo exists purely for silo's own AMQP traffic and has no Celery consumer attached to it. The producer initializer creates the queue defensively, but on a per-service-vhost topology the queue never carries messages.

What I'd like

An env var on the silo container that disables the Celery Producer initializer's queue.declare, e.g.:

SILO_DECLARE_CELERY=false       # or DISABLE_CELERY_PRODUCER=true, naming is yours

The same knob would be useful in any topology where silo doesn't feed a Celery worker pool on its own vhost.

Workarounds tried

  • Broker-side cleanup via rabbitmqctl delete_queue -- regresses on every silo reconnect.
  • Pre-declaring the queue as a stable artifact -- still leaves the empty queue in the topology export and doesn't address the producer's intent.
  • Alert-side carve-out targeting the (/plane-silo, celery) vhost+queue pair -- shipped as a stopgap, but doesn't address the underlying re-declare loop.

Environment

  • Plane Enterprise self-hosted on Kubernetes (oke001, OKE)
  • silo image: makeplane/silo-commercial:v2.3.4
  • RabbitMQ 4.0.9 (Cluster Operator), default_queue_type=quorum cluster-wide
  • One vhost per service (/plane-api, /plane-silo, /plane-pi, /plane-flux)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions