relay stops relaying

[jap]

Self-Hosted Version

26.3.1

CPU Architecture

x86_64

Docker Version

29.2.1

Docker Compose Version

5.1.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade

Steps to Reproduce

1. Start the whole stack using docker compose
2. Wait

Expected Result

Sentry should keep on keeping on.

Actual Result

Sentry stops processing new metrics and events.

Looking at logs, we get tons of entries similar to the ones below in the relay container, and the start of those entries matches the end of processing.

relay-1  | 2026-03-24T12:44:30.887984Z ERROR relay_kafka::producer::utils: failed to produce message to Kafka (delivery callback) error=Message production error: MessageTimedOut (Local: Message timed out) error.sources=[MessageTimedOut (Local: Message timed out)] payload_len=45808 tags.topic="ingest-events"

These errors are produced for all kinds of topics.

Restarting the relay container makes things move again, but in flight data is lost :(

Event ID

No response

[jap]

Note that the kafka container does not log any errors.

[aldy505]

What about your CPU resource usage, and your load average? Is it full of backlogs?

[inoa-jboliveira]

I believe I got if not the same issue, a very similar. Memory usage skyrocketed. Then I limited relay memory and the swap skyrocketed.
It is more stable memory limited than not. But it is very bad. I let claude loose looking for logs in the containers for hints and it pointed me to this issue.

This is quite serious and I might need to downgrade sentry. What version would be safe to do so? and can i revert the database migrations?

Below is the claude report:

Relay Degradation Summary

Env: Self-hosted Sentry 26.3.1 (upgraded from 25.11.x), 61 GiB RAM, 16 GiB swap

Problem

Relay degrades after ~1 hour with MessageTimedOut on ingest-spans. Host swap fills completely.

Root Cause

Memory leak via zombie flusher processes in process-spans.

The SpanFlusher (in sentry/spans/consumers/process/flusher.py) detects unhealthy workers and calls process.kill() + spawns a replacement □~@~T but the old process doesn't die (threads stuck in librdkafka I/O). Each leaked process holds ~1 GiB. Over hours they accumulate, exhaust RAM, push everything to swap, increase Kafka/Redis latency, cause more timeouts, and create a death spiral.

At time of investigation: 3 leaked flusher workers alive (~3 GiB) when only 1 should exist.

Immediate Mitigation

Cap process-spans memory in docker-compose.override.yml:

services:
  process-spans:
    deploy:
      resources:
        limits:
          memory: 4G

This forces a container restart when leaked processes exceed the budget. (EDITOR NOTE: Actually this just forces swap usage)

Upstream fix needed

The SpanFlusher._create_process_for_shards() restart path must:

• Call .join(timeout) on the old process after .kill()
• Verify the old process is actually dead before spawning replacement
• If .join() times out, escalate (e.g., SIGTERM □FR SIGKILL □FR force new worker
  without the old one's resources leaking)

SpanFlusher restart logic to .join() old processes before spawning replacements.

[aldy505]

Relay degrades after ~1 hour with MessageTimedOut on ingest-spans. Host swap fills completely.

Before you came up with this problem description, what was your container log?