Skip to content

fix(portal): ping Redis on a separate connection for the readiness check - #2990

Merged
Brad-Edwards merged 3 commits into
devfrom
fix/portal-redis-health-ping
Nov 19, 2026
Merged

Brad-Edwards merged 3 commits into
devfrom
fix/portal-redis-health-ping

Conversation

@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Problem

The channel-layer Redis readiness check (config/health_checks.py) sent and received a message through the shared channel layer. A concurrent WebSocket receive() on that layer makes channels_redis raise:

RuntimeError: Two event loops are trying to receive() on one channel layer at once!

About one in four /health/ checks failed while Redis was healthy. On the live event tenant this kept 5 to 20 of 68 portal pods out of the load balancer.

Change

  • The check sends PING on its own process-wide Redis client.
  • The client uses the first configured channel-layer host, its TLS options (ssl_ca_data, ssl_cert_reqs), and 2-second socket and connect timeouts.
  • A real Redis outage still fails the check.
  • Remove the unused channel-layer round-trip code and its tests.

This runs on the live event tenant since 2026-11-19 18:46 UTC.

Verification

  • In a live portal pod, 480 of 480 concurrent PING checks (48 threads) passed.
  • tests/mission_control/test_health.py: 23 passed. New tests cover PING success, PING failure, the client options and timeouts, and concurrent checks that share one client.
  • ruff check, ruff format --check, and lint-imports: pass.

The channel-layer Redis readiness check sent and received a message
through the shared channel layer. A concurrent WebSocket receive() on
that layer makes channels_redis raise "Two event loops are trying to
receive() on one channel layer at once!". About one in four health checks
failed while Redis was healthy. On the live event tenant this kept 5 to
20 of 68 portal pods out of the load balancer.

The check now sends PING on its own process-wide Redis client. The client
uses the first configured channel-layer host, its TLS options, and
2-second socket and connect timeouts. Remove the unused channel-layer
round-trip code and its tests. Add tests for PING success, PING failure,
the client options, and concurrent checks.
…check

The non-TLS channel layer emits a (host, port) pair, not a URL. The Redis
readiness check passed that pair to redis.Redis.from_url(), which raised
AttributeError, so the stack smoke test saw /health fail. Build the client
from the pair, and pass the timeouts as named arguments so mypy accepts
the redis-py signature. Add a test for the host and port form.
SonarCloud reported the Any type hints on the probe client. Use the
redis.Redis type, imported only for type checking.
@Brad-Edwards
Brad-Edwards merged commit 1fb1e5c into dev Nov 19, 2026
101 checks passed
@Brad-Edwards
Brad-Edwards deleted the fix/portal-redis-health-ping branch November 19, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant