Skip to content

fix(RedisStreams): recover connection pool after a failed connect instead of caching it forever#1808

Open
rebaseandpanic wants to merge 1 commit into
dotnetcore:masterfrom
rebaseandpanic:fix/redisstreams-connection-pool-recovery
Open

fix(RedisStreams): recover connection pool after a failed connect instead of caching it forever#1808
rebaseandpanic wants to merge 1 commit into
dotnetcore:masterfrom
rebaseandpanic:fix/redisstreams-connection-pool-recovery

Conversation

@rebaseandpanic

Copy link
Copy Markdown

Description:

The Redis Streams transport's connection pool could get permanently stuck after a
failed initial connect and never recover, even once Redis was reachable again. The
process stayed alive but the transport was dead — publishes and consumes silently
stopped, with no crash or exception surfaced to health checks.

Root cause: AsyncLazyRedisConnection derived from Lazy<Task<RedisConnection>>,
which caches the connect Task permanently. If that first connect faulted, the
faulted Task was cached forever — every later access rethrew the same cached
exception and the connect factory was never retried. Because the pool is a singleton
whose slots are created once, nothing ever reset the poisoned slot. Worse, connection
selection dereferenced the throwing CreatedConnection getter for every slot, so a
single poisoned slot took the whole pool down for both publish and consume.

This only reproduces on CAP + Redis Sentinel: a normal abortConnect=false
connect does not throw (the multiplexer reconnects itself), but the Sentinel path
(GetSentinelMasterConnection) throws — which is what faults the cached Task. That
narrow combination is why it hasn't surfaced before.

Issue(s) addressed:

  • N/A

Changes:

  • Drop the Lazy<Task<RedisConnection>> base. Hold the connect Task under a lock and
    recreate it when it faulted or was cancelled, while reusing a healthy or in-flight Task.
  • Replace the throwing CreatedConnection getter with a non-throwing
    TryGetCreatedConnection(out ...). Pool selection and Dispose skip poisoned slots, so
    one bad slot can no longer take the pool down.
  • In the connect factory, dispose the multiplexer of every failed attempt and throw after
    exhausting retries, so the Task faults and the slot is recreated on the next access —
    instead of caching a disconnected multiplexer as "healthy" forever.
  • Make pool disposal atomic (Interlocked) and clean up in-flight connects.
  • Add DotNetCore.CAP.RedisStreams.Test with unit tests (a connection-factory seam is
    injected so the retry/recovery logic runs against a fake multiplexer, no live Redis).

Incidentally fixes two latent defects in the old retry loop: a disconnected multiplexer
returned as success after exhausting attempts, and up to four leaked multiplexers per cycle.

Affected components:

  • DotNetCore.CAP.RedisStreams (connection pool only — IConnectionPool.LazyConnection.cs,
    IConnectionPool.Default.cs). No public API of CAP itself changes.

How to test:

dotnet test test/DotNetCore.CAP.RedisStreams.Test/

The tests cover: recovery after a failed connect once Redis returns, a disconnected
multiplexer never being cached as healthy, every failed attempt being disposed (no leak),
a single poisoned slot not taking down selection of healthy slots, and pool disposal
(exact-once, in-flight cleanup). All deterministic — no network, no Thread.Sleep.

Additional notes (optional):

BREAKING CHANGE: AsyncLazyRedisConnection no longer derives from
Lazy<Task<RedisConnection>> and no longer exposes CreatedConnection. This is
unavoidable — the Lazy<Task<T>> base is the bug. The type is internal transport
plumbing with no intended external consumers; I'd suggest marking it internal (happy to
do that here if you prefer).

One coverage gap left intentionally: the least-loaded selection path (comparing non-zero
ConnectionCapacity) isn't unit-tested, because forcing a non-zero
ServerCounters.TotalOutstanding on a fake would require reflecting into
StackExchange.Redis internals — too brittle to put in the suite. The behaviour is
unchanged from before.

Checklist:

  • I have tested my changes locally
  • I have added necessary documentation (if applicable)
  • I have updated the relevant tests (if applicable)
  • My changes follow the project's code style guidelines

…tead of caching it forever

AsyncLazyRedisConnection derived from Lazy<Task<RedisConnection>>, which caches the
connect Task permanently. When the initial connect faulted — the Redis Sentinel path
(GetSentinelMasterConnection) throws, unlike a normal abortConnect=false connect — the
faulted Task was cached forever: every subsequent access rethrew the same exception and
the factory was never retried. Worse, a single poisoned slot took the whole pool down,
because connection selection dereferenced the throwing CreatedConnection getter for every
slot. The pool is a singleton and its slots are created once, so the transport stayed dead
until the pod restarted while the process kept running — a silent failure with no crash or
alert. Reproduces only on CAP + Redis Sentinel, which is why it never surfaced upstream.

Fix:
- Drop the Lazy<Task<RedisConnection>> base; hold the connect Task under a lock and recreate
  it when it faulted or was cancelled, reusing a healthy or in-flight Task otherwise.
- Replace the throwing CreatedConnection getter with a non-throwing
  TryGetCreatedConnection(out ...); pool selection and Dispose skip poisoned slots so one
  bad slot can no longer take the pool down.
- In the connect factory, dispose the multiplexer of every failed attempt and throw after
  exhausting retries, so the Task faults and the slot is recreated on next access — instead
  of caching a disconnected multiplexer as "healthy" forever.
- Make pool disposal atomic (Interlocked) and clean up in-flight connects.

This incidentally fixes two latent defects in the old retry loop: a disconnected multiplexer
returned as success after exhausting attempts, and up to four leaked multiplexers per cycle.

Adds DotNetCore.CAP.RedisStreams.Test covering recovery after failover, no-cache of a
disconnected multiplexer, per-attempt disposal, poisoned-slot isolation, and pool disposal.

BREAKING CHANGE: AsyncLazyRedisConnection no longer derives from Lazy<Task<RedisConnection>>
and no longer exposes CreatedConnection. It is internal transport plumbing; consider making
it internal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant