fix(RedisStreams): recover connection pool after a failed connect instead of caching it forever#1808
Open
rebaseandpanic wants to merge 1 commit into
Conversation
…tead of caching it forever AsyncLazyRedisConnection derived from Lazy<Task<RedisConnection>>, which caches the connect Task permanently. When the initial connect faulted — the Redis Sentinel path (GetSentinelMasterConnection) throws, unlike a normal abortConnect=false connect — the faulted Task was cached forever: every subsequent access rethrew the same exception and the factory was never retried. Worse, a single poisoned slot took the whole pool down, because connection selection dereferenced the throwing CreatedConnection getter for every slot. The pool is a singleton and its slots are created once, so the transport stayed dead until the pod restarted while the process kept running — a silent failure with no crash or alert. Reproduces only on CAP + Redis Sentinel, which is why it never surfaced upstream. Fix: - Drop the Lazy<Task<RedisConnection>> base; hold the connect Task under a lock and recreate it when it faulted or was cancelled, reusing a healthy or in-flight Task otherwise. - Replace the throwing CreatedConnection getter with a non-throwing TryGetCreatedConnection(out ...); pool selection and Dispose skip poisoned slots so one bad slot can no longer take the pool down. - In the connect factory, dispose the multiplexer of every failed attempt and throw after exhausting retries, so the Task faults and the slot is recreated on next access — instead of caching a disconnected multiplexer as "healthy" forever. - Make pool disposal atomic (Interlocked) and clean up in-flight connects. This incidentally fixes two latent defects in the old retry loop: a disconnected multiplexer returned as success after exhausting attempts, and up to four leaked multiplexers per cycle. Adds DotNetCore.CAP.RedisStreams.Test covering recovery after failover, no-cache of a disconnected multiplexer, per-attempt disposal, poisoned-slot isolation, and pool disposal. BREAKING CHANGE: AsyncLazyRedisConnection no longer derives from Lazy<Task<RedisConnection>> and no longer exposes CreatedConnection. It is internal transport plumbing; consider making it internal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
The Redis Streams transport's connection pool could get permanently stuck after a
failed initial connect and never recover, even once Redis was reachable again. The
process stayed alive but the transport was dead — publishes and consumes silently
stopped, with no crash or exception surfaced to health checks.
Root cause:
AsyncLazyRedisConnectionderived fromLazy<Task<RedisConnection>>,which caches the connect
Taskpermanently. If that first connect faulted, thefaulted
Taskwas cached forever — every later access rethrew the same cachedexception and the connect factory was never retried. Because the pool is a singleton
whose slots are created once, nothing ever reset the poisoned slot. Worse, connection
selection dereferenced the throwing
CreatedConnectiongetter for every slot, so asingle poisoned slot took the whole pool down for both publish and consume.
This only reproduces on CAP + Redis Sentinel: a normal
abortConnect=falseconnect does not throw (the multiplexer reconnects itself), but the Sentinel path
(
GetSentinelMasterConnection) throws — which is what faults the cachedTask. Thatnarrow combination is why it hasn't surfaced before.
Issue(s) addressed:
Changes:
Lazy<Task<RedisConnection>>base. Hold the connectTaskunder a lock andrecreate it when it faulted or was cancelled, while reusing a healthy or in-flight
Task.CreatedConnectiongetter with a non-throwingTryGetCreatedConnection(out ...). Pool selection andDisposeskip poisoned slots, soone bad slot can no longer take the pool down.
exhausting retries, so the
Taskfaults and the slot is recreated on the next access —instead of caching a disconnected multiplexer as "healthy" forever.
Interlocked) and clean up in-flight connects.DotNetCore.CAP.RedisStreams.Testwith unit tests (a connection-factory seam isinjected so the retry/recovery logic runs against a fake multiplexer, no live Redis).
Incidentally fixes two latent defects in the old retry loop: a disconnected multiplexer
returned as success after exhausting attempts, and up to four leaked multiplexers per cycle.
Affected components:
DotNetCore.CAP.RedisStreams(connection pool only —IConnectionPool.LazyConnection.cs,IConnectionPool.Default.cs). No public API of CAP itself changes.How to test:
dotnet test test/DotNetCore.CAP.RedisStreams.Test/The tests cover: recovery after a failed connect once Redis returns, a disconnected
multiplexer never being cached as healthy, every failed attempt being disposed (no leak),
a single poisoned slot not taking down selection of healthy slots, and pool disposal
(exact-once, in-flight cleanup). All deterministic — no network, no
Thread.Sleep.Additional notes (optional):
BREAKING CHANGE:
AsyncLazyRedisConnectionno longer derives fromLazy<Task<RedisConnection>>and no longer exposesCreatedConnection. This isunavoidable — the
Lazy<Task<T>>base is the bug. The type is internal transportplumbing with no intended external consumers; I'd suggest marking it
internal(happy todo that here if you prefer).
One coverage gap left intentionally: the least-loaded selection path (comparing non-zero
ConnectionCapacity) isn't unit-tested, because forcing a non-zeroServerCounters.TotalOutstandingon a fake would require reflecting intoStackExchange.Redis internals — too brittle to put in the suite. The behaviour is
unchanged from before.
Checklist: