PROD-5987: fix race condition in HzCacheMemoryLocker causing disposed semaphore errors

[JoelNygren-Norce]

Summary\n\nHzCacheMemoryLocker stores SemaphoreSlim instances in a MemoryCache with 5-minute sliding expiration. The post-eviction callback disposes the semaphore, but there's a race between GetSemaphore() returning a reference and WaitAsync()/Wait() using it — the MemoryCache can evict+dispose the semaphore in that window, causing ObjectDisposedException to propagate as a fatal "Could not acquire lock" error.\n\nFix: catch ObjectDisposedException in both AcquireLockAsync and AcquireLock, remove the stale lockCache entry, and retry with a fresh semaphore:\n\ndiff\n+try\n+{\n acquired = await semaphore.WaitAsync(timeout, token);\n+}\n+catch (ObjectDisposedException)\n+{\n+ lockCache.Remove(key);\n+ semaphore = GetSemaphore(cacheName, cacheInstanceId, key, logger);\n+ acquired = await semaphore.WaitAsync(timeout, token);\n+}\n\n\nObserved in production Admin on 2026-06-10 (09:07–09:52 UTC, ~2,375 errors). The race was amplified by background serialization failures (StartUpdateChecksumAndNotify) causing contention and GC pressure.\n\nJira: PROD-5987">

Link to Devin session: https://app.devin.ai/sessions/3e7c5915fd8640e69f53c7ad184e302d
Requested by: @JoelNygren-Norce

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[Copilot]

Pull request overview

This PR addresses a race in HzCacheMemoryLocker where SemaphoreSlim instances stored in MemoryCache can be evicted/disposed between GetSemaphore() and Wait()/WaitAsync(), causing ObjectDisposedException to escape and be treated as a fatal “Could not acquire lock” error.

Changes:

• Added maxLockRetries to HzCacheMemoryLockerOptions to control retry behavior on disposed semaphores.
• Wrapped SemaphoreSlim.WaitAsync() in a retry loop that removes the stale cache entry and re-creates the semaphore when ObjectDisposedException occurs.
• Applied the same retry pattern to the synchronous SemaphoreSlim.Wait() path.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.