Skip to content

feat: Implement automatic reclaimer backoff jitter to prevent thundering herd #122

Open

Description

@Shreesh-Sree

Feature Summary

The queue reclaimer processes pending or stalled submissions across the Redis 3-tier priority streams (ac:submissions:run, submit, bulk). Under high node restart events or cluster network recovery, concurrent reclaimers may query Redis at the same millisecond, risking a thundering herd.

Proposed Implementation

  1. Add full jitter (rand.Float64() * baseInterval) to the reclaimer loop sleep interval in internal/queue.
  2. Add Prometheus counter ac_reclaimer_claims_total{stream="...", result="recovered|poison"}.
  3. Provide unit test verifying jitter distribution and absence of synchronized bursts.

Acceptance Criteria

  • Jittered interval prevents synchronous lock-step queries.
  • Clean metrics incrementing upon each claim cycle.

Activity

  1. harichillin commented on Nov 12, 2026

    @harichillin
    Contributor

    /assign

  2. github-actions commented on Nov 12, 2026

    @github-actions

    Issue assigned to @harichillin (FCFS)! 🎉\n\nPlease submit a PR when ready and reference this issue in your PR description (e.g. Fixes #122). If you need to drop this issue, comment /unassign so someone else can pick it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestperformancePerformance improvements and benchmarksworker

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions