Skip to content

fix(scheduler): correct claim retry queue (ladder, give-up, native)#106

Merged
ephpm-claude[bot] merged 1 commit into
mainfrom
fix/retry-queue-corrections
Jul 11, 2026
Merged

fix(scheduler): correct claim retry queue (ladder, give-up, native)#106
ephpm-claude[bot] merged 1 commit into
mainfrom
fix/retry-queue-corrections

Conversation

@ephpm-claude

@ephpm-claude ephpm-claude Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

The claim retry queue merged in #104 ships three correctness bugs. This patches them in place on current main (the queue is config-gated but enabled by default, so it runs on every host). Supersedes my earlier #103, which went stale once #104/#105 moved main.

  1. Ladder never advances / give-up never fires. fireDue deleted the popped item from the index, so runOne's failure re-Add built a fresh item — attempts reset to 1 (stuck at ~30s) and firstFailure reset every fire (MaxAge never triggered). An unclaimable job retried every 30s forever, burning the very shared token it was waiting on. Fix: fireDue keeps the item in the index across a fire.
  2. Retries dropped after one attempt. retryHandler always returned nil, and runOne dropping on nil removed the item handleQueued had just re-Added synchronously. Fix: retryHandler returns the real claim error via a context pointer that enqueueRetryIfEligible fills in (also suppressing the duplicate enqueue); the rate-limit class is preserved so snap-to-reset still works.
  3. Native macОS didn't self-healhandleNativeMacOSJob was the one claim path not wired to the queue. Wired it up.

Tests

  • TestRetryQueue_StillFailingRetryStaysEnqueued drives the real fireDue→retryHandler→claim path and fails before this change.
  • TestRetryQueue_LadderAdvancesThroughFirePath asserts the delay grows 30s→1m→2m and firstFailure is preserved.
  • Full scheduler suite green, -race and go vet clean.

🤖 Generated with Claude Code

The claim retry queue merged in #104 has three correctness bugs; fix
them in place (config-gated and enabled by default, so this runs on
every fleet host):

1. Backoff ladder never advances / give-up never fires. fireDue deleted
   the popped item from the index, so runOne's failure re-Add built a
   fresh item — attempts reset to 1 (stuck at schedule[0] ~30s) and
   firstFailure reset every fire (MaxAge never triggered). A job that
   couldn't be claimed retried every 30s forever, burning the shared
   installation token it was waiting on. Fix: fireDue keeps the item in
   the index across a fire.

2. Retries dropped after one attempt. With (1) half-fixed you get the
   opposite: retryHandler always returned nil, and runOne dropping on
   nil removed the item handleQueued had just re-Added synchronously —
   so a job self-healed at most once. Fix: retryHandler returns the real
   claim error (captured via a context pointer that enqueueRetryIfEligible
   fills in, which also suppresses the duplicate enqueue). runOne drops
   on success / advances on failure, with the rate-limit error class
   preserved so snap-to-reset still applies on later attempts.

3. Native macOS claim failures didn't self-heal — handleNativeMacOSJob
   was the one claim path not wired to the queue (Windows already routes
   through handleLocalJob). Wired it up.

Tests: TestRetryQueue_StillFailingRetryStaysEnqueued drives the real
fireDue->retryHandler->claim path (fails before this change);
TestRetryQueue_LadderAdvancesThroughFirePath asserts the delay grows
30s->1m->2m and firstFailure is preserved. Full scheduler suite green,
-race and go vet clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ChVRC9ZrvQarrctDwSy1S5
@ephpm-claude ephpm-claude Bot merged commit e12dc1a into main Jul 11, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant