fix(scheduler): claim retry queue with jittered backoff + fresh rate gauges#104
Merged
Conversation
ephemerd_github_api_rate_remaining was defined but never written, so it
sat at 0 forever. Operators looking at Grafana couldn't tell 'budget
truly exhausted' from 'no data yet' from 'stale reading after window
reset'.
Wrap the GitHub HTTP client in a rateTrackingTransport that parses
X-RateLimit-{Limit,Remaining,Reset} on EVERY response (200 or 429/5xx
error — errors carry rate headers too, and that's exactly when we most
need them). Introduce three companion gauges:
ephemerd_github_api_rate_limit (ceiling for the window)
ephemerd_github_api_rate_reset_seconds (unix ts of next reset)
ephemerd_github_api_rate_updated_seconds (when we last observed)
Use rate_updated_seconds vs time() to age out a stale reading in
dashboards / alerts.
Also expose Client.RateSnapshot() for callers that want to bias
retry backoff based on the live rate state (used by the upcoming
scheduler claim-retry queue).
Guard against responses that don't emit rate headers (some GraphQL
paths) — those must NOT clobber the last known value to zero.
…re scheduling
GitHub does not re-deliver workflow_job webhooks. When ephemerd's
initial claim/JIT-runner-registration attempt failed with a transient
error (rate-limit exhausted, transient 5xx, network blip), the queued
job was dropped forever - operators noticed only via missing runs and
manually re-queued each job by hand.
Add an in-memory priority-heap retry queue keyed by jobKey. Failed
claims are re-scheduled on a jittered ladder (30s, 1m, 2m, 5m, 10m by
default), giving up after MaxAge (default 90m).
Hooks:
- handleLocalJob, handleLinuxJob, handleMacOSJob classify the claim
error via classifyErr and enqueue on retryable failures. The old
blind time.Sleep(backoffDuration) and 5s sleep are gone - the sem is
released first so we do not hold a slot idle across the wait.
- handleCompleted Drops the retry key so a job picked up elsewhere
(peer daemon sharing the installation token, or manual cancel) stops
getting reattempted.
- The event loop also Drops on 'in_progress', matching the same intent.
Rate-aware backoff: when the last GitHub API response reported
remaining=0 with a fresh (< 5m old) update timestamp and reset > now,
the next attempt is snapped to reset + [5s, 25s) jitter. This avoids
burning attempts against a provably-exhausted budget, and the jitter
prevents multiple daemons sharing the installation token from
stampeding at the exact same second.
Error classification (classifyErr) sorts errors into
non_retryable / rate_limit / server_side / network / unknown_retryable.
404/422/401/permission-denied are non-retryable; 429 and the go-github
RateLimitError/AbuseRateLimitError typed errors are rate_limit; 5xx
plus string-match fallbacks ("bad gateway", "gateway timeout") are
server_side; net.Error is network. Unknown errors are treated as
retryable within backoff/MaxAge - the whole point is to stop losing
jobs to unexpected transient failures.
Wiring:
- New RetryConfig on scheduler.Config, plumbed from [runner.claim_retry]
in the TOML (enabled defaults to true - losing jobs is worse than
extra retries).
- RateHint closure in cmd/ephemerd/main.go walks activeProviders for a
GitHub provider and binds to its RateSnapshot; nil when no rate-aware
provider is present.
- providers/github.Provider exposes RateSnapshot() forwarding to the
underlying github.Client (added in the earlier rate-transport commit).
Tests: retry_queue_test covers backoff progression (schedule ladder,
clamp past end), MaxAge give-up, Drop on completion, non-retryable
classification, rate-aware snap-to-reset with a fake clock, and the
full classifyErr table. retry_integration_test uses the existing
claimErrorProvider to end-to-end verify that a retryable failure in
handleLocalJob puts one entry in the queue and that a completed webhook
drops it. All new tests use table-driven or deterministic clock
patterns; no time.Sleep in scheduling logic.
This was referenced Jul 11, 2026
ephpm-claude Bot
added a commit
that referenced
this pull request
Jul 11, 2026
…106) The claim retry queue merged in #104 has three correctness bugs; fix them in place (config-gated and enabled by default, so this runs on every fleet host): 1. Backoff ladder never advances / give-up never fires. fireDue deleted the popped item from the index, so runOne's failure re-Add built a fresh item — attempts reset to 1 (stuck at schedule[0] ~30s) and firstFailure reset every fire (MaxAge never triggered). A job that couldn't be claimed retried every 30s forever, burning the shared installation token it was waiting on. Fix: fireDue keeps the item in the index across a fire. 2. Retries dropped after one attempt. With (1) half-fixed you get the opposite: retryHandler always returned nil, and runOne dropping on nil removed the item handleQueued had just re-Added synchronously — so a job self-healed at most once. Fix: retryHandler returns the real claim error (captured via a context pointer that enqueueRetryIfEligible fills in, which also suppresses the duplicate enqueue). runOne drops on success / advances on failure, with the rate-limit error class preserved so snap-to-reset still applies on later attempts. 3. Native macOS claim failures didn't self-heal — handleNativeMacOSJob was the one claim path not wired to the queue (Windows already routes through handleLocalJob). Wired it up. Tests: TestRetryQueue_StillFailingRetryStaysEnqueued drives the real fireDue->retryHandler->claim path (fails before this change); TestRetryQueue_LadderAdvancesThroughFirePath asserts the delay grows 30s->1m->2m and firstFailure is preserved. Full scheduler suite green, -race and go vet clean. Claude-Session: https://claude.ai/code/session_01ChVRC9ZrvQarrctDwSy1S5 Co-authored-by: Luther Monson <luther.monson@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase of the surviving pieces of closed #101 onto a moved main (post #95-#99).
What's in
pkg/scheduler/retry_queue.go+ tests +scheduler.gohooks +[runner.claim_retry]config +main.gowiring incl.buildRateHint). Fixes queued jobs being lost forever when the initial claim / JIT registration hits a transient error (webhooks are not re-delivered). Jittered ladder (30s/1m/2m/5m/10m, MaxAge 90m), rate-aware snap-to-reset when GitHub reports remaining=0.pkg/github/rateTrackingTransport+pkg/metrics).ephemerd_github_api_rate_remainingwas defined but never written; now parsed from every response (success and error) plus companion_limit,_reset_seconds,_updated_secondsgauges for staleness detection.Provider.RateSnapshot()feeds the retry queue's rate-aware backoff.What's dropped from #101
fix(startup): reflect actual discovery mode in log message(a243809) — fully superseded by feat(webhook): add tunnel = "external" for externally-managed ingress #98, which reworked webhook startup logging with more cases (external tunnel, external ingress, polling) than the original commit anticipated.fix(scheduler): record label-mismatched jobs in seen to stop re-adjudication(2e9421a) — per instructions, dropped in favor of fix(scheduler): stop re-provisioning undispatchable zombie jobs #95's approach. fix(scheduler): stop re-provisioning undispatchable zombie jobs #95 caps provisioning attempts per job (zombie guard), which does NOT cover label-mismatched jobs that return before the seen/pending writes. So there is a residual gap: on polling providers, a queued job whose OS labels do not match this platform is re-adjudicated and re-logged every poll tick until GitHub removes it. Debug-level log, but drains API budget. Fix belongs in a separate small PR that reordershandleQueuedto writeseenbefore returning on label mismatch.Conflicts resolved
pkg/scheduler/scheduler.go(New()): mergedattemptsmap +nativeMacSem(fix(scheduler): stop re-provisioning undispatchable zombie jobs #95, feat(native): native macOS runner mode for trusted repos #91) with thes := &Scheduler{...}refactor from the retry commit.pkg/config/config.go(RunnerConfig): kept feat(native): native macOS runner mode for trusted repos #91'sMacOSfield alongside the newClaimRetryfield;ClaimRetryTomltype placed afterMacOSRunnerConfigmethods.cmd/ephemerd/main.go(scheduler config literal): merged feat(native): native macOS runner mode for trusted repos #91'sMaxNativeMac/MacOSModeForRepo/NativeMacUser/RunnerDirfields with the newRetry: retryCfgfield.Validation
go build ./pkg/... ./cmd/...clean (repo-levelgo build ./...exits 1 on this Windows box due to a pre-existingruntime/cgoissue unrelated to this change; main behaves the same).go vet ./pkg/scheduler/... ./pkg/github/... ./pkg/config/...clean.go test ./pkg/scheduler/... ./pkg/github/... ./pkg/config/...all green. Retry-queue fake-clock tests from fix(scheduler): claim retry queue, rate-aware backoff, seen-on-skip, fresh rate gauges #101 pass unmodified.TestHandleQueued_ZombieSkip,TestCleanSeen_PrunesAttempts,TestHandleQueued_SkipsMismatchedLabels(from fix(scheduler): stop re-provisioning undispatchable zombie jobs #95) still pass.Closes issues fixed by original #101 (dropped webhook-job claim failures; stale rate gauges).