Skip to content

Make a running Command wait out GitHub quota exhaustion instead of failing #1997

Closed

Description

@JacobStephens2

Parent

#1994

What to build

A running Command that hits GitHub's quota waits and carries on instead of failing, and says so. This applies to a Ticket run, Spec run, Review run, Fix run, Security run, Triage run, Custom process and the Command a Pass dispatched.

  • Primary exhaustion. When thirdshift's own GitHub call, read or write, fails on primary quota exhaustion, it records the Quota wait and waits until reset + 1 minute + a random 0–60 seconds, then retries the same call. The total wait per Command is capped at 2 hours; past the cap, the call fails exactly as today. This covers CI watch, Delivery (push, pull request creation and update, labels, comments, Self-merge) and everything else that goes through the GitHub adapter.
  • Interruption. Stop and other interruption end an ordinary wait at once. Work that already finishes after interruption keeps waiting within the cap: Claim ending, Failed ticket run salvage, confirmed Self-merge completion and Spec accounting. So an issue is not left in-progress because the quota ran out at the end.
  • Secondary limits. A secondary-limit failure waits 60 seconds and retries, up to 5 times, then fails as today. It is not recorded as a Quota wait.
  • Visible. While waiting, the Command record's stage is Quota wait until <time>, shown by status, show and the Console. The previous stage resumes afterwards. A Command that waited at all gets one Run notification line: Waited <duration> for the GitHub quota (<resource>).
  • Unchanged. Slow keeps timing the run wait included; a Quota wait is not a failure and never adds agent-failed; agent sessions' own gh calls are untouched.
  • docs/guide.md: describe the wait, the cap, the secondary-limit retry and the notification line where CI watch, Delivery and Run notifications are described. Regenerate the user manual and update hand-written site pages that describe how a run fails on GitHub errors.

Acceptance criteria

  • A Ticket run whose CI watch read hits primary exhaustion waits, resumes after the reset and delivers its pull request, ending as it would have without the exhaustion (end to end against the fake gh, reset a few seconds ahead).
  • A write during Delivery that hits primary exhaustion waits and then succeeds.
  • Claim cleanup after Stop waits for the reset and leaves the labels a Stop leaves; an ordinary wait ends at once on Stop.
  • Past the 2-hour total, the Command fails exactly as it does today (unit test with injected clock).
  • A secondary-limit failure is retried every 60 seconds up to 5 times, then fails as today, and records no Quota wait.
  • status, show and the Console show Quota wait until <time> as the stage while waiting.
  • The Run notification carries the wait line only for a Command that waited.
  • docs/guide.md and the generated manual describe the behaviour; npm run docs:check --prefix site passes.
  • scripts/affected-tests main passes.

Activity

  1. JacobStephens2 commented on Nov 11, 2026

    @JacobStephens2
    OwnerAuthor

    This was generated by AI during triage.

    Additional evidence from investigation of duplicate #1973 against Base commit 6f4ba5d: quota failures bypass retries both in CI watch (src/ci.rs:273, via src/github/check_read.rs:46) and ordinary GitHub adapter reads (src/github.rs:1246). The existing tests permanent_and_quota_read_failures_are_not_retried and check_reads_classify_transport_and_api_failures_without_hiding_diagnostics both passed with cargo test --bin thirdshift github::execution_tests:: -- --exact. These tests currently encode immediate failure on primary exhaustion and should be reconciled with the new quota-wait policy while preserving non-quota permanent-error behavior.

    #1973 also reports repeated Ticket failures on #1895 and #1898, alongside #1968, and a Review failure reading stephens.page PR #30. Those incident logs were not independently inspected. They reinforce the need for #1997's adapter-wide coverage rather than a CI-only fix.

  2. JacobStephens2 commented on Nov 11, 2026

    @JacobStephens2
    OwnerAuthor

    Closed by #2007, merged into issue-1994 by a thirdshift Ticket run with Self-merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions