Spec: wait out GitHub quota exhaustion instead of failing (Quota wait) #1994

This was generated by AI from a grill-with-docs session on #739 (decisions Q1–Q24 in that conversation, summarised on #739).

Problem Statement

Every host in the fleet authenticates gh as the same GitHub user, so every Pass and every running Command on every host draws on one 5,000-per-hour REST quota and one GraphQL quota. When that quota runs out, thirdshift keeps going as if nothing happened:

• Scheduled Passes keep starting and fail on their first GitHub read, minute after minute, until the reset. Some of those failures send misleading Run notifications (an Architect run reporting review failed before any review started).
• A Command that is already running fails outright on a quota error, however far it has got. On 2026-11-10 the Pickup run of Deepen the Grader panel's retained-call lifecycle #1968 stopped in CI watch after 1h 6m because one REST read hit API rate limit exceeded. The reset came about 70 minutes later; the run ended failed and needed a manual Continuation.
• Passes spend the last of the quota with no thought for Commands already running, which need it more.

The Inbox already protects itself: it reads both quotas from gh api rate_limit, keeps a 500-unit reserve and pauses its own GitHub reads until the reset. Nothing else does.

Solution

thirdshift recognises a Quota wait: the time until GitHub's quota for a GitHub user resets, during which Passes are skipped and running Commands wait instead of failing.

• Each machine keeps one Quota wait record, separately for each GitHub host, authenticated user and resource (REST core and GraphQL). It is written whenever a GitHub call fails on primary quota exhaustion, or a quota check finds a resource below the reserve. The Inbox reads and writes the same record.
• Before any GitHub work, every Pass checks the record and the current quota. It is skipped while either resource is in a Quota wait or below a fixed reserve of 500. It exits 0, records the reason in the Activity log only when that reason changes, and sends no email. status and the Console Inbox show the Quota wait.
• A running Command that hits primary quota exhaustion waits until the reset, then carries on. This covers reads and writes alike, from CI watch to Delivery to Claim cleanup. Its stage shows the Quota wait, and its Run notification says how long it waited. A short secondary rate limit gets a bounded retry instead.

User Stories

1. As the Day shift, I want a Ticket run in CI watch to survive a quota exhaustion, so that an hour of work is not lost to a reset that comes on its own.
2. As the Day shift, I want a running Command that hits a quota exhaustion during Delivery (pushing, opening or updating the pull request, labelling) to wait and carry on, so that the run ends as it would have without the exhaustion.
3. As the Day shift, I want a Spec run, a Review run, a Fix run, a Security run, a Triage run and a Custom process to wait the same way, so that no kind of Command is the weak link.
4. As the Day shift, I want a waiting Command to wait until the reset GitHub reports, plus one minute, plus a random 0–60 seconds, so that it resumes as soon as it safely can and waiting Commands on one machine do not all resume in the same second.
5. As the Day shift, I want a Command's total Quota wait across its run capped at 2 hours, beyond which it fails as it does today, so that a run never hangs indefinitely on a quota that keeps running out.
6. As the Day shift, I want Stop to end a running Command's Quota wait at once, so that I can still stop a Command that is waiting.
7. As the Day shift, I want Claim cleanup that runs after a Stop or other interruption to wait for the reset rather than fail, so that an issue is never left in-progress because the quota ran out at the end.
8. As the Day shift, I want a waiting Command's stage, in status, show and its Console row, to read Quota wait until <time>, so that I can see why it is not moving.
9. As the Day shift, I want a Command that waited to say in its Run notification how long it waited and for which resource, e.g. Waited 47 min for the GitHub quota (REST core), so that a long run is explained.
10. As the Day shift, I want Slow to keep measuring a Ticket run as it does now, wait included, so that the existing signal does not change meaning.
11. As the Day shift, I want a running Command that hits a secondary rate limit to wait 60 seconds and retry, up to 5 times, before failing as today, so that a short abuse throttle does not end a run.
12. As an operator running scheduled Passes, I want every Pass (Pickup, Architect, Security, Triage, Review with an author, Fix with no pull request, and a Custom process started with no issue) to check the quota before any GitHub work, so that no Pass fails minute after minute while the quota is out.
13. As an operator, I want a Pass skipped while either resource is in a Quota wait, so that it cannot fail on the resource it happens to need.
14. As an operator, I want a Pass skipped while either resource is below a reserve of 500, so that Commands already running keep enough quota to finish.
15. As an operator, I want the quota check itself to cost nothing from the quota (gh api rate_limit), so that checking does not make the problem worse.
16. As an operator, I want a Pass to go ahead as it does today when the quota check itself fails (for example on a network error), so that a broken check never stops all factory work.
17. As an operator, I want a skipped Pass to exit 0 and send no email, so that my scheduler sees no failure and my inbox stays quiet.
18. As an operator, I want a skipped Pass to record skipped: GitHub Quota wait until <time> (<resource>: <n> left) in the Activity log only when the reason changes, so that a long wait leaves one line, not one per Pass.
19. As an operator, I want the same rule whether a Pass is started by cron or typed by hand, so that a manual Pass during a Quota wait behaves predictably.
20. As an operator, I want status and the Console Inbox to show GitHub Quota wait until <time> beside the Pickup last fired lines, so that I can see at a glance why the factory is idle.
21. As an operator with several hosts on one GitHub user, I want each machine to learn of a Quota wait from its own first failed call or reserve check, so that no shared infrastructure between hosts is needed.
22. As an operator with more than one GitHub user or GitHub host configured, I want each user's and host's quota tracked separately, so that one account's exhaustion never holds back another.
23. As an operator, I want the Inbox's existing quota pause to use the same Quota wait record, so that the machine has one view of the quota.
24. As an operator, I want docs/guide.md to say that every host authenticated as the same GitHub user shares one quota, and that scheduling should be budgeted per user across hosts, so that I size my crontabs correctly.
25. As a maintainer, I want the Glossary to define Quota wait, so that issues and docs use one term for it.
26. As a maintainer, I want an agent session's own gh calls left as they are, so that this work stays inside thirdshift's own GitHub calls.

Implementation Decisions

• One record per machine. A Quota wait record lives under the thirdshift state directory. It is keyed by GitHub host, authenticated user login and resource (REST core, GraphQL), holds the reset time and the last observed remaining count, and never stores a token or other secret. Every Pass, every Command and the Inbox on the machine read and write it. Writes are atomic so concurrent Commands cannot corrupt it. An entry whose reset time has passed is ignored.
• How it is learned. (a) A thirdshift GitHub call fails with GitHub's primary-quota diagnostics (the messages operational_cause already recognises as rate-limit exhaustion, other than the secondary-limit message). thirdshift then reads gh api rate_limit for the reset time and records it. (b) A Pass's admission check reads gh api rate_limit and records any resource below the reserve. (c) The Inbox's existing quota observations record into it. If no reset time can be read, thirdshift assumes one 15 minutes ahead.
• Pass admission. Every Pass checks the record and then gh api rate_limit before any other GitHub work, including the Sweep. It sits beside the existing Command-limit admission, so a Pass skipped for the quota takes no slot. The reserve is a fixed 500 for each resource, the Inbox's existing figure, not a User config setting. The skip uses the existing skip path: exit 0, nothing on stdout, Activity log line only when the reason changes, no Run notification. If the check fails for any reason other than exhaustion, the Pass proceeds.
• Waiting in a running Command. The GitHub adapter's primary-quota failure (read or write) becomes a wait on the record's reset time, plus one minute, plus a random 0–60 seconds, then a retry of the same call. Total waiting per Command is capped at 2 hours; past the cap, the call fails exactly as today. The wait is interruptible like other waits (Stop and interruption end it), except for work that already finishes after interruption (Claim ending, Failed ticket run salvage, confirmed Self-merge completion, Spec accounting), which keeps waiting within the cap.
• Secondary limits. A secondary-limit failure in a running Command waits 60 seconds and retries, up to 5 times, then fails as today. Secondary limits are not recorded as a Quota wait and do not affect Pass admission.
• Showing it. While waiting, a Command's record stage is Quota wait until <time>. status, show and the Console render it like any other stage, and the stage before it resumes afterwards. The Run notification gains one line when the Command waited at all, with the total wait and the resource. status and the Console Inbox show a machine-level GitHub Quota wait until <time> line while the record holds one.
• Unchanged. Slow's timing; the Failure limit (a Quota wait is not a failure); Claims and reservations; agent sessions' own gh calls; crontabs (thirdshift never edits them); ADR 0009's short-lived OS-scheduled Passes; ADR 0011's Activity log behaviour.
• Glossary. T1 adds Quota wait to GLOSSARY.md once the behaviour exists, defined without implementation detail.

Testing Decisions

• Test external behaviour only: what a Pass or Command does, prints, records and labels when GitHub reports exhaustion, not how the record is stored.

• End-to-end seam: the thirdshift binary against the fake gh (tests/fakes/gh.rs). The fake already answers gh api rate_limit from rate_limits, replaces it with next_rate_limits after each answer, and fails reads with gh's real rate-limit diagnostics. Tests set reset times a few seconds ahead so real waits stay short. Cover:

  • a Pass skipped during a Quota wait and below the reserve, with its single Activity log line and exit 0;
  • a Pass that proceeds when rate_limit itself fails;
  • a Ticket run in CI watch that hits exhaustion, waits, resumes and delivers;
  • Claim cleanup after Stop waiting for the reset;
  • the Run notification line;
  • the status line.

• Unit seam: inline tests on the Quota wait module with an injected clock and random source. Cover:

  • the 2-hour cap;
  • the 0–60 second spread;
  • the secondary-limit retry count;
  • separation by host, user and resource;
  • expiry of past entries;
  • the 15-minute fallback when no reset time is available.

  The Inbox's existing quota tests move onto this module.

• Prior art: src/github/execution_tests.rs (fake GitHub failures and transient retries), src/ci.rs inline tests (bounded retry of check reads), the Inbox quota tests next to src/github/quota.rs and src/logs/github.rs, and the Pass skip tests in tests/ for Command-limit skips.

Out of Scope

• Measuring and cutting the GitHub reads idle Passes make (Measure and cut the GitHub reads idle Passes make #1992).
• GitHub App authentication for the fleet (Authenticate the fleet as a GitHub App to raise its API quota #1993).
• Sharing a Quota wait between hosts.
• A numeric read budget or maximum pickup latency target.
• Retrying agent sessions' own gh calls.
• Changing any host's crontab.

Further Notes

• Defer scheduled passes during GitHub quota exhaustion and reduce idle polling #739 holds the original evidence: the October 2026 GraphQL exhaustion, and the 2026-11-10 loss of Deepen the Grader panel's retained-call lifecycle #1968 in CI watch with the fleet's Pass counts. It is closed as superseded by this Spec.
• The 2026-11-10 operator mitigation slowed the fleet's crontabs from about 840 to about 330 Pass starts an hour. Slower schedules lower the odds of exhaustion but cannot rule it out, and do nothing for a Command already running when it happens.