Skip to content

Spec: wait out GitHub quota exhaustion instead of failing (Quota wait) - #2004

Draft
JacobStephens2 wants to merge 29 commits into
mainfrom
issue-1994
Draft

JacobStephens2 wants to merge 29 commits into
mainfrom
issue-1994

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Passes, running Commands and the Inbox share machine-local GitHub Quota waits, separated by GitHub host, authenticated user and resource. Scheduled work preserves a 500-unit reserve; active work survives primary exhaustion and explains the wait in its Run notification.

Pass → shared wait or either quota below 500 → quiet skip, exit 0, no email
     → check unavailable                    → continue
Command GitHub call → primary exhaustion → record reset → interruptible wait → retry
                    → secondary limit   → 60-second retry, at most five retries
Inbox ↔ same quota record → status, Console warning and reset recovery

Primary waits resume after the reset plus one minute and 0–60 seconds of jitter, within a two-hour budget shared with child Runs. Completion work, including Claim cleanup after Stop, retains its completion policy. Waiting overlays the stage without resetting Slow's timing. Activity skips are deduplicated, including alternating Custom processes. The guide, Glossary and public site explain the shared quota across hosts.

Whole-Spec review also fixed Custom-process quota recovery before issue validation, branch selection and Claim creation, and made admission retain a valid low-quota observation when login discovery fails. Duplicate regression setup was consolidated.

Closes #1994

Evidence

Execution evidence

Testing seams: the thirdshift binary against fake gh, observed through CLI outcomes, Activity and Command records, notifications and Console HTTP; the Quota wait module's injected clock and random source.

  • Before: cargo test --test process issue_branch_process_retries_quota_during_branch_selection -- --exact --nocapture failed with gh pr list failed: GraphQL: API rate limit exceeded (HTTP 403) and left: Some(1), right: Some(0).
    After: the same command passed: 1 passed; 0 failed. The regression's fake implementation session also supplies its required final message so the recovered Process can finish Delivery.
  • Before: cargo test --test pass_quota exhausted_quotas_skip_even_when_login_cannot_be_probed -- --exact --nocapture failed its quota-skip assertion, printing no Ready issue on acme/widgets after observing zero quota.
    After: the same command passed: 1 passed; 0 failed.
  • scripts/affected-tests refs/remotes/origin/main: site tests passed (94.39s), locked build passed (0.47s), and the complete Console browser rerun passed all 190 tests (196.52s). Rust coverage completed in segments after the unrelated assertion below; all 101 integration targets were accounted for, alongside the binary's unit suite.
  • The initial Console run passed 189 tests and timed out waiting for a row in Inbox row Dismiss works in ready for review with JavaScript on a phone. The exact focused reproduction, node --test --test-name-pattern='^Inbox row Dismiss works in ready for review with JavaScript on a phone$' console/tests/inbox.test.cjs, passed (1 passed; 0 failed); the complete Console rerun also passed.
  • The Rust run passed 1,772 unit tests (3 ignored) and the early integration targets. Its Experiment file passed 190 tests (1 ignored) and failed an unrelated substring assertion: tests/experiments/process.rs:111 found 421 inside a generated directory hash (...ca842103...), rather than private PR data. cargo test --test experiments process::process_session_evidence_check_all_private_fix_identifiers_and_abbreviated_commits -- --exact --nocapture then passed (1 passed; 0 failed, 9.67s). All remaining 81 integration targets passed using cargo test --test <target> selectors; the exact expanded command is retained in remaining-rust.log. The initial Rust run took 1,465.08s, including 3m04s compilation; the resumed targets took 1,234.48s with eight test threads. The affected-test entry point did not finish green in one invocation; both unrelated failures and their passing reruns are recorded here.

Visual evidence

Matched Inbox captures use Chromium with the same fixture and expanded Last fired Passes, at 1440×900 desktop and 390×844 phone viewports. Before uses refs/remotes/origin/main; After shows the shared Quota wait beside the Pass information. The capturing machine's hostname is anonymized. Screenshots are retained with the Session evidence; this session does not push evidence or repository branches.

Review coverage

Independent Standards and Spec reviewers inspected all 53 original changed paths and reviewed the fixes, including the additional src/processes/source.rs change. Both axes cover all 54 final changed paths. Unread changed files: none. Generated-output exclusions: none; the generated manual's changed hunks were read directly. Reports and counts are retained in .thirdshift-review-Ddsd5K/standards.md, spec.md and summary.json.

The fixed point was pinned when review began: refs/remotes/origin/main at 09f0c6c49e2c056230c38053b8d86a42d9a214f5. That remote-tracking ref advanced during the long validation run; review coverage remains against the pinned commit.

Unaddressed findings

Standards

None. The one duplication finding was addressed by one regression scenario covering both Custom-process name pairs.

Spec

None. Both behavioral findings were reproduced, fixed and retained as regressions. No finding requires a Day shift or Spec decision.

Merge Danger

Door: two-way. Reverting restores the previous quota handling; machine-local quota and wait-total sidecars contain no token and require no migration.

Blast Radius: factory. Every Pass and thirdshift-owned GitHub call shares this policy. Stop, completion cleanup, account separation, notification privacy and Inbox recovery are covered by regressions. The branch remains a draft for thirdshift's remaining review and publication steps.

JacobStephens2 and others added 29 commits November 10, 2026 20:26
Share machine-local GitHub Quota waits with the Inbox
Skip every Pass during GitHub Quota waits
Preserve Pass admission alongside running Command quota recovery. Keep admission probes from waiting, reconcile quota and Security privacy fixtures with admission, and wait for the asynchronous Inbox refresh in its quota-resume test.
The issue-1994 merge into main added base_ci_skipped alongside the quota History selector. Preserve the Kind-based activity_write interface so that clean textual merge compiles, while named Process quota waits still select their own history. Fixes #2012.
Quota warnings expire with the clock before the background read necessarily completes. The CI regression raced that expiry against the GraphQL call-count assertion. Change the fake pull request to merged and wait for its Console row to reach Resolved, keeping the pre-reset no-read check and resumed-read assertion intact.
Fix inherited CI failures on the Quota wait branch
Keep the quota-reset regression synchronized with the observed GitHub state and preserve the Activity log Kind interface alongside named Process quota history.

Validated with cargo check, focused Console and recording tests, cargo fmt --check, all 190 Console browser tests, and scripts/affected-tests origin/issue-1994 (site and full Rust suites).
Wait out GitHub quota exhaustion in running Commands
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spec: wait out GitHub quota exhaustion instead of failing (Quota wait)

1 participant