Skip to content

Wait out GitHub quota exhaustion in running Commands - #2007

Merged
JacobStephens2 merged 10 commits into
issue-1994from
issue-1997
Nov 11, 2026
Merged

JacobStephens2 merged 10 commits into
issue-1994from
issue-1997

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

GitHub call fails
  primary quota → record wait → reserve shared Command budget (≤2h)
                → show Quota wait → wait until reset + 1m + jitter → retry
  secondary limit → wait 60s → retry up to 5 times
  cap reached → preserve the existing failure path

Ordinary waits stop on interruption; completion work retains its existing interruption policy. Child Ticket runs share the Command budget and one notification total. Preflight waits are visible and stoppable, and skipped Passes finish their provisional records.

Closes #1997

Seams under test: GitHub adapter and injected-clock wait planner; shared Command budget across child processes; Ticket Delivery/CI watch/Stop/notifications against fake gh; Command recording and the readers behind status, show and the Console.

Evidence

Execution evidence

  • Repair validation on 890be3c3: PLAYWRIGHT_BROWSERS_PATH="$PWD/.affected-tests/playwright" scripts/affected-tests origin/issue-1994 passed: all 68 site checks (81.14s) and the full Rust cargo test suite across 102 binaries (4,214 passed, 0 failed, 99 existing ignored tests; 1,799.46s with a warm build cache, compilation included). No source or check changes were required.
  • Before: target/debug/deps/quota_wait-0ece3cefb6364f52 --exact delivery_write_waits_for_primary_quota_and_notifies_once, against the preserved Base binary: exit 101; gh pr ready 1 --repo acme/widgets failed: GraphQL: API rate limit exceeded (HTTP 403); expected exit 0, got 1.
    After: same command: 1 passed; 0 failed.
  • Before: .thirdshift-review-ijm3DU/preflight_wait: exit 101, 0 active runs, Stop failed no Command matches widgets#7.
    After: same command: exit 0, 1 active run, Quota wait until ..., Stop exit 0. Regression retained in tests/quota_wait.rs.
  • Before: .thirdshift-review-ijm3DU/quota_skip: exit 101; successful Pickup skip appeared as Died.
    After: same command: exit 0; Command appears in Done as skipped. Regression retained in tests/quota_wait.rs.
  • cargo test --test quota_wait: 8 passed; 1 ignored subprocess entry point. Covers primary reads/writes, Stop and completion cleanup, cap, secondary limits, child accounting, preflight and skipped Passes.
  • The focused Console, Slow timing, injected-clock cap, stage restoration and diagnostic classifier checks passed. cargo check --bin thirdshift, formatting and manual regeneration checks passed.
  • scripts/affected-tests main: passed, 4892.66s total. Go, Console/core Python, TypeScript typechecking and 62 tests, 68 site tests, Console browser tests, the full Rust suite, and release-generator checks all passed. The full Rust cargo test took 4077.91s with compilation included and 3 test threads; nextest was unavailable. The Security privacy fixture was rebuilt before its full-suite invocation and also passed its exact targeted test.
  • CI is green on b0caa96200d0821bb21efcc8d647d11f68c6e2bf: Rust/lint checks, site and Console browser checks, plus both Linux release builds.

Visual evidence

Manual, Delivery — 1440×900

Before
After

Manual, Delivery — 390×844

Before
After

How it works, Delivery — 1440×900

Before
After

How it works, Delivery — 390×844

Before
After

Review

Fixed point: refs/remotes/origin/issue-1994 (f55519a7105ebd45631dc92bdfca15b3b05a6fd3), three-dot diff. Independent Standards and Spec reviewers raised 1 and 2 findings respectively; all addressed. Reports and counts are retained in .thirdshift-review-ijm3DU/ beside the Session evidence.

Coverage: Spec read all 22 paths preceding the final compiler-required test-adapter match arm in src/run.rs and the cap-boundary privacy fixture in tests/security_run.rs; those final hunks were left unread by both reviewers (Spec had read src/run.rs as supporting context). The exact Security privacy test passed. Standards also left src/command.rs unread after the later Spec fix; other paths were read during its initial review/refactor follow-up. The later preflight fix received the Spec follow-up. Neither axis excluded generated output.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Reversible code and documentation changes; existing Command records remain readable.

Blast Radius: factory

GitHub adapter calls can now wait within the shared cap. Stop remains interruptible; completion operations retain their existing policy. Agent sessions' own gh calls are unaffected.

CI notes

  • test, run 43006727793: unrelated intermittent false positive in unchanged tests/experiments/process.rs:111; the raw JSON check for 421 matched CI’s generated Experiment directory hash 942154523413c43b6c6ce84ec2d5b25318812d888504e51a7750c8f3a6d1c7ad. TMPDIR=<temporary directory containing 421> cargo test --test experiments process::process_session_evidence_check_all_private_fix_identifiers_and_abbreviated_commits -- --exact --nocapture reproduced the same assertion (exit 101) on both issue-1994 (63a64ad4) and issue-1997 (890be3c3); the test, fixture and Experiments execution code are identical between those branches. The ordinary test and all affected checks passed locally.

Built with codex codex-cli 0.167.1 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 marked this pull request as ready for review November 11, 2026 05:20
Preserve Pass admission alongside running Command quota recovery. Keep admission probes from waiting, reconcile quota and Security privacy fixtures with admission, and wait for the asynchronous Inbox refresh in its quota-resume test.
Keep the quota-reset regression synchronized with the observed GitHub state and preserve the Activity log Kind interface alongside named Process quota history.

Validated with cargo check, focused Console and recording tests, cargo fmt --check, all 190 Console browser tests, and scripts/affected-tests origin/issue-1994 (site and full Rust suites).
@JacobStephens2
JacobStephens2 merged commit e712d48 into issue-1994 Nov 11, 2026
19 of 20 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1997 branch November 11, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant