Wait out GitHub quota exhaustion in running Commands

GLOSSARY.md

@@ -298,7 +298,7 @@ The token counts and cost reported in one **Session trace**, grouped by observed
 A **Command**'s whole tree of work, its child **Ticket runs** and all of their **Session traces** included. Their shared identity keeps the tree together across sessions.
 
 **Command log**:
-Everything one **Ticket run**, **Spec run**, **Architect run**, **Pickup run**, **Security run** or **Custom process** printed, kept in a file of its own with its repository's other logs, stamped like its **Session traces**. The start stamp is for display and grouping; Commands with the same start time still have distinct storage ownership for their Command logs, Session traces and **Command records**. An Architect run's, Pickup run's, Security run's or Custom process's covers the Spec runs or Ticket runs it dispatched, and a Spec run's covers its **Ticket runs**, as a Ticket run's covers its **Base fix**: none of those keeps one of its own. It is kept once the command starts work: a Pickup run, Architect run or Security run skipped before doing any work keeps none, and neither does a Ticket run that fails before it starts work, such as on **Origin match**.
+Everything one **Ticket run**, **Spec run**, **Architect run**, **Pickup run**, **Security run** or **Custom process** printed, kept in a file of its own with its repository's other logs, stamped like its **Session traces**. The start stamp is for display and grouping; Commands with the same start time still have distinct storage ownership for their Command logs, Session traces and **Command records**. An Architect run's, Pickup run's, Security run's or Custom process's covers the Spec runs or Ticket runs it dispatched, and a Spec run's covers its **Ticket runs**, as a Ticket run's covers its **Base fix**: none of those keeps one of its own. It is kept once the command starts work, or earlier when a **Quota wait** needs a visible, stoppable Command record. Without such a wait, a Pickup run, Architect run or Security run skipped before doing any work keeps none, and neither does a Ticket run that fails before it starts work, such as on **Origin match**.
 _Avoid_: cron log, output log, run log (a **Ticket run** is one kind of command)
 
 **Log inventory**:

docs/guide.md

@@ -45,6 +45,8 @@ From a clone of the issue's repository, `thirdshift <Issue URL>`:
 7. In a **Ticket run** that Self-merges (`thirdshift merge <Issue URL>`), does the **Self-merge**: once the pull request is open, ready for review, mergeable and green, thirdshift merges it into the Base branch with a merge commit, on exactly the head commit whose CI it watched (`gh pr merge --merge --match-head-commit <sha>`). Just before each attempt, it reads the PR's `reviewDecision`: `REVIEW_REQUIRED` or `CHANGES_REQUESTED` ends the Ticket run as **review required**, a policy refusal, without attempting the merge or taking another Repair round. An empty or `APPROVED` decision, or a failed read, keeps the existing merge behavior. It never uses GitHub's auto-merge ([ADR-0004](adr/0004-self-merge-by-thirdshift-not-github-auto-merge.md)). A merge that fails goes back round step 6, within the same budgets, and is tried again on the new green head. If that round finds nothing to fix, the refusal is a **policy refusal**, such as merge commits being disallowed or a review being required. After the merge, thirdshift deletes the Issue branch on `origin`, and closes the issue if it is still open, with the comment `Closed by #<pr>, merged into <base> by a thirdshift Ticket run with Self-merge.` GitHub closes it on its own only for a merge into the repository's default branch, and then thirdshift leaves it alone. Once the issue is closed, either way, thirdshift takes `in-progress` off it, [removing the Claim](#when-the-claim-ends).
 8. Cleans up: removes the worktree, the local Issue branch and the temporary directory the Factory skills were written out to, whether the Ticket run succeeded or not. The one exception is a **Failed ticket run** whose work could not be pushed: see below.
 
+CI watch quota exhaustion follows the [Command's Quota wait policy](#delivery-and-pull-request-identity): reset plus one minute and 0–60 seconds of jitter, within a shared two-hour cap. Secondary limits retry after 60 seconds, at most five times. Neither is a CI failure or a reason to start a Repair while retries remain.
+
 ### Worktree ownership and cleanup
 
 A factory-owned checkout, on an **Issue branch** for a **Ticket run** or detached for an **Architecture review**.
@@ -63,6 +65,10 @@ See [ADR-0014](adr/0014-successful-review-acquisition-authorizes-stale-disposal.
 
 What a **Ticket run**, or a **Spec run** for its **Spec PR**, does from its worktree once its work begins. It runs the opening session (the implement session, or the **Spec review**), pushes, marks the pull request ready, keeps it mergeable and green through the **Repair loop**, and, in a **Ticket run** that Self-merges, does the **Self-merge**.
 
+When thirdshift's own GitHub reads or writes hit primary quota exhaustion, the running **Command** records a **Quota wait**, waits until reset plus one minute and a random 0–60 seconds, then retries the same call. This includes CI watch and Delivery's pull request updates, labels, comments and Self-merge, across Ticket, Spec, Review, Fix, Security and Triage runs, Custom processes and work a Pass dispatched. The Command and its child Ticket runs share a two-hour total wait cap; a call that would exceed it fails as before. A secondary-limit failure instead waits 60 seconds and retries up to five times, without recording a Quota wait.
+
+While waiting, `status`, `show` and the Console show `Quota wait until <time>` as the stage; afterwards the previous stage resumes. Slow timing includes the wait. Stop or another interruption ends an ordinary wait immediately. Claim ending, Failed ticket run salvage, confirmed Self-merge completion and Spec accounting still wait within the cap after interruption. A Quota wait itself is no failure and adds no `agent-failed` label. Agent sessions' own `gh` calls keep their existing behavior.
+
 For a Spec PR, Delivery ends Ticket accounting's capture before the Spec review, allowing that session to replace the PR. The first successful observation after the opening session, or during Failed ticket run completion after an opening failure, captures one pull request number and URL in the issue's repository; all later observations and transitions use that identity. Readiness and every mergeability poll validate its expected **Issue branch** and **Base branch**, open state and readiness where required. A missing, closed or retargeted captured PR fails Delivery; a replacement from the same branch is never substituted.
 
 A Delivery that fails takes the **Failed ticket run** path, drafting that same captured open PR after preserving work. Spec PR checklist restoration uses that Delivery capture before readiness and for final accounting after failure.
@@ -2861,6 +2867,8 @@ Setup validates the Credentials before asking questions on a terminal. For email
 
 #### Run notifications
 
+A Command that waited for primary GitHub quota includes one line, `Waited <duration> for the GitHub quota (<resource>)`, totaling its waits and its child Ticket runs' waits. A Command that never waited has no such line; secondary-limit retries add none.
+
 `--email` (or `email`) asks a Ticket run for a **Run notification**: one email, sent when the Ticket run ends, whatever the outcome. The word after the flag is the address only if it contains `@` and doesn't start with `https://`, so the Issue URL is never taken for it; otherwise the email goes to the matching `email.to_by_repo` entry, falling back to `email.to`. With `email.always = true` in the [User config](#user-config), a Ticket run asks for one without the flag, and `--no-email` (or `no-email`) skips it for that Ticket run; an address after `--email` still wins over both configured recipients. Giving a flag twice, or any two of `--email`, `--email-unless-merged` and `--no-email`, is an argument error.
 
 `--email-unless-merged` (or `email-unless-merged`), optionally followed by an address as `--email` is, asks for a Run notification that is emailed only when the command did not end `merged`, whatever else it ended as. A merged one is still saved beside the [Command log](#logs), with nothing sent, unless it is a [Security run](#security-runs) whose fix left its finding open or whose audit left findings waiting for the Day shift: that one is emailed. It makes the same checks as `--email` before any work.

site/docs/index.html

@@ -102,6 +102,7 @@
 <li><p>Cleans up: removes the worktree, the local Issue branch and the temporary directory the Factory skills were written out to, whether the Ticket run succeeded or not. The one exception is a <strong>Failed ticket run</strong> whose work could not be pushed: see below.</p>
 </li>
 </ol>
+<p>CI watch quota exhaustion follows the <a href="#delivery-and-pull-request-identity">Command&#39;s Quota wait policy</a>: reset plus one minute and 0–60 seconds of jitter, within a shared two-hour cap. Secondary limits retry after 60 seconds, at most five times. Neither is a CI failure or a reason to start a Repair while retries remain.</p>
 <details class="fold"><summary><h3>Worktree ownership and cleanup</h3></summary>
 <div class="manual-content" id="worktree-ownership-and-cleanup">
 <p>A factory-owned checkout, on an <strong>Issue branch</strong> for a <strong>Ticket run</strong> or detached for an <strong>Architecture review</strong>.</p>
@@ -114,6 +115,8 @@
 <details class="fold"><summary><h3>Delivery and pull request identity</h3></summary>
 <div class="manual-content" id="delivery-and-pull-request-identity">
 <p>What a <strong>Ticket run</strong>, or a <strong>Spec run</strong> for its <strong>Spec PR</strong>, does from its worktree once its work begins. It runs the opening session (the implement session, or the <strong>Spec review</strong>), pushes, marks the pull request ready, keeps it mergeable and green through the <strong>Repair loop</strong>, and, in a <strong>Ticket run</strong> that Self-merges, does the <strong>Self-merge</strong>.</p>
+<p>When thirdshift&#39;s own GitHub reads or writes hit primary quota exhaustion, the running <strong>Command</strong> records a <strong>Quota wait</strong>, waits until reset plus one minute and a random 0–60 seconds, then retries the same call. This includes CI watch and Delivery&#39;s pull request updates, labels, comments and Self-merge, across Ticket, Spec, Review, Fix, Security and Triage runs, Custom processes and work a Pass dispatched. The Command and its child Ticket runs share a two-hour total wait cap; a call that would exceed it fails as before. A secondary-limit failure instead waits 60 seconds and retries up to five times, without recording a Quota wait.</p>
+<p>While waiting, <code>status</code>, <code>show</code> and the Console show <code>Quota wait until &lt;time&gt;</code> as the stage; afterwards the previous stage resumes. Slow timing includes the wait. Stop or another interruption ends an ordinary wait immediately. Claim ending, Failed ticket run salvage, confirmed Self-merge completion and Spec accounting still wait within the cap after interruption. A Quota wait itself is no failure and adds no <code>agent-failed</code> label. Agent sessions&#39; own <code>gh</code> calls keep their existing behavior.</p>
 <p>For a Spec PR, Delivery ends Ticket accounting&#39;s capture before the Spec review, allowing that session to replace the PR. The first successful observation after the opening session, or during Failed ticket run completion after an opening failure, captures one pull request number and URL in the issue&#39;s repository; all later observations and transitions use that identity. Readiness and every mergeability poll validate its expected <strong>Issue branch</strong> and <strong>Base branch</strong>, open state and readiness where required. A missing, closed or retargeted captured PR fails Delivery; a replacement from the same branch is never substituted.</p>
 <p>A Delivery that fails takes the <strong>Failed ticket run</strong> path, drafting that same captured open PR after preserving work. Spec PR checklist restoration uses that Delivery capture before readiness and for final accounting after failure.</p>
 <p>If a Delivery gate observes that its pull request has already merged into the expected Base branch, and its merged head includes all local work (the exact local head, with no uncommitted changes), the Ticket run ends as merged, even when it originally asked only for ready for review. Delivery stops further publication, PR edits and Self-merge work. A merged PR that leaves local work out still fails and retains that work; a merged state alone never establishes completion.</p>
@@ -2824,6 +2827,7 @@
 <p>Setup validates the Credentials before asking questions on a terminal. For email, <code>email-test</code>, the Console and a Run or a Pass that asks for a notification read them only when <code>RESEND_API_KEY</code> is unset or empty. A missing file just means no key from it. The file accepts quoted <code>resend.key</code>, <code>claude.setup_token</code> and <code>opencode.api_key</code> values, and an <code>[opencode.keys]</code> table of quoted keys by OpenCode provider name. Invalid TOML, unknown sections or keys (such as <code>kye</code>), and values of other types stop the command with exit <code>1</code>, naming the file and the offending key, and nothing is sent. An OpenCode Trial&#39;s provider key goes there by hand until <code>setup</code> asks for it: one per provider under <code>[opencode.keys]</code>, such as <code>fireworks-ai = &quot;…&quot;</code> and <code>baseten = &quot;…&quot;</code>, or a single <code>api_key</code> under <code>[opencode]</code> for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with <code>GROK_HOME=~/.thirdshift/trials/grok grok login</code>; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a <code>warning:</code> line on stderr saying to <code>chmod 600</code> it. Nothing is sent to check the key itself. When Resend accepts the email, it prints <code>accepted by Resend; check your inbox</code> and exits <code>0</code>; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won&#39;t send from, it prints Resend&#39;s error text word for word, with where the key came from, and exits <code>1</code>. It gives up after 30 seconds without an answer.</p>
 <details class="fold"><summary><h4>Run notifications</h4></summary>
 <div class="manual-content" id="run-notifications">
+<p>A Command that waited for primary GitHub quota includes one line, <code>Waited &lt;duration&gt; for the GitHub quota (&lt;resource&gt;)</code>, totaling its waits and its child Ticket runs&#39; waits. A Command that never waited has no such line; secondary-limit retries add none.</p>
 <p><code>--email</code> (or <code>email</code>) asks a Ticket run for a <strong>Run notification</strong>: one email, sent when the Ticket run ends, whatever the outcome. The word after the flag is the address only if it contains <code>@</code> and doesn&#39;t start with <code>https://</code>, so the Issue URL is never taken for it; otherwise the email goes to the matching <code>email.to_by_repo</code> entry, falling back to <code>email.to</code>. With <code>email.always = true</code> in the <a href="#user-config">User config</a>, a Ticket run asks for one without the flag, and <code>--no-email</code> (or <code>no-email</code>) skips it for that Ticket run; an address after <code>--email</code> still wins over both configured recipients. Giving a flag twice, or any two of <code>--email</code>, <code>--email-unless-merged</code> and <code>--no-email</code>, is an argument error.</p>
 <p><code>--email-unless-merged</code> (or <code>email-unless-merged</code>), optionally followed by an address as <code>--email</code> is, asks for a Run notification that is emailed only when the command did not end <code>merged</code>, whatever else it ended as. A merged one is still saved beside the <a href="#logs">Command log</a>, with nothing sent, unless it is a <a href="#security-runs">Security run</a> whose fix left its finding open or whose audit left findings waiting for the Day shift: that one is emailed. It makes the same checks as <code>--email</code> before any work.</p>
 <p>A Ticket run that asks for a notification, by either flag or by <code>email.always</code> or <code>email.unless_merged</code>, makes the same checks as <code>email-test</code> before any other work: an address is known, and a key is found, in <code>RESEND_API_KEY</code> or else the Credentials. If either fails, or the Credentials are broken, the Ticket run stops, exits <code>1</code> naming what&#39;s wrong (with no key, the message above listing every way to give one), and sends nothing. A Ticket run that asks for no notification never reads the Credentials, so a broken file can&#39;t stop it, and <code>update</code>, <code>version</code> and <code>help</code> never read it either. Once they pass, every way the Ticket run ends sends exactly one notification, after its outcome is final and its cleanup done: ready for review, merged, a <a href="#failed-ticket-runs">Failed ticket run</a> (including a later preflight failure such as an origin mismatch), or interrupted by Ctrl-C, SIGTERM or a closed terminal.</p>

site/how-it-works/index.html

@@ -124,6 +124,7 @@ <h1>How thirdshift works</h1>
     <details class="wrap guide-section fold">
       <summary class="sec-head"><p class="eyebrow">04 / From a published change to a checked result</p><h2 id="delivery-title">Deliver against the latest Base branch</h2></summary>
       <div class="guide-content" id="delivery">
+      <p>A running Command whose own GitHub call hits primary quota exhaustion waits until reset plus one minute and a random 0–60 seconds, then retries. CI watch and Delivery share a two-hour total cap with child Ticket runs. A secondary limit gets up to five retries, 60 seconds apart. During a primary wait, <code>status</code>, <code>show</code> and the Console show <code>Quota wait until &lt;time&gt;</code>; Slow timing includes the wait. Stop interrupts ordinary waits; Claim ending and other completion work still finish within the cap. The Run notification totals primary waits in one <code>Waited &lt;duration&gt; for the GitHub quota (&lt;resource&gt;)</code> line. Waiting itself adds no <code>agent-failed</code>; agent sessions' own <code>gh</code> calls are unchanged.</p>
       <p>A pull request existing is not the finish line. <strong>Delivery</strong> runs the opening session, then keeps the captured pull request mergeable and checks CI. This orchestration is thirdshift's code; a <strong>Repair</strong> is another agent session started when the branch needs work.</p>
       <p>For an existing open pull request, <code>thirdshift fix &lt;PR URL or number&gt;</code> starts a <a href="/docs/#fix-runs">Fix run</a> directly at Delivery. Ready and draft PRs by Trusted accounts are accepted, with or without closing issues. It uses the PR's own Base branch, reserves the PR while running and gives each Repair trusted PR and linked-issue context. The usual merge, Base fix, Harness and email defaults apply; failure preserves work and offers <code>thirdshift fix</code> again.</p>
       <p>With no PR target, <code>thirdshift fix [base &lt;branch&gt;] [--author &lt;login&gt;]</code> is a Pass. Without <code>--author</code>, it first checks the Base branch's origin tip: a failed first push attempt gets one failed-jobs re-run and a skip while it runs. Persistent or non-rerunnable failures get a CI-fix session on <code>fix-&lt;base&gt;-&lt;short-sha&gt;</code> and a repair PR targeting the Base branch, treating every red check as its own. Existing PRs from that branch prevent duplicates, failed drafts included. Merge and email policy apply, and Self-merge closes the Base branch's CI-red reports and Base fix issues.</p>