Skip to content

fix(install): give the proxy uv install the same retry window - #907

Open
tend-agent wants to merge 1 commit into
fix/claude-install-retry-31298989023from
fix/proxy-uv-install-retry-31298989023
Open

fix(install): give the proxy uv install the same retry window#907
tend-agent wants to merge 1 commit into
fix/claude-install-retry-31298989023from
fix/proxy-uv-install-retry-31298989023

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

Stacked on #906 — based on its branch, so the diff here is only the install-proxy-uv.sh half. GitHub retargets this to main when #906 merges.

Problem

#906 widens the retry window on install-claude-binary.sh after a CDN 403 burst cost review-reviewers a matrix leg. Reviewing it surfaced that shared/steps/install-proxy-uv.sh carries the identical loop — for i in 1 2 3, sleep $((i * 5)), after 3 attempts — and shares every property the rationale rests on:

  • It runs in the same job (claude/action.yaml), ~80 lines ahead of the claude install, so it is equally ahead of the agent step and equally invisible to Report failure, which is gated on steps.claude.outcome == 'failure'. A run lost here files no tend-outage row.
  • It fetches over the same runner egress, on all five review-reviewers legs at once, so a rate limit hits them together and an unjittered backoff has them retry together.
  • Its three attempts span the same ~15s window the observed blip outlasted.

A 403 from astral.sh instead of claude.ai loses the leg identically. Fixing only the claude half leaves the exposure roughly halved rather than closed.

Change

Same shape as #906: five attempts, exponential backoff, jitter — 5, 10, 20, 40 seconds plus 0–9s each. Happy path unchanged (one fetch, no sleep).

Worth stating the same ceiling caveat: when curl fails fast the added cost to a doomed install is ~95s; when the CDN hangs, each attempt burns the full timeout 60 and the worst case goes from ~195s to ~411s.

Tests

install-proxy-uv.sh had no test coverage at all. Four tests added, reusing the fixture pattern #906 establishes (_fake_bin, FAKE_SLEEP_RECORDING, the attempt/sleep readers) with a curl fake that emits astral's installer shape — read on stdin by sh -s --, so it stays POSIX rather than bash. Reusing those helpers is why this is stacked rather than parallel; the alternative was a second copy of each fake and a conflict at the end of the file.

Three of the four fail against the pre-fix script:

Test Against the base branch
rides out a 403 burst (4 failures then success) fails — gives up after 3
backs off exponentially fails — sleeps [5, 10]
reddens when every attempt fails, naming the count fails — says "after 3 attempts"
installs first try without sleeping passes (happy path is unchanged)

Full generator suite: 387 passed. shellcheck -S warning clean on the modified script; ruff check and ruff format --check clean.

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry loop is now byte-identical in install-claude-binary.sh and install-proxy-uv.sh with no shared source of truth, so the next adjustment to the window — or a bug in the jitter expression — has to be found and applied twice, and the two rationale comments already differ in wording while describing the same policy. The repo has the mechanism for this: shared/steps/lib/run-issue.sh is sourced by both rate-limit-preflight.sh and report-failure.sh via . "${SCRIPT_DIR}/lib/run-issue.sh". Extraction isn't free here, though — the claude copy runs inside sudo -u "$SANDBOX" env … bash <<'EOF' as the sandbox user, so a sourced helper would need the lib path threaded through that env and the file readable by the sandbox UID, which I haven't verified. A follow-up rather than a change to this PR, but worth deciding deliberately: the duplication is what lets the two windows drift apart later.

Nothing else — the loop tracks #906 line for line, shellcheck -S warning is clean on the modified script, and the four new tests fail against the base branch's script in exactly the three places the PR body claims.

@tend-agent

Copy link
Copy Markdown
Collaborator Author

Extracted, in #908, stacked on this branch so the diff there is only the extraction.

The sandbox boundary you flagged as unverified is real, and it rules out the obvious approach: install-claude-binary.sh runs its loop as the sandbox UID, and nothing grants that UID read access to the action's own checkout — setup-sandbox.sh grants traversal (o+x) on the workspace's ancestors only, and the action lives elsewhere under _actions/. So . "${SCRIPT_DIR}/lib/retry.sh" from inside the heredoc would be resting on whatever mode the runner happened to leave on the action checkout.

Concatenating the lib onto the front of the stdin stream sidesteps it — the file is read on the runner side of the privilege drop, and sudo -u/env/the pinned XDG dirs are untouched:

cat "${SCRIPT_DIR}/lib/retry.sh" - <<'EOF' \
  | sudo -u "$SANDBOX" env HOME="$AGENT_HOME" … bash
set -euo pipefail
retry_install "claude $CLAUDE_VERSION" \
  "curl -fsSL https://claude.ai/install.sh | bash -s -- '$CLAUDE_VERSION'"
EOF

One correction to the review: the two loops aren't byte-identical — the fetched command and both echo strings differ, so what's shared is the five lines of window mechanics (ATTEMPTS, the backoff expression, the jitter, the timeout 60, the set -o pipefail inside bash -c). That's still exactly the part that drifts, and the pipefail line is the one whose omission fails silently, so it's worth having once.

The eight tests from #906 and #907 pass unmodified against the lib, which is the useful signal: the window is now asserted from both sides of the sudo boundary. Full suite 387 passed, shellcheck -S warning clean.

Left this PR alone rather than folding the extraction in — but if three PRs for one window is worse than a slightly wider one, squashing #908 into this branch loses nothing.

@tend-agent

Copy link
Copy Markdown
Collaborator Author

First production occurrence of this exact path failing, from a review-reviewers matrix leg an hour ago — plus two details that change how the window should be sized.

Run 31422711935, leg review-reviewers (numbagg/numbagg). The step ran duration_ms=195022 and ended ::error::failed to install uv 0.12.1 after 3 attempts, so the leg was lost before the agent booted and never re-ran. Until now the uv install being "as exposed as the claude one" was an argument by analogy in this PR's test docstring; it is now observed.

The failure shape is a stall, not a fast-403 burst. Every attempt hit the timeout 60 ceiling — 19:10:02Z start, retry lines at 19:11:02Z and 19:12:07Z, error at 19:13:17Z, i.e. 60s + 5s + 60s + 10s + 60s. The claude.ai case this PR is modelled on spent all three attempts inside ~15s, which is what makes "three attempts is too few to cross a blip" the right diagnosis there. Here the window was already 195s and still lost the run. Five attempts at exponential backoff adds roughly 75s of sleep, but the dominant term is the per-attempt timeout, so the window grows to ~6 min mostly by sitting in two more 60s stalls. Worth deciding whether the lever is attempt count or a shorter per-attempt budget (--connect-timeout / curl's own --retry) so more attempts fit the same wall-clock.

Siblings on the same run installed fine seconds either side, which argues against the shared-egress rate-limit rationale given for jitter in #908. review-reviewers (max-sixty/worktrunk) printed uvx 0.12.1 at 19:10:01Z and review-reviewers (PRQL/prql) at 19:10:08Z — both overlapping the numbagg leg's first stalled attempt. Matrix legs get separate runners with separate egress, so this looks per-connection rather than a limit that hits the legs together. Jitter still costs nothing; the reasoning behind it just does not hold up here.

Nothing in the log says why. curl -LsSf is silenced and timeout SIGTERMs it before -S prints anything, so the three attempts produced zero diagnostics — no DNS, TLS, or HTTP status to distinguish a stall from a 403. The entire trace is two uv install attempt N failed; retrying lines and the final error. Whatever window this lands on, capturing the failing attempt's exit status would make the next occurrence classifiable instead of another guess.

Sources
  • Failed leg: run 31422711935, job review-reviewers (numbagg/numbagg), 19:09:53Z→19:13:20Z.
  • Step under test at the ref that ran: shared/steps/install-proxy-uv.sh at 0.1.14.
  • Timings above are read from the job log; sibling uvx 0.12.1 lines are from the same run's other legs.

max-sixty pushed a commit that referenced this pull request Aug 11, 2026
…will convert (#914)

## Problem

Filing an issue in this repo is not a note to a maintainer —
`tend-triage` fires on `issues`. Twice in 66 minutes this morning a run
filed an issue describing a fix it had already fully scoped, and triage
then re-derived that fix from the issue body and opened the PR:

| Issue filed | Triage boots | Turns / cost | PR opened |
|---|---|---|---|
| [#909](#909) 07:18:53Z |
[31300864191](https://github.com/max-sixty/tend/actions/runs/31300864191)
07:18:56Z | 64 / $4.08 |
[#910](#910) `fix/issue-909`
07:25:17Z |
| [#911](#911) 07:39:37Z |
[31301666194](https://github.com/max-sixty/tend/actions/runs/31301666194)
07:39:40Z | 58 / $4.20 |
[#912](#912) `fix/issue-911`
07:49:37Z |

Three seconds from issue to triage boot in both cases. Neither issue was
under-specified: #909 named `codex/action.yaml:120-124` and proposed
extracting to `shared/steps/install-codex-cli.sh` sourcing
`lib/retry.sh`; #911 named `proxy/setup-sandbox.sh:319` and proposed the
same treatment. Both PRs implemented exactly the proposal. Triage's own
session log confirms the conversion — run 31300864191's artifact is
`claude-session-logs-n909`, and it ran `gh pr create --head
fix/issue-909`, classifying the reporter as "the bot's own review of
#908".

So the issue bought nothing and cost $8.28 of agent time plus two
threads. The filing run could have opened both PRs itself.

<details><summary>Wider context — the cascade this sat inside</summary>

A single transient CDN 403 lost one matrix leg of `review-reviewers` run
[31297986524](https://github.com/max-sixty/tend/actions/runs/31297986524)
at 06:03Z (`curl: (22) ... 403`, three attempts inside 15.7 s, the other
four legs installing the same version from the same runner at the same
moment succeeded). Between 06:29Z and 07:56Z that produced #906, #907,
#908, #909, #910, #911 and #912 — five stacked PRs and two issues, none
merged, across ~14 agent runs and roughly $30 at list prices.

The engineering is sound and correctly atomic; this PR does not argue
with any of it. Only the two issue hops are pure overhead, and they are
the part with a mechanical fix.

</details>

## Solution

A short section in `running-in-ci` — the skill every workflow loads —
stating that an issue in this repo is picked up by triage, so a fix you
can already scope should be opened as a PR in the same run. The escape
hatches stay: too large or ambiguous to fix, needs a maintainer
decision, or verification is out of reach from CI.

This generalizes a rule that already exists but only in two skills:
`review-runs` ("Issue (fallback): Only for problems too large or
ambiguous to fix directly") and `review-reviewers` carry it, so the runs
that have it don't do this. The runs that did — a `tend-review` session
and a `tend-mention` session — load neither.

## Gate assessment

- **Evidence level**: High — 2 occurrences this window, and the
self-amplifying-output family has been recorded in prior windows
(2026-08-04: review→fix ping-pong over five rounds; two bot PRs seven
minutes apart on one root cause).
- **Structural**: yes for the costly half. The bot's choice to file
rather than fix is stochastic, but once an issue exists the triage
conversion has no decision point — it fires on the `issues` event every
time.
- **Change type**: targeted fix (one short section), normal evidence
bar.
- **Both gates**: pass.

Found by `/tend-ci-runner:review-runs` on run
[31302531474](https://github.com/max-sixty/tend/actions/runs/31302531474).

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant