Skip to content

fix(ci): retry pnpm setup so a slow npm registry stops ejecting PRs (BLO-28813) - #1410

Merged
kkroo merged 2 commits into
masterfrom
omar/pnpm-setup-retry
Aug 19, 2026
Merged

fix(ci): retry pnpm setup so a slow npm registry stops ejecting PRs (BLO-28813)#1410
kkroo merged 2 commits into
masterfrom
omar/pnpm-setup-retry

Conversation

@kkroo

@kkroo kkroo commented Aug 19, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • Every CI job here installs a pinned pnpm before it can run a single test, via pnpm/action-setup
  • @v6 cannot install a pinned version in one step: it npm cis a bootstrap pnpm pinned inside the action (v11.19.0), then unconditionally runs pnpm self-update 9.15.4
  • That self-update resolves @pnpm/exe@9.15.4 plus every per-platform optional dep against registry.npmjs.org and verifies signatures fail-closed — a Linux runner fetches @pnpm/win-arm64 to do it
  • So a slow registry kills jobs at step 2. On 2026-08-19 that killed three in sixteen minutes; one was a merge-queue candidate, and it was ejected
  • @v4 did not behave this way (pnpm install pnpm@<target> --no-lockfile — one call, no verifier), so this is a v4→v6 regression, not a new registry problem
  • This pull request routes the 18 v6 call sites through a local composite that retries once with jittered backoff and widens the fetch deadline
  • The benefit is that a registry blip costs ninety seconds instead of ejecting a PR from the merge queue

Linked Issues or Issue Description

Path (B) — no GitHub issue; tracked on the Paperclip board as BLO-28813, and the problem is described in full above and below.

Symptom. Failed step is Setup pnpm; everything downstream is skipped, so the job reports failure having run zero tests.

Switching pnpm from v11.19.0 to v9.15.4...
[WARN] GET https://registry.npmjs.org/@pnpm%2Flinux-arm64 error (23). Will retry in 10 seconds. 2 retries left.
[ERR_PNPM_PNPM_ENGINE_IDENTITY_UNVERIFIABLE] Refusing to run pnpm@9.15.4: its npm registry
signature could not be verified (@pnpm/exe@9.15.4: The operation was aborted due to timeout).
time (UTC) run job
00:18:06 32200464275 k8s-ro seed transport cold start
00:22:45 32200730405 policy
00:34:39 32201423561 General tests (server 4/4)ejected #1404 from the merge queue

Blast radius. 21 call sites; 18 on v6 (pr.yml ×8, release-verify ×4, release ×3, plus 3 singles). pr.yml fans out one per shard, so a PR run makes dozens of simultaneous chances to hit it.

Not a duplicate. Searched all open+closed PRs for pnpm / action-setup / setup-pnpm: the nearest are #1300 (postinstall symlink race), #1273 (override drift), and lockfile refreshes — different failure modes. No ROADMAP overlap; this is not core feature work.

What Changed

  • New .github/actions/setup-pnpm composite: attempt → jittered backoff → retry, with the retry deliberately not tolerated so a second failure is the job's failure.
  • 18 v6 call sites routed through it. The three v4 sites are left alone on purpose — v4 never had this failure mode, so migrating them would hand them an exposure they don't have.
  • npm_config_fetch_timeout raised to 120s. Not arbitrary: one failure logged Request took 70506ms — the request completed, then was aborted under pnpm's 60s default.
  • No version: input; the action reads packageManager instead, collapsing 21 duplicated pins to one (it hard-errors on disagreement, so there is no silent path).
  • policy job timeout 5 → 10, because the wrapper's worst case is ~5m and at 5 a degraded registry became a job timeout — same ejection, more runner burned.
  • scripts/__tests__/pnpm-setup-retry.test.mjs (7 tests) pinning all of the above, wired into the policy job.
  • pnpm-setup-retry-proof.yml + flaky-registry-proxy.mjs: proves the retry actually fires, since normal CI never takes that branch.

Verification

The retry branch is proven, not assumed — run 32211415212, both jobs green:

# gating_actually_evaluates (unreachable registry, no proxy, no timing)
pnpm setup failed; retrying once in 19s (BLO-28813)
composite outcome=failure elapsed=38s

# recovers_on_the_second_attempt (registry rejects first burst, then redirects)
registry saw rejected=3 served=3
pnpm on PATH: 9.15.4 (pinned 9.15.4)

Only the retry step can fail the composite (attempt one carries continue-on-error), so outcome=failure proves the retry ran rather than being silently skipped — which matters because steps.*.outcome inside a composite is undocumented.

Guard tests mutation-verified. Each of these fails the specific test claiming to catch it, checked against a green tree every time: reintroduce a bare @v6 site; SHA-pin a direct call; bump a v4 site to v6; delete timeout-minutes; lower it to 5; duplicate the version pin; tolerate the retry too; restore the 60s timeout; re-enable in-process retries; set pmOnFail in the action or in a workflow.

node --test ./scripts/__tests__/pnpm-setup-retry.test.mjs → 7/7. Sibling CI guards (ci-cache-routing, check-github-runner-labels, merge-group-concurrency) still pass.

Risks

  • Low for the happy path. Healthy runs are unchanged (~15–30s); the retry only costs time when something is already broken.
  • A sustained outage still fails, by design. One retry survives a blip, not a 16-minute brownout. Deliberate — see below.
  • pmOnFail=ignore was NOT taken, though the error text suggests it. It doesn't skip the fetch, it skips the version switch, leaving the job running pnpm 11 against a pnpm 9 lockfile in a repo that pins packageManager, blocks lockfile edits in PRs, and monitors drift. A test blocks anyone adding it later under outage pressure.
  • Known trade, stated: npm_config_fetch_retries: "0" is npm's namespace, so it also disables retries for the bootstrap npm ci, not just the self-update. Each stage trades 3 back-to-back attempts on one connection for 2 on fresh processes 10–25s apart with double the deadline. The knob cannot separate the stages.
  • Open, not papered over: pnpm config get fetch-timeout proves the value reaches config, not that the engine-identity verifier honours it over its own deadline. That needs a separate measurement (raised by review).
  • actionlint does not validate uses: ./... paths — verified by pointing a site at a nonexistent action and getting silence. Its clean report is not evidence here; the PR's own CI is, since pr.yml carries 8 of the sites.

Model Used

Claude Opus 5 — exact model ID claude-opus-5[1m], 1M context window, extended thinking enabled, agentic tool use via Claude Code (shell, GitHub API, file edits). Review findings incorporated from allyblockcast (pr-review-toolkit + gstack/review + native-codex lenses).

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots — N/A, CI only
  • I have updated relevant documentation to reflect my changes — rationale documented inline in action.yml; no docs file covers pnpm setup
  • I have considered and documented any risks above
  • All Paperclip CI gates are green — pending; this box is what the run is deciding
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not yet reported
  • I will address all Greptile and reviewer comments before requesting merge

…BLO-28813)

pnpm/action-setup@v6 cannot install a pinned pnpm in one step. It `npm ci`s a
bootstrap pnpm pinned inside the action itself (v11.19.0), then unconditionally
runs `pnpm self-update 9.15.4`. Because the action's bootstrap pin can never
equal our packageManager pin, that self-update always runs, and pnpm 11's
engine-identity check resolves `@pnpm/exe@9.15.4` plus every one of its
per-platform optional dependencies against registry.npmjs.org and verifies the
signatures fail-closed. A Linux runner fetches `@pnpm/win-arm64` to do it.

So every job depended on a healthy registry before it could run a single test.
On 2026-08-19 that killed three jobs in sixteen minutes with
ERR_PNPM_PNPM_ENGINE_IDENTITY_UNVERIFIABLE, one of them a merge-queue candidate
that got ejected as a result. v4 did not behave this way -- it ran
`pnpm install pnpm@<target> --no-lockfile`, one call, no verifier -- so this is
a regression the v6 upgrade brought in, not a new registry problem.

Route the 18 v6 call sites through .github/actions/setup-pnpm, which retries
once with jittered backoff and widens npm_config_fetch_timeout to 120s. 120s is
not arbitrary: one of the failures logged a request that completed in 70,506ms
and was aborted anyway under pnpm's 60s default. pnpm's own in-process retries
are disabled there so the two retry layers cannot multiply into a worst case
that outruns a job budget -- and because the policy job's budget was 5 minutes,
below that worst case, it moves to 10.

Deliberately NOT setting pmOnFail=ignore, which the error text suggests. That
does not skip the fetch, it skips the version switch, leaving the job running
pnpm 11 against a pnpm 9 lockfile. Failing closed is correct; the missing piece
was tolerance around it.

The three v4 sites are left alone on purpose. v4 never had this failure mode,
so migrating them would hand them an exposure they do not currently have.

Honest limit: one retry survives a blip, not a sustained outage. It turns the
common case from "PR ejected from the merge queue" into "this job took another
ninety seconds".

scripts/__tests__/pnpm-setup-retry.test.mjs pins all of the above. Every
assertion was mutation-tested: reintroducing a bare @v6 site, dropping the
policy timeout back to 5, removing the checkout the wrapper depends on,
duplicating the version pin, tolerating the retry too, restoring the 60s
timeout, re-enabling in-process retries, and setting pmOnFail in either the
action or a workflow each fail the specific test that claims to catch it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 19, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-28813

1 similar comment
@allyblockcast

allyblockcast Bot commented Aug 19, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-28813

@allyblockcast

allyblockcast Bot commented Aug 19, 2026

Copy link
Copy Markdown

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

  • Missing section: ## Thinking Path
  • Missing section: ## What Changed
  • Missing section: ## Risks
  • Missing section: ## Model Used
  • Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 721e313

Strong diagnosis and an unusually honest write-up. Two things worth resolving before this lands, both about the fix's own blast radius rather than the diagnosis.

I verified the mechanism rather than taking the description on trust: pnpm/action-setup@v6 src/install-pnpm/run.ts does await rm(dest, {recursive:true, force:true}) on entry (the "retry starts clean" claim holds), runCommand(cmd, args, {cwd: dest}) for self-update passes no env so it inherits process.env (the timeout knob does reach the failing call), and continue-on-error is a documented runs.steps[*] key for composite actions. I also ran node --test ./scripts/__tests__/pnpm-setup-retry.test.mjs against the files at this head: 7/7 pass. policy and Worktree install are already green on this head, so the local-action path resolves.

Critical Issues (0)

Important Issues (2)

  • [native-codex] .github/actions/setup-pnpm/action.yml:93npm_config_fetch_retries: "0" silently disables npm's retries for the bootstrap npm ci, not just pnpm's.

    • The comment reasons only about pnpm ("pnpm's in-process retries are off on purpose... 2 in-process x 2 step attempts = 6 registry stalls"), but this is a plain npm_config_* variable and run.ts builds the bootstrap environment as {...process.env, [pathKey]: ...} — so npm ci, which also pulls @pnpm/exe from registry.npmjs.org, inherits it and drops from fetch-retries: 2 (3 attempts) to 1. Across the wrapper that is 2 total bootstrap attempts where master has 3.
    • The diagnosed failures were all at the self-update/engine-identity stage, so this trades away resilience at a stage that was not failing. It is probably close to a wash once the 120s timeout is counted, but it is not currently a considered trade — it is a side effect.
    • The two layers are not separable via this knob, so the cheap fix is to keep the value and correct the comment to say both npm and pnpm retries are disabled and why that is acceptable. As written the comment will mislead whoever next tunes this.
  • [pr-review-toolkit: tests] scripts/__tests__/pnpm-setup-retry.test.mjs:1 — the retry branch has no execution coverage anywhere, and the suite reads as though it does.

    • test("the wrapper retries once, and the retry is allowed to fail the job") asserts only that the YAML text contains id: first, if: steps.first.outcome == 'failure', and RANDOM. It cannot detect the failure mode that matters: if steps.<id>.outcome does not evaluate as expected for a uses: step inside a composite action, the retry never fires and the whole mitigation is inert. GitHub documents continue-on-error as a composite runs.steps[*] key but does not document steps.*.outcome availability inside composite actions — the contexts reference only calls out secrets as unavailable — so this rests on convention rather than a documented guarantee.
    • CI on this head cannot close the gap either: the first attempt succeeds, so the branch is never taken. The failure would surface for the first time during the next registry brownout, precisely when you are relying on it.
    • Cheap way to close it: one workflow_dispatch scratch job that forces attempt one to fail (for example npm_config_registry: http://127.0.0.1:1) and asserts the pnpm setup failed; retrying once line appears and the job still ends green. Running it once by hand and linking the run in the PR would be enough; it does not need to stay in the matrix.

Suggestions (3)

  • [gstack/review] scripts/__tests__/pnpm-setup-retry.test.mjs:94 — the if (!declared) continue; escape hatch rewards deleting a timeout over declaring a tight one. A future 3-minute job that adopts the wrapper either inflates to 10 or drops timeout-minutes and silently inherits GitHub's 360m default, which is strictly worse merge-queue protection than the 5 it had — and the comment tells the author that is fine. Consider asserting that a wrapper job declares some budget, with the 10m floor applying above a threshold.
  • [gstack/review] scripts/__tests__/pnpm-setup-retry.test.mjs:74matchAll(/uses: pnpm\/action-setup@v(\d+)/g) only catches @vN tags. A SHA-pinned uses: pnpm/action-setup@<sha>, the direction repos usually drift toward for supply-chain reasons, bypasses the guard entirely. Matching pnpm/action-setup@ and allow-listing the three v4 sites by path would fail closed instead.
  • [native-codex] .github/actions/setup-pnpm/action.yml:86 — the evidence for 120s proves config plumbing, not that the failing code path honours it. pnpm config get fetch-timeout returning 120000 shows the value is read into config; it does not show the engine-identity verifier's @pnpm/exe resolution uses fetch-timeout rather than its own deadline. Worth one direct check — drop it to something absurdly low and confirm the identity check fails faster — otherwise the retry may be doing all the work here.

Strengths

  • Root cause is traced to the actual mechanism — the unconditional self-update and the fail-closed per-platform identity check — not to "the registry was flaky", and the v4 to v6 regression framing is correct.
  • The pmOnFail=ignore refusal is right, and encoding it as a test so nobody adds it later under outage pressure is the best part of this diff.
  • Dropping version: in favour of packageManager collapses 21 pins to one and removes a real drift surface; readTargetVersion throws on disagreement, so there is no silent path.
  • Jitter with a stated reason (eight simultaneous pr.yml jobs), the checkout-ordering test that guards the new implicit dependency on package.json, and the deliberate decision to leave the three v4 sites alone are all well argued.
  • "Honest limits" and the note that actionlint does not validate uses: ./... are exactly the disclosures that make a CI change reviewable.

Recommended Action

  1. No Critical issues — nothing blocks on correctness of the happy path.
  2. Address the two Important items this cycle: correct or re-scope the fetch-retries knob, and exercise the retry branch once so the mitigation is known to fire.
  3. Consider the Suggestions opportunistically; the SHA-pin gap in the guard is the one most likely to bite later.

…(BLO-28813)

Addresses Ally's two Important findings at head 721e313, plus the two guard
gaps she flagged as most likely to bite later.

1. `npm_config_fetch_retries: "0"` was justified in the comment as if it only
   affected pnpm. It does not — `npm_config_*` is npm's namespace and run.ts
   spawns the bootstrap `npm ci` with `{...process.env}`, so it disables npm's
   retries too, at a stage that was not failing. The value stays; the reasoning
   was the defect. The comment now states both stages are affected and shows
   the arithmetic: each stage trades 3 back-to-back attempts on one connection
   for 2 attempts on fresh processes 10-25s apart with double the deadline.
   Fewer raw attempts, better distributed, and bounded — leaving both layers on
   would be up to 6 registry stalls, which does not fit the 10-minute floor the
   guard test asserts. Also noted that this knob cannot separate the two stages,
   so they move together.

2. The retry branch had no execution coverage, and the suite read as though it
   did. GitHub documents `continue-on-error` as a composite `runs.steps[*]` key
   but does NOT document `steps.*.outcome` availability inside a composite. If
   it does not evaluate there, the `if:` is false, the retry is skipped, the
   first failure stays swallowed by `continue-on-error`, and the composite exits
   0 WITH NO PNPM INSTALLED. Inert mitigation, silent until a brownout.

   `pnpm-setup-retry-proof.yml` closes it. The decisive job needs no proxy and
   no timing: run the wrapper against an unreachable registry, and assert the
   composite's outcome is `failure`. Only the retry step can fail this composite
   — the first attempt carries continue-on-error — so `failure` proves the retry
   ran, and anything else proves it was skipped. A second job drives a local
   registry that rejects the first burst and then redirects upstream, asserting
   attempt one was actually rejected (an unrejected run is reported as
   inconclusive, not a pass) and the pinned pnpm still lands.

   It runs on pushes touching the wrapper, not workflow_dispatch alone:
   dispatch requires the file on the default branch, so a dispatch-only proof is
   un-runnable on the PR that introduces it, and path-scoping re-proves it
   whenever the wrapper is edited instead of relying on memory.

3. The direct-call guard matched `@v(\d+)`, so a SHA pin — the direction repos
   drift for supply-chain reasons — bypassed it entirely and got reported as
   "no direct calls". Now matches `pnpm/action-setup@<anything>` and allow-lists
   exact (file, ref) pairs, so an unlisted file OR a listed file on a different
   ref both fail closed.

4. The timeout guard skipped jobs with no declared budget, which made deleting
   `timeout-minutes` the cheapest way to satisfy the 10m floor — and trading a
   tight budget for GitHub's 360m default is strictly worse merge-queue
   protection. Absence now has to be allow-listed. Only
   `release-penstock-scope.yml:publish` is listed; deliberately NOT given a
   timeout, because it is an approval-gated Rust/sccache publish and picking a
   ceiling for a release job unilaterally is the riskier change.

Not done, and stated rather than quietly skipped: Ally's third Suggestion is
right that `pnpm config get fetch-timeout` proves config plumbing, not that the
engine-identity verifier honours it rather than its own deadline. That needs a
separate measurement.

Guard tests still 7/7, and the four new/changed assertions are mutation-tested:
a SHA-pinned direct call, a v4 site bumped to v6, a deleted `timeout-minutes`,
and a lowered one each fail the specific test that claims to catch them. The
flaky-registry proxy was smoke-tested locally — 503 through the burst window,
302 to the real registry after, redirect resolving to live metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kkroo
kkroo requested a review from allyblockcast August 19, 2026 03:14
@allyblockcast

allyblockcast Bot commented Aug 19, 2026

Copy link
Copy Markdown

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

  • Missing section: ## Thinking Path
  • Missing section: ## What Changed
  • Missing section: ## Risks
  • Missing section: ## Model Used
  • Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

@kkroo

kkroo commented Aug 19, 2026

Copy link
Copy Markdown
Author

@allyblockcast — both Important findings addressed at 505ab07f7, and the second one is now measured rather than argued.

Important 2: the retry branch is no longer unproven

You were right that this was the load-bearing gap: steps.*.outcome availability inside a composite action is undocumented, and if it didn't evaluate, the composite would exit 0 with no pnpm installed — inert, silent until a brownout.

New workflow pnpm-setup-retry-proof.yml, run 32211415212, both jobs green:

gating_actually_evaluates — no proxy, no timing. Run the wrapper against an unreachable registry and assert the composite's outcome. Only the retry step can fail it, since attempt one carries continue-on-error:

pnpm setup failed; retrying once in 19s (BLO-28813)
composite outcome=failure elapsed=38s
Retry gating verified: the composite failed only because attempt two ran and failed.

The backoff line proves steps.first.outcome == 'failure' evaluated; 38s against a connection-refused registry is only reachable by paying the backoff.

recovers_on_the_second_attempt — a local registry that rejects the first burst then redirects upstream:

pnpm setup failed; retrying once in 22s (BLO-28813)
registry saw rejected=3 served=3
pnpm on PATH: 9.15.4 (pinned 9.15.4)

Attempt one was genuinely rejected, attempt two recovered with the pinned version. A run where nothing was rejected is reported as inconclusive, not a pass.

It triggers on pushes touching the wrapper rather than workflow_dispatch alone — dispatch needs the file on the default branch, so a dispatch-only proof is un-runnable on the PR introducing it, and path-scoping re-proves it whenever the wrapper is edited.

Important 1: the comment was the defect, not the value

Correct on all counts — npm_config_* is npm's namespace and run.ts spawns the bootstrap npm ci with {...process.env}, so I disabled retries at a stage that wasn't failing. The value stays (dropping it breaks the 10m floor the guard asserts); the comment now states both stages and shows the arithmetic: each trades 3 back-to-back attempts on one connection for 2 on fresh processes 10–25s apart with double the deadline. Also noted the knob can't separate the stages, so they move together.

Suggestions

  • SHA-pin gap — taken, and it was worse than described: a SHA pin was reported as "no direct calls". Now matches pnpm/action-setup@<any ref> and allow-lists exact (file, ref) pairs, so an unlisted file or a listed file on a different ref fails closed. Mutation-tested both ways.
  • Timeout escape hatch — taken. Absence of timeout-minutes must now be allow-listed, so deleting a tight budget for GitHub's 360m default fails instead of passing. Only release-penstock-scope:publish is listed, and deliberately not given a timeout: picking a ceiling for an approval-gated Rust/sccache publish unilaterally is the riskier change.
  • fetch-timeout evidencenot done, and you're right. pnpm config get fetch-timeout proves the value is read into config, not that the engine-identity verifier honours it rather than its own deadline. Unresolved; it needs its own measurement, and I'd rather say so than imply the 120s is proven.

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 505ab07

Both Important findings are resolved, and the second one is resolved the right way — by measurement, not by argument. Approving.

I did not take the linked run on trust. I pulled the raw logs for run 32211415212 and confirmed it is attached to this exact head with both jobs green, and the runner's own step trace is more decisive than the quoted stdout:

##[end-action id=pnpm.first;outcome=failure;conclusion=success;duration_ms=11027]
##[start-action display=Back off before retrying pnpm setup;id=pnpm.__run]
pnpm setup failed; retrying once in 19s (BLO-28813)
##[start-action display=Setup pnpm (retry);id=pnpm.retry]
##[end-action id=pnpm.retry;outcome=failure;conclusion=failure;duration_ms=5026]

outcome=failure;conclusion=success on first is continue-on-error swallowing the failure, and the two gated steps then start. That is direct evidence that steps.first.outcome evaluates inside a composite action — the undocumented behaviour the whole mitigation rests on. The recovery job shows the mirror image: first outcome=failure, 22s backoff, retry outcome=success, registry saw rejected=3 served=3, pnpm on PATH: 9.15.4 (pinned 9.15.4).

One thing worth saying plainly, because the checks page is actively misleading here: the wall of red on this head is not this PR. Every one of those jobs has conclusion: cancelled, not failure — PR run 32211417209 was superseded by 32212757768 on the same SHA. There are zero failure conclusions in that run, policy / Helm chart / Vendored claude_k8s adapter completed green, and every cancelled job recorded Setup pnpm -> success before it was killed. The wrapper resolves and installs on the real call sites. policy step 11 also ran the guard suite green, so the tightened matchers below are passing, not just written.

Prior Findings Dispositioned (2)

  • prior:721e313 important 1 — fixed — .github/actions/setup-pnpm/action.yml:87 — the comment now names the mechanism rather than reasoning only about pnpm: npm_config_* is npm's namespace, run.ts spawns the bootstrap with {...process.env}, so both stages are affected. The arithmetic is spelled out per stage (3 back-to-back attempts on one connection → 2 attempts on fresh processes 10–25s apart at double the deadline), and line 100 records that the knob cannot separate the stages so they must move together. The value is retained, which is right — dropping it puts the worst case at 6 stalls × 120s = 12m, through the 10m floor the guard asserts.
  • prior:721e313 important 2 — fixed — .github/workflows/pnpm-setup-retry-proof.yml:40gating_actually_evaluates closes it deterministically. Because attempt one carries continue-on-error, the composite can only report failure if the retry step actually ran, so asserting outcome == 'failure' against a connection-refused registry isolates exactly the inert-mitigation failure mode, with no dependence on timing (the elapsed check is correctly demoted to a warning). Verified green at this head, and the push-on-paths trigger is the right call — a workflow_dispatch-only proof is un-runnable on the PR that introduces it.

Critical Issues (0)

Important Issues (0)

Suggestions (2)

  • [native-codex] .github/workflows/pnpm-setup-retry-proof.yml:125served is computed and echoed but never asserted, so job two cannot distinguish "attempt one failed, attempt two recovered" from "attempt one rode out the burst and succeeded by itself". firstRequestAt is set by the very first request, so rejected >= 1 is true by construction; the guard therefore only catches the zero-traffic case (a cached pnpm on an ARC runner), which is worth catching but is not the claim in the job name. BURST_MS is 8s while attempt one's duration is unbounded, so a loaded runner could serve its later requests. Asserting served > 0 alongside it, or a >=10s gap between the last rejected:true and the first rejected:false in the JSONL, would make the end-to-end claim self-verifying. Job one is unaffected and remains the decisive proof — this is corroborating evidence, which is why it is a suggestion.
  • [gstack/review] scripts/__tests__/pnpm-setup-retry.test.mjs:43readdir(...).filter((n) => n.endsWith(".yml")) makes every guard in this file blind to a .yaml workflow, which GitHub accepts identically. A workflow added as .yaml would silently escape the direct-call ban, the 10m floor, and the checkout-ordering check all at once. Latent today (all 25 workflows are .yml, and setup-pnpm is the only composite action), but it is the same fail-open shape as the SHA-pin gap you just closed, and /\.ya?ml$/ costs nothing.

Strengths

  • The response to the retry-coverage finding is the standout: rather than asserting more YAML text, it isolates the one branch that could not be exercised and proves it against a registry that cannot answer. Structuring the assertion so that only the retry step can produce failure is what makes it evidence instead of a smoke test.
  • Reporting a run where nothing was rejected as inconclusive rather than a pass is the detail that keeps this proof honest as it ages.
  • The SHA-pin guard now keys on (file, ref) pairs and fails closed on both axes — an unlisted file or a listed file on a different ref — which is strictly stronger than the version-shaped match I suggested.
  • Making absent timeout-minutes require an explicit allow-list entry removes the perverse incentive exactly; declining to invent a ceiling for the approval-gated Rust/sccache publish, and saying why, is the right call rather than a gap.
  • Dropping version: in favour of packageManager collapses the pins to one source and, for release-verify.yml, means a verified ref is now exercised with the pnpm that ref actually pins.
  • Conceding the fetch-timeout point as unresolved instead of dressing up config plumbing as proof is worth more to the next reader than a third green check would have been.

Recommended Action

  1. No Critical or Important issues — this is good to land once the superseding PR run finishes green.
  2. Both suggestions are guard-hardening, not correctness; take them opportunistically.
  3. The fetch-timeout question stays open by agreement. Worth its own measurement — drop it to something absurdly low and confirm the engine-identity check fails faster — since if the verifier uses its own deadline, the retry is carrying this fix alone.

@kkroo
kkroo added this pull request to the merge queue Aug 19, 2026
kkroo added a commit that referenced this pull request Aug 19, 2026
…s (BLO-28818)

Addresses both Important findings from review of c624ba1.

1. A signal-killed server was misreported as alive. `child.exitCode` is null
   BOTH while a child runs and after it dies from a signal — Node records the
   signal in `child.signalCode` instead — so the `exitCode !== null` guard never
   fired for a SIGKILL. The probe then polled the full 120s and threw a message
   that hard-asserted "(process still alive)".

   That is not incidental to this PR. An OOM-kill of a server co-resident with
   embedded Postgres in a CI shard is a leading candidate for the very stall
   being investigated, and this PR's own reasoning ("a dead child throws a
   different error") used that guard to conclude the observed flake was a stall
   rather than a death. Under a signal kill the old code would not have thrown
   a different error either — so the message stated as fact something the code
   could not establish.

   Verified on this runtime:
     spawn("sleep",["30"]); kill("SIGKILL")
     -> exitCode: null  signalCode: SIGKILL  killed: true

   Both throw paths now report the observed pair, and the loop exits early on a
   signal kill instead of burning the remaining budget.

2. The orienting comment cited BLO-28813 — a real but unrelated issue (pnpm
   registry retries, PR #1410) — so a reader chasing the log-capture rationale
   landed somewhere actively misleading. The tense implied the fix landed
   elsewhere, when this diff IS the change.

Also takes all three suggestions, each mutation-verified:

- The tail budget now counts BYTES. `readFileSync(…, "utf8")` returns a string,
  so the old `slice(-n)` counted UTF-16 code units: the server logs through
  pino-pretty, whose glyphs ("◇ │ ✓ └") are 7 units but 15 bytes, so an "8000
  byte" cap admitted multiples of that and the "last N of M bytes" figure was
  simply wrong. Reads a Buffer and slices bytes.
- A zero budget is a floor, not "unlimited". `slice(-0)` is `slice(0)` — the
  whole string — so asking for nothing returned everything.
- The per-file read-error branch is tested. The previous message claimed this
  was not portably constructible ("running as root, chmod 000 is still
  readable"); that was wrong. A subdirectory inside the log dir throws EISDIR
  from readFileSync on every platform and every uid — confirmed here as uid
  1000 — and a rotated-log subdirectory is a plausible real layout.

Mutation-verified (each reverted after): char-based slice -> caught by "budgets
in BYTES"; clamp removed -> caught by "zero budget as a floor"; per-file guard
removed -> caught by "unreadable entry". Tree green before and after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged via the queue into master with commit b877eba Aug 19, 2026
27 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant