Repository navigation
Description
What happens
Since the 0.20.0 roll, every fleet host logs four times an hour:
[WARN] forge_identity: reader app gateway withdrawn from 2amlogic all for 299s (end from default; refusal reported remaining=- reset=-; release.checksum via the gh choke point); its other owners and resources keep serving — W4-A
Every withdrawal on worker-2 from 00Z to 04Z has the same cause, release.checksum. While it lasts, every 2AMLogic core read is shed even though both reader apps report usable budget:
[INFO] gh_invocation: step_engine shed — no reader can serve 2amlogic core until …Z (router: budget — app …: usable, book 5723/15000 …; app …: usable, book 3593/15000 …)
[WARN] work_finder: 2AMLogic/<repo>: step listing failed (read shed: no reader can serve it until …; deferred, not sent to the writer (loom-shed)); using the listing from 337s ago
Read-shed WARNs per host per hour (worker-2): 252 (00Z, roll at ~00:03), then 762, 950, 960, 960. Before the roll they ran 190–390/h. The same shape is on all six hosts (700–1,100/h).
Why it is wrong
- A refusal on one resource (a release-asset checksum read, which reports no
remaining/reset, i.e. not a budget refusal) withdraws the gateway for the entire owner (2amlogic all) for the default 299 s. That is about 20 min/h per host in which no reader serves 2AMLogic. - It probably also caused the startup release-tag verification to time out, which disabled workspace resync fleet-wide (companion issue).
- The work finder falls back to stale listings (5–6 min old), so dispatch keeps going, but everything else that reads is deferred.
What to do
- Scope a refusal without budget headers to the resource or operation that drew it (here
release.checksum), not to the owner's whole core budget. - Find why
release.checksumis refused for the reader apps since 0.20.0 (an asset download redirect, a permission, or the repo move to 2AMLogic) and fix or route that read. - Do not apply the default 299 s to a refusal that carries no
reset; use a short backoff.
Acceptance
- A refused
release.checksumread no longer sheds unrelated 2AMLogic reads. - Read-shed WARNs return to the pre-0.20.0 level or below, measured by the fleet check's DAEMON-LOG section.
- A test: a resource-scoped refusal leaves other operations on the same owner routable.
Related: #14730, #11838, #12245.
Filed from an attended fleet check on 2026-11-20.
Curator findings · 2026-11-20
Traced on main at 21f029145. The resolver, reader route, gateway router and withdrawal planner have identical source at tag v0.20.0 (git diff v0.20.0..21f029145 over the four files below).
fetch_asset_sha256runs the reader-only commandgh release download <tag> -R <repo> -p <binary>.sha256 -D <dir> --clobber. This is a multi-request CLI operation. Its caller receives no digest on failure.- In
classify_failure_in→plan_withdrawal, the reported combination of scopeall, sourcedefault, about 300 seconds and no secondary-limit suffix identifiesFailure::Credential. HTTP 401 or “bad credentials” can select that branch. Its logger always printsremaining=- reset=-. Those dashes do not establish what headers GitHub sent or which request failed. - The failing request within the download is still unknown. Capture the failing stage (release lookup or asset fetch), response host name, status and sanitized error class, then compare with a regular issue read through the same gateway profile. Omit credentials, headers containing secrets, and signed redirect URLs. A measured refutation of the redirect or permission hypothesis is a valid diagnosis.
Implementation guidance
Use the observed failing stage to fix the checksum read on a supported reader path. When a refusal is specific to the asset operation, keep its effect local to that operation and use a short, bounded retry delay when no reset is supplied. The existing reader chain can retry past the gateway. Preserve the existing rules for a genuinely invalid owner credential, explicit throttling and a gateway outage. A configured reader pool that cannot serve the checksum must defer it without spending the writer credential.
Acceptance criteria
- The checksum read retrieves the published platform digest through a supported reader when available. An asset-specific refusal leaves an ordinary core read on the same gateway, owner and repo routable immediately. Document the failing stage and the evidence for the chosen fix.
- When all eligible checksum readers refuse, the checksum can be retried after a documented short interval below the 300-second credential withdrawal. The attempt neither marks a budget bucket empty nor reaches the writer. An explicit reset or Retry-After on a genuine rate limit retains its applicable policy.
- Live verification after rollout: attach the fleet check’s DAEMON-LOG counts for comparable full UTC hours, with host, time window and deployed version. Compare each host with its own pre-roll baseline. For worker-2 the reported range was 190–390 read-shed WARNs/hour. Confirm the count returns to baseline or below and the checksum-triggered owner-wide withdrawal stops across the affected hosts.
Test Plan
- Extend the fake-
ghreader-route fixture with the observed asset failure. At one fixed clock instant, run the checksum read, then a same-repo issue read. Assert which reader served each call and that the issue read succeeded. Exercise checksum retry at expiry and success through an eligible alternate reader. - Use a separate control for an actually invalid gateway credential on ordinary reads. Run the existing withdrawal tests for primary limits, secondary limits and gateway outage handling alongside the new regression. Verify the resolver still reads the expected digest from a successful download or equivalent reader path.
- Collect the fleet counts above with the same WARN filter as the original report. Keep that post-roll check open until evidence is available.
Affected Files
loom-daemon/src/release_resolve/resolve.rsandrelease_resolve/resolve/tests.rsunder the same source root — checksum retrieval and its success/failure fixtures.loom-daemon/src/gh_invocation/reader_route_v2.rsand siblingreader_route_v2_tests.rs— retry and shed decisions across successive reads.loom-daemon/src/forge_identity/withdrawal.rsandloom-daemon/src/forge_gateway.rs— verify the failure classification and gateway eligibility at the point selected by the diagnosis. Existing planner controls live inforge_identity/withdrawal_tests.rs.
Out of Scope
- Startup provenance retries and resync observability belong to workspace_resync: a failed release-tag verification at startup is never retried, so no host resyncs any workspace on 0.20.0 #14738. That issue cites this one as a possible trigger. The causal link remains unverified.
- First daemon start on a new host cannot fetch the fleet store: fleet-config render is shed when readers are configured and no daemon is running #14730 covers first-start routing before a daemon is running. Neither issue is a prerequisite for the checksum fix.
- Fleet-wide changes to credential or rate-limit backoff, writer fallback, and WARN suppression.
Activity
loom-fleet-dispatch commentedon Nov 20, 2026
PR #14739's
loom:curatingclaim is held by the curator of sweepstep-curator-14739-1795150261on hosthost-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assignedupdated_at, renewed while the curator works and released when it is done. Another curator that finds this lease live stands down before touching a worktree. Seedefaults/docs/lease-record.md§ "PR-phase leases".rjwalters commentedon Nov 20, 2026
Companion: #14738. On 0.20.0 the startup release-tag verification timed out on every host and was never retried, disabling workspace resync fleet-wide. These withdrawals are the likely cause.
loom-fleet-dispatch commentedon Nov 20, 2026
Added the traced credential-failure branch, the evidence needed to identify which asset request failed, and a two-read regression test to the issue body. Fleet WARN counts remain a post-roll acceptance check.
Reviewed the cross-reference from #14738. Its startup resync work is independent. This issue covers the checksum refusal and its effect on other reads.
loom dashboard
loom-fleet-dispatch commentedon Nov 20, 2026
PR #14739's
loom:evaluatingclaim is held by the champion of sweepstep-champion_promote-14739-1795150433on hosthost-cd042b53. This comment is a PR lease record (#9401): its liveness signal is its own forge-assignedupdated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. Seedefaults/docs/lease-record.md§ "PR-phase leases".loom-fleet-dispatch commentedon Nov 20, 2026
Champion Review: APPROVED
This issue has been evaluated and promoted to
loom:issuestatus. All quality criteria passed:Goal Alignment: Stops a resource-scoped
release.checksumrefusal from withdrawing the whole 2AMLogic reader gateway, which sheds every core read on the fleet (~20 min/h per host). The cited files exist onmain, and the acceptance criteria and test plan are concrete. The post-rollout fleet-count check stays open as a live-verification item.Ready for Builder to claim.
Automated by Champion role
loom dashboard
loom-fleet-dispatch commentedon Nov 20, 2026
PR #14739's
loom:evaluatingclaim is held by the champion of sweepstep-champion_promote-14739-1795150535on hosthost-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assignedupdated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. Seedefaults/docs/lease-record.md§ "PR-phase leases".10 remaining items
loom-fleet-dispatch commentedon Nov 20, 2026
This Builder step's lease (sweep
step-builder-14739-1795152548on hosthost-d0749b00) is released and no Builder runs for it: the Builder step is confirmed finished (exited). (#12822)loom dashboard
loom-fleet-dispatch commentedon Nov 20, 2026
Builder step ended without a PR (reason:
unclassified). It left no.no-changes-neededconclusion, so this is a failed pass, not a deliberate no-op. The daemon will retry under its failure ladder.rjwalters commentedon Nov 20, 2026
Auditor evidence from existing telemetry
In the reported November 20, 00:00–04:00 UTC window, Loom recorded 75
release.checksumexits carrying HTTP 401 across five daemon hosts. The status was parsed from GitHub CLI stderr. All 75 were booked to the gateway credential class. For 31 of the 75, an ordinary non-release read returned 200 in the same minute on the same daemon host, with the same exact repository, owner and credential class. These joins identify the daemon's recorded class. Individual reader instances and the remote endpoint that received each failing request are unknown.The central
github-readtraces in that window show 310 release-lookup 200s, 20 lookup 500s and four lookup 502s. They show zero release-lookup 401s, zero release asset-route spans, and zero gateway spans on the failure trace IDs. The stored telemetry does not identify the failed subrequest's stage or response host. An owner-wide credential rejection appears unlikely given the successful ordinary-read controls.GitHub CLI provides a concrete asset-fetch hypothesis. The tag lookup uses the repo host and a relative REST path. The subsequent download uses
DoRequestwith the absolute asset API URL from the release JSON. Upstream documents that these requests select their own destination outside the sharedapi_hostresolver. That makes a direct asset request toapi.github.complausible, while the actual response host and stage in these incidents remain unverified. GitHub CLI download source · GitHub CLI routing documentationThe existing Unix-socket GitHub CLI fixture can observe the authority and status for each stage with fake credentials and a synthetic release and checksum asset. It can compare a gateway-relative ordinary read with the download under the same temporary profile. Loom's fake-
ghrouting fixture separately covers owner-wide withdrawal on ordinary credential failures. This pass used existing traces and source inspection. Hermetic transport fixture · Loom routing fixtureloom-fleet-dispatch commentedon Nov 20, 2026
This issue's
loom:buildingclaim was acquired by builder stepstep-builder-14739-1795154558on hosthost-d0749b00at 2026-11-20T06:02:40.011227694+00:00. This comment is a lease record (Issue #6179, Epic #6165) — its liveness signal is this comment's own forge-assignedupdated_at, never a timestamp embedded in this text. Seedefaults/docs/lease-record.mdfor the format contract this establishes, anddefaults/docs/lease-renewal.mdfor how the owning sweep keeps it fresh for the lifetime of its claim.loom dashboard
loom-fleet-dispatch commentedon Nov 20, 2026
This Builder step's lease (sweep
step-builder-14739-1795154558on hosthost-d0749b00) is released and no Builder runs for it: the Builder step is confirmed finished (exited). (#12822)loom dashboard
rjwalters commentedon Nov 20, 2026
This issue is being worked by an in-session
/loom:sweeprunattended-codex-14739on hosthost-d9142cf3, published at 2026-11-20T06:05:16Z. This comment is a lease record (Issue #6320, Epic #6165) in the identical formatloom-daemon's dispatch-time write uses (#6179) — its liveness signal is this comment's own forge-assignedupdated_at, never the timestamp in this text. It is renewed bysweep-lease-renew.sh(#6180) for as long as this sweep's own process is alive, and simply ages out afterwards. Readers: the daemon's reclamation gate (#6286) refuses to reclaim this claim while the lease is fresh, dispatch-time ordering (#6287) yields to it, andsweep-lease-fence.sh(#6309) fences this sweep's own push/PR-open against it. Seedefaults/docs/lease-record.md.rjwalters commentedon Nov 20, 2026
This lease (sweep
attended-codex-14739on hosthost-d9142cf3) was released withloom-daemon lease release(#12103): the claimant stood down and its renewal loop was stopped.rjwalters commentedon Nov 20, 2026
Review impact observed on November 20
Two Fleet Judge passes on the same worker ended without a verdict during the reader-shed window. PR #14766 could no longer read formal reviews, and PR #14253 could no longer read its exact-head checks and main ruleset. Both stayed review-requested after their code-review passes. The local attended lane could read their exact hosted statuses as green. Formal-review gate refusal · CI gate refusal
This records the review impact. The two Judge receipts do not identify which operation withdrew the reader pool during this interval.
loom dashboard