Skip to content

forge_identity: a release.checksum refusal withdraws the reader gateway for the whole owner, shedding every 2AMLogic read ~20 min/h since 0.20.0 #14739

Open

Description

@rjwalters

What happens

Since the 0.20.0 roll, every fleet host logs four times an hour:

[WARN] forge_identity: reader app gateway withdrawn from 2amlogic all for 299s (end from default; refusal reported remaining=- reset=-; release.checksum via the gh choke point); its other owners and resources keep serving — W4-A

Every withdrawal on worker-2 from 00Z to 04Z has the same cause, release.checksum. While it lasts, every 2AMLogic core read is shed even though both reader apps report usable budget:

[INFO] gh_invocation: step_engine shed — no reader can serve 2amlogic core until …Z (router: budget — app …: usable, book 5723/15000 …; app …: usable, book 3593/15000 …)
[WARN] work_finder: 2AMLogic/<repo>: step listing failed (read shed: no reader can serve it until …; deferred, not sent to the writer (loom-shed)); using the listing from 337s ago

Read-shed WARNs per host per hour (worker-2): 252 (00Z, roll at ~00:03), then 762, 950, 960, 960. Before the roll they ran 190–390/h. The same shape is on all six hosts (700–1,100/h).

Why it is wrong

  • A refusal on one resource (a release-asset checksum read, which reports no remaining/reset, i.e. not a budget refusal) withdraws the gateway for the entire owner (2amlogic all) for the default 299 s. That is about 20 min/h per host in which no reader serves 2AMLogic.
  • It probably also caused the startup release-tag verification to time out, which disabled workspace resync fleet-wide (companion issue).
  • The work finder falls back to stale listings (5–6 min old), so dispatch keeps going, but everything else that reads is deferred.

What to do

  • Scope a refusal without budget headers to the resource or operation that drew it (here release.checksum), not to the owner's whole core budget.
  • Find why release.checksum is refused for the reader apps since 0.20.0 (an asset download redirect, a permission, or the repo move to 2AMLogic) and fix or route that read.
  • Do not apply the default 299 s to a refusal that carries no reset; use a short backoff.

Acceptance

  • A refused release.checksum read no longer sheds unrelated 2AMLogic reads.
  • Read-shed WARNs return to the pre-0.20.0 level or below, measured by the fleet check's DAEMON-LOG section.
  • A test: a resource-scoped refusal leaves other operations on the same owner routable.

Related: #14730, #11838, #12245.

Filed from an attended fleet check on 2026-11-20.

Curator findings · 2026-11-20

Traced on main at 21f029145. The resolver, reader route, gateway router and withdrawal planner have identical source at tag v0.20.0 (git diff v0.20.0..21f029145 over the four files below).

  • fetch_asset_sha256 runs the reader-only command gh release download <tag> -R <repo> -p <binary>.sha256 -D <dir> --clobber. This is a multi-request CLI operation. Its caller receives no digest on failure.
  • In classify_failure_in → plan_withdrawal, the reported combination of scope all, source default, about 300 seconds and no secondary-limit suffix identifies Failure::Credential. HTTP 401 or “bad credentials” can select that branch. Its logger always prints remaining=- reset=-. Those dashes do not establish what headers GitHub sent or which request failed.
  • The failing request within the download is still unknown. Capture the failing stage (release lookup or asset fetch), response host name, status and sanitized error class, then compare with a regular issue read through the same gateway profile. Omit credentials, headers containing secrets, and signed redirect URLs. A measured refutation of the redirect or permission hypothesis is a valid diagnosis.

Implementation guidance

Use the observed failing stage to fix the checksum read on a supported reader path. When a refusal is specific to the asset operation, keep its effect local to that operation and use a short, bounded retry delay when no reset is supplied. The existing reader chain can retry past the gateway. Preserve the existing rules for a genuinely invalid owner credential, explicit throttling and a gateway outage. A configured reader pool that cannot serve the checksum must defer it without spending the writer credential.

Acceptance criteria

  • The checksum read retrieves the published platform digest through a supported reader when available. An asset-specific refusal leaves an ordinary core read on the same gateway, owner and repo routable immediately. Document the failing stage and the evidence for the chosen fix.
  • When all eligible checksum readers refuse, the checksum can be retried after a documented short interval below the 300-second credential withdrawal. The attempt neither marks a budget bucket empty nor reaches the writer. An explicit reset or Retry-After on a genuine rate limit retains its applicable policy.
  • Live verification after rollout: attach the fleet check’s DAEMON-LOG counts for comparable full UTC hours, with host, time window and deployed version. Compare each host with its own pre-roll baseline. For worker-2 the reported range was 190–390 read-shed WARNs/hour. Confirm the count returns to baseline or below and the checksum-triggered owner-wide withdrawal stops across the affected hosts.

Test Plan

  • Extend the fake-gh reader-route fixture with the observed asset failure. At one fixed clock instant, run the checksum read, then a same-repo issue read. Assert which reader served each call and that the issue read succeeded. Exercise checksum retry at expiry and success through an eligible alternate reader.
  • Use a separate control for an actually invalid gateway credential on ordinary reads. Run the existing withdrawal tests for primary limits, secondary limits and gateway outage handling alongside the new regression. Verify the resolver still reads the expected digest from a successful download or equivalent reader path.
  • Collect the fleet counts above with the same WARN filter as the original report. Keep that post-roll check open until evidence is available.

Affected Files

  • loom-daemon/src/release_resolve/resolve.rs and release_resolve/resolve/tests.rs under the same source root — checksum retrieval and its success/failure fixtures.
  • loom-daemon/src/gh_invocation/reader_route_v2.rs and sibling reader_route_v2_tests.rs — retry and shed decisions across successive reads.
  • loom-daemon/src/forge_identity/withdrawal.rs and loom-daemon/src/forge_gateway.rs — verify the failure classification and gateway eligibility at the point selected by the diagnosis. Existing planner controls live in forge_identity/withdrawal_tests.rs.

Out of Scope

Activity

  1. added
    loom:triageNew issue awaiting Curator enhancement. Applied by: humans or any agent filing an issue (intake).
    on Nov 20, 2026
  2. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    PR #14739's loom:curating claim is held by the curator of sweep step-curator-14739-1795150261 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the curator works and released when it is done. Another curator that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

  3. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Companion: #14738. On 0.20.0 the startup release-tag verification timed out on every host and was never retried, disabling workspace resync fleet-wide. These withdrawals are the likely cause.

  4. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    Added the traced credential-failure branch, the evidence needed to identify which asset request failed, and a two-read regression test to the issue body. Fleet WARN counts remain a post-roll acceptance check.

    Reviewed the cross-reference from #14738. Its startup resync work is independent. This issue covers the checksum refusal and its effect on other reads.

    loom dashboard

  5. added
    loom:curatedCurator-enriched, awaiting approval. Additive; coexists with loom:issue. Applied by: Curator only.
    points:3Story-point size class 3. Applied by: Curator via loom-daemon curation-complete.
    and removed
    loom:triageNew issue awaiting Curator enhancement. Applied by: humans or any agent filing an issue (intake).
    loom:curatingCurator is enhancing this issue. Applied by: Curator only (claim label).
    on Nov 20, 2026
  6. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    PR #14739's loom:evaluating claim is held by the champion of sweep step-champion_promote-14739-1795150433 on host host-cd042b53. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

  7. added
    loom:evaluatingChampion is evaluating this proposal (claim label, stale after 15m). Applied by: Champion only.
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    and removed
    loom:evaluatingChampion is evaluating this proposal (claim label, stale after 15m). Applied by: Champion only.
    on Nov 20, 2026
  8. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    Champion Review: APPROVED

    This issue has been evaluated and promoted to loom:issue status. All quality criteria passed:

    • Clear problem statement
    • Technical feasibility
    • Implementation clarity
    • Value alignment
    • Scope appropriateness
    • Quality standards
    • Risk assessment
    • Completeness

    Goal Alignment: Stops a resource-scoped release.checksum refusal from withdrawing the whole 2AMLogic reader gateway, which sheds every core read on the fleet (~20 min/h per host). The cited files exist on main, and the acceptance criteria and test plan are concrete. The post-rollout fleet-count check stays open as a live-verification item.

    Ready for Builder to claim.


    Automated by Champion role

    loom dashboard

  9. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    PR #14739's loom:evaluating claim is held by the champion of sweep step-champion_promote-14739-1795150535 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

  10. 10 remaining items

  11. removed
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    on Nov 20, 2026
  12. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    This Builder step's lease (sweep step-builder-14739-1795152548 on host host-d0749b00) is released and no Builder runs for it: the Builder step is confirmed finished (exited). (#12822)

    loom dashboard

  13. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    Builder step ended without a PR (reason: unclassified). It left no .no-changes-needed conclusion, so this is a failed pass, not a deliberate no-op. The daemon will retry under its failure ladder.

  14. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Auditor evidence from existing telemetry

    In the reported November 20, 00:00–04:00 UTC window, Loom recorded 75 release.checksum exits carrying HTTP 401 across five daemon hosts. The status was parsed from GitHub CLI stderr. All 75 were booked to the gateway credential class. For 31 of the 75, an ordinary non-release read returned 200 in the same minute on the same daemon host, with the same exact repository, owner and credential class. These joins identify the daemon's recorded class. Individual reader instances and the remote endpoint that received each failing request are unknown.

    The central github-read traces in that window show 310 release-lookup 200s, 20 lookup 500s and four lookup 502s. They show zero release-lookup 401s, zero release asset-route spans, and zero gateway spans on the failure trace IDs. The stored telemetry does not identify the failed subrequest's stage or response host. An owner-wide credential rejection appears unlikely given the successful ordinary-read controls.

    GitHub CLI provides a concrete asset-fetch hypothesis. The tag lookup uses the repo host and a relative REST path. The subsequent download uses DoRequest with the absolute asset API URL from the release JSON. Upstream documents that these requests select their own destination outside the shared api_host resolver. That makes a direct asset request to api.github.com plausible, while the actual response host and stage in these incidents remain unverified. GitHub CLI download source · GitHub CLI routing documentation

    The existing Unix-socket GitHub CLI fixture can observe the authority and status for each stage with fake credentials and a synthetic release and checksum asset. It can compare a gateway-relative ordinary read with the download under the same temporary profile. Loom's fake-gh routing fixture separately covers owner-wide withdrawal on ordinary credential failures. This pass used existing traces and source inspection. Hermetic transport fixture · Loom routing fixture

  15. added
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    and removed
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    on Nov 20, 2026
  16. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    This issue's loom:building claim was acquired by builder step step-builder-14739-1795154558 on host host-d0749b00 at 2026-11-20T06:02:40.011227694+00:00. This comment is a lease record (Issue #6179, Epic #6165) — its liveness signal is this comment's own forge-assigned updated_at, never a timestamp embedded in this text. See defaults/docs/lease-record.md for the format contract this establishes, and defaults/docs/lease-renewal.md for how the owning sweep keeps it fresh for the lifetime of its claim.

    loom dashboard

  17. added
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    and removed
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    on Nov 20, 2026
  18. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    This Builder step's lease (sweep step-builder-14739-1795154558 on host host-d0749b00) is released and no Builder runs for it: the Builder step is confirmed finished (exited). (#12822)

    loom dashboard

  19. added
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    and removed
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    on Nov 20, 2026
  20. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    This issue is being worked by an in-session /loom:sweep run attended-codex-14739 on host host-d9142cf3, published at 2026-11-20T06:05:16Z. This comment is a lease record (Issue #6320, Epic #6165) in the identical format loom-daemon's dispatch-time write uses (#6179) — its liveness signal is this comment's own forge-assigned updated_at, never the timestamp in this text. It is renewed by sweep-lease-renew.sh (#6180) for as long as this sweep's own process is alive, and simply ages out afterwards. Readers: the daemon's reclamation gate (#6286) refuses to reclaim this claim while the lease is fresh, dispatch-time ordering (#6287) yields to it, and sweep-lease-fence.sh (#6309) fences this sweep's own push/PR-open against it. See defaults/docs/lease-record.md.

  21. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    This lease (sweep attended-codex-14739 on host host-d9142cf3) was released with loom-daemon lease release (#12103): the claimant stood down and its renewal loop was stopped.

  22. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Review impact observed on November 20

    Two Fleet Judge passes on the same worker ended without a verdict during the reader-shed window. PR #14766 could no longer read formal reviews, and PR #14253 could no longer read its exact-head checks and main ruleset. Both stayed review-requested after their code-review passes. The local attended lane could read their exact hosted statuses as green. Formal-review gate refusal · CI gate refusal

    This records the review impact. The two Judge receipts do not identify which operation withdrew the reader pool during this interval.

    loom dashboard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).loom:curatedCurator-enriched, awaiting approval. Additive; coexists with loom:issue. Applied by: Curator only.points:3Story-point size class 3. Applied by: Curator via loom-daemon curation-complete.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions