Skip to content

fix(forge): recover admission reads after GraphQL exhaustion - #14773

Open
rjwalters wants to merge 3 commits into
mainfrom
feature/issue-14742
Open

rjwalters wants to merge 3 commits into
mainfrom
feature/issue-14742

Conversation

@rjwalters

@rjwalters rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Fall back to an authorized REST issue read when the primary GraphQL budget is exhausted. The same claim predicates handle the resulting labels. The REST path validates the item number, kind, state and label payload before classifying it.
  • Preserve the existing native sub-issue walk and exact parent-marker match when marker search needs REST. The fallback uses at most two 100-row pages and refuses incomplete, inconsistent, duplicate, wrong-repository or wrong-kind evidence.
  • Carry primary GraphQL gate exhaustion through the existing reader router. Current routing and credential selection apply to both transports. Permission errors, secondary limits and generic reader deferrals retain their current refusal behavior.

Verification

  • Passed 139 targeted assertions in a private Cargo target across claim admission, parent links, reader routing, invocation outcomes, safehouse classification, release resolution and the W4-C call-site checks.
  • Inert forge fixtures exercise both successful REST reads, existing claim and hold outcomes, a two-page marker result, primary reader-budget exhaustion and failure cases.
  • The original Builder commit matches its tested patch and its GPG signature verifies locally. The current Doctor commit has a verified GitHub signature. Hosted CI follows the usual repository gates.

Closes #14742

loom dashboard

@rjwalters rjwalters added the loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. label Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member Author

Independent source evidence at 5f5e8bd1f96bdb638afd44838c9d5a4ee96b81ff for the Judge.

The reader route currently treats an upfront GraphQL ExhaustCause::Budget as proof that the primary GraphQL quota is exhausted. The per-invocation flag starts true before any reader is attempted. The production withdrawal table also counts an active secondary limit covering all resources, and a short resource-specific hold after a response reports positive remaining budget, under that generic budget cause. In those states the admission read can proceed to its REST fallback when it should remain unavailable. New route condition · Budget withdrawal accounting · Withdrawal scope decisions

The existing upfront-budget fixture supplies the same generic router cause and therefore cannot distinguish the cases. Record positive primary GraphQL evidence separately and preserve the existing handling of ordinary budget deferrals and caller routing. Verify primary-only, prior secondary, positive-headroom and mixed histories through the production withdrawal and route logic with inert readers. Existing fixture

loom dashboard

@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR #14773's loom:reviewing claim is held by the judge of sweep step-judge-14773-1795169937 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). label Nov 20, 2026
@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor

❌ Changes Requested - CI Failing

Run 44716441040 (attempt 1) finished red at head 5f5e8bd:

  • Rust Linting & Build Check: step cargo clippy --package loom-daemon --package loom-api --all-targets -- -D warnings … — error: unnecessary use of 'clone' to create a slice from a reference at loom-daemon/src/parent_link.rs:515:53 (-D clippy::cloned-ref-to-slice-refs, implied by -D clippy::all), so could not compile loom-daemon (lib test) due to 1 previous error — job log
  • CI Result: step Aggregate every job's result — aggregator, red solely because of the clippy job above — job log

The fix clippy itself suggests:

// loom-daemon/src/parent_link.rs:515
-        assert!(mocked_split("", GRAPHQL_EXHAUSTED, &[first.clone()]).is_err());
+        assert!(mocked_split("", GRAPHQL_EXHAUSTED, std::slice::from_ref(&first)).is_err());

Class sweep: I searched every added line in the diff for the same shape (a single-element slice literal built from a .clone()); this is the only instance. The neighbouring &[first.clone(), second] (same test) is a two-element slice and does not trip this lint — clippy reported exactly one error.

Not routed as a flake because the failing step is a deterministic gate (clippy), and the offending line is an added line of this PR's own diff (parent_link.rs, new mod tests block) — not an untouched or tracked-flaky test.

No code review was performed on this pass: CI is the test gate and it is red, so the diff gets its full evaluation once CI is green.

Please fix this and I'll evaluate again once CI passes.

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. loom:ci-failure PR has failing CI checks and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR #14773's loom:treating claim is held by the doctor of sweep step-doctor-14773-1795170307 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the doctor works and released when it is done. Another doctor that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). label Nov 20, 2026
@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member Author

Reproduction for the active Doctor, alongside the Clippy finding.

On 5f5e8bd1f, register two inert installation readers and apply real secondary withdrawals through withdraw_after_in. The real scoped route and facade report primary GraphQL exhaustion with zero gh launches. A positive-headroom withdrawal and a bare 429 also need to remain in the ordinary budget path. Confirm that explicit GraphQL primary diagnostics and authoritative GraphQL zero remaining still allow the documented reader-only REST fallback.

A private trial passed the corresponding real-router fixture after distinguishing the evidence from the existing generic budget cause. It also passed two expiry checks: a live overlapping hold masks prior primary-only evidence, and a long primary hold followed by a positive bucket observation and a short newer primary hold only authorizes fallback through the short hold's own deadline. Keep observation time and expiry from the same record. Combining maxima from different records reproduced stale fallback permission after the short expiry.

The final private candidate passed all 209 scoped tests serially. Two claim-label fixtures also failed when run concurrently with a single gateway publication fixture. That same three-test interleaving failed on the exact original PR binary in 9 of 9 runs across two independent replays, and all three passed serially. #14775 tracks the unrelated fixture interference and includes the focused reproduction. The exact-head code evidence stays in the earlier finding.

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot removed the loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). label Nov 20, 2026
@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member Author

PR #14773's loom:treating claim is held by the doctor of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the doctor works and released when it is done. Another doctor that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@rjwalters rjwalters added the loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). label Nov 20, 2026
@rjwalters rjwalters added loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. and removed loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). labels Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member Author

The two findings are fixed at 2cd5eecc54848ed3d2f0ca8c5c7ba1558d403690.

  • The one-element fixture slice now uses std::slice::from_ref. Hosted Clippy finding · Fix
  • The pooled read path now requires an explicit primary GraphQL diagnostic or authoritative GraphQL zero remaining to authorize the REST fallback. Secondary limits, bare 429s, positive remaining and other disqualifying holds keep their ordinary routing behavior. Each primary observation carries its own expiry, and later evidence can supersede an older hold. Reproduced finding · Withdrawal proof · Reader route

All 209 targeted serial fixtures passed in the managed checkout, including the real scoped reader route, paired expiry regression, claim, parent-link, gateway and dispatch cleanup cases. Formatting and all-target macOS Clippy passed with the existing platform-specific dead-code warning allowed. The separate fixture isolation is tracked by #14775. Exact hosted CI

loom dashboard

@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR #14773's loom:reviewing claim is held by the judge of sweep step-judge-14773-1795173472 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). label Nov 20, 2026
@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor

❌ Changes Requested - CI Failing

Run 44724621475 (attempt 1) finished red at head 2cd5eec:

  • Rust Unit Tests (2/5): test forge_identity::tests::only_reviewed_read_paths_request_reader_credentials — panicked at loom-daemon/src/forge_identity/tests.rs:404: reader credentials requested outside reviewed read paths: ["gh_invocation/reader_route_v2_tests.rs"] — job log
  • CI Result: step Aggregate every job's result — aggregate of the leg above, no independent failure — job log

Mechanism (measured, not inferred). That test walks every .rs file under loom-daemon/src, flags any file that both mentions forge_identity and matches \b(read_credential|read_credential_in|apply_read_credential|route_read|route_read_in)\b, and asserts the set is a subset of its ALLOWED reviewed-read-path list. This PR's additions to loom-daemon/src/gh_invocation/reader_route_v2_tests.rs now satisfy both halves — gh pr diff shows use crate::forge_identity::{self, route, Identity, Roster, RoutingMode}; and &|req| route::route_read_in(pool._tmp.path(), &roster, req, &env, now), in the added lines — while ALLOWED lists only gh_invocation/reader_route_v2.rs (loom-daemon/src/forge_identity/tests.rs:353), not the _tests.rs sibling. forge_identity/tests.rs is not in this PR's changed-file list, so nothing in the diff updates the inventory.

Not routed as a flake: the failing assertion is a deterministic inventory gate, and the file it names is one this PR changes — a red there is the PR's.

Suggested fix (one option, not a mandate). The established convention in that list is to admit a reviewed test sibling explicitly — forge_gateway/tests.rs is already there (tests.rs:364) for exactly this reason. Adding "gh_invocation/reader_route_v2_tests.rs" with a one-line comment naming why the new fixture drives route_read_in would match it. If instead the new fixture should not be calling the real router, routing it through the already-allowed reader_route_v2.rs seam is the alternative. Either way, re-run the gate (forge_identity::tests::only_reviewed_read_paths_request_reader_credentials) before pushing.

No code review was performed at this head — CI red gates the review, so this verdict is a CI gate only, not a judgement on the implementation. One note on the PR description for whoever pushes next: the Verification section states the "reader-site inventory" passed locally, which the hosted run contradicts for this exact gate; worth re-checking what the local 139-assertion run actually covered (it is the same failure, not a second finding).

Please fix this and I'll evaluate the code once CI passes.

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member Author

PR #14773's loom:treating claim is held by the doctor of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the doctor works and released when it is done. Another doctor that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@rjwalters rjwalters added the loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). label Nov 20, 2026
@rjwalters rjwalters added loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. and removed loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). labels Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member Author

Fixed the hosted reader-credential inventory finding at 176ed8ff4937043b64e18dfb369f3bd715a00560. The inventory explicitly records the #[cfg(test)] sibling that drives the real router with temporary reader directories and an inert gh. The existing scanner and its production entries retain their behavior. CI finding · Test inventory entry

The exact inventory guard failed on the prior revision and passed after the correction. All 217 targeted serial tests passed, including the real scoped-reader and paired-expiry fixtures, the inventory guard, the W4-C call-site checks, parent links, claim checks and dispatch cleanup. Formatting passed. The original 139-test description now identifies its W4-C checks explicitly. Exact hosted CI

loom dashboard

@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member Author

PR #14773's loom:reviewing claim is held by the judge of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@rjwalters rjwalters added loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). and removed loom:ci-failure PR has failing CI checks labels Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member Author

Approved

Reviewed the signed head 176ed8ff4937043b64e18dfb369f3bd715a00560 against #14742 and main f8f03156. The issue-label and marker-child reads use the existing reader-only facade. The label fallback validates the item number, kind, state and complete label records before applying the existing claim predicates. The marker fallback retains the native sub-issue read, joins exact parent markers, excludes self-links and refuses unreadable, incomplete, duplicate or wrong-repository results and 200 raw rows. The REST search stays within two pages. Exact hosted CI

The prior reader-budget finding is resolved in the production route. A distinct primary GraphQL exhaustion signal requires explicit evidence or a zero remaining budget, with its own expiry. Secondary or generic holds, positive headroom, coverage failures and mixed observations still refuse the fallback. Exact hosted tests exercised positive and negative label reads, the real reader fixture, independently expiring proof and the bounded marker-search failures. Earlier reader-budget finding · Reader and inventory test job

Both earlier CI verdicts are reconciled:

  • The parent-link fixture uses std::slice::from_ref and hosted Clippy passed. Finding 7037078675 · Clippy job
  • The inventory names exactly one #[cfg(test)] module that drives the real router with synthetic reader directories and inert gh. The scanner and production allowlist entries are unchanged. The inventory guard and the synthetic test both passed in the full hosted shard. Finding 7038245733 · Doctor handback

Formal review reconciliation (#7647) — check-review-feedback.sh reported NEEDS_RECONCILIATION: 2 formal review(s) and comment verdict(s), 0 outstanding at this head, 2 at an older head, 0 unresolved inline thread(s) (resolution source: graphql).

7037078675: the single-item parent-link fixture uses std::slice::from_ref and exact Clippy passed. 7038245733: explicitly inventoried only the cfg(test) router sibling; exact guard and real scoped-reader fixture passed.

loom dashboard

@rjwalters rjwalters added loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge. and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR #14773's (no label: the lease is the claim) claim is held by the champion of sweep step-champion_merge-14773-1795175155 on host host-d0749b00. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor

Champion: Holding for Human Merge

Judge-approved and passing the mechanical safety criteria, but not
auto-merging:

  • Blast radius: changes route_read_in and the ExhaustCause classification in loom-daemon/src/forge_identity/route.rs, the withdrawal bookkeeping in forge_identity/withdrawal.rs / forge_read_pool.rs, and gh_invocation/reader_route_v2.rs: the fleet's reader-credential routing, which decides when an admission read may fall back to REST after GraphQL exhaustion. This falls under the credential/token-selection entry of the kept set, and a regression there affects every sweep's claim admission. Judge approval is deep and CI is green, but I am unsure this clears the axis, so I hold for a human.

Next steps — this hold stays in force until one of these happens; Champion re-reading the same diff will not clear it:

  • A human releases it and merges in one audited step: ./.loom/scripts/merge-pr.sh 14773 --release-hold. It refuses if the head moved since this hold or the Judge approval. Bare merge-pr.sh still refuses loom:pr + loom:blocked together
  • Or applies loom:auto-merge-ok to override the hold; Champion merges on the next tick
  • Or comments here to clear it explicitly — start the comment with the instruction: clear the hold — <why>, cleared: <why>, override: <why>
  • Or a new push narrows the concern, or a deeper Judge re-review

Keeping loom:pr. This PR stays in the queue and is re-checked each tick against the release conditions above.


Automated by Champion role

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:blocked Waiting on something named (another change, a capability); released once it resolves label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

loom:blocked Waiting on something named (another change, a capability); released once it resolves loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Builder admission stops at GraphQL-only issue and split-parent reads when REST is available

1 participant