Skip to content

Isolate synthetic gateway state from parallel claim-label tests #14775

Closed

Description

@rjwalters

Reproduction

Two claim-label fixtures fail when run beside the gateway publication fixture, even though all three pass serially. The original test binary built for Loom PR #14773 at 5f5e8bd1f96bdb638afd44838c9d5a4ee96b81ff returned 3 passed with one thread. In two separate offline replays, running the same three exact tests with three threads failed both claim-label tests in 9 of 9 invocations:

  • forge_gateway::tests::a_pass_publishes_each_owner_and_rewrites_api_host_on_every_publish
  • forge_check_claim::tests::read_claim_labels_distinguishes_claimed_clean_and_unknown
  • forge_check_claim::tests::exhausted_graphql_labels_use_the_same_claim_predicates_via_rest

The claim assertions see Unknown where they expect AwaitingCurator or Closed. The first claim-label test and the gateway publisher already exist on main f8f031568. The second claim-label test comes from #14773. The private trial for #14773 reproduced the same interleaving, while its 209-test serial run passed. The fixtures supplied synthetic readers and an inert fake CLI throughout. Related PR evidence

Cause

The publication fixture sets process-global gateway state. An unscoped claim-label read in another fixture sees that state through host_has_read_pool(). The production reader-only guard then sheds the call before its fake CLI can answer. The gateway tests' own fixture mutex currently leaves the claim probes outside its scope. This is interference between otherwise independent fixtures. Publisher · Reader-only guard

Acceptance

  • Reproduce the overlapping fixtures on the current base. Isolate the synthetic gateway publication from unrelated claim-label fixtures while preserving the actual gateway and claim routing assertions.
  • Exercise the original focused parallel interleaving repeatedly after the correction. On main, use the existing claim-label fixture. Include the new REST fallback fixture if fix(forge): recover admission reads after GraphQL exhaustion #14773 is present. Check that the isolated claim fixtures still run their stub and assert the expected labels.
  • Retain the test that a genuinely live route can cause an unscoped reader-only invocation to shed. Keep the production guard and default Rust test parallelism. Exercise the existing gateway, reader-route and claim suites together with offline fixtures.
  • Keep the change on the test surface. This issue is independent of the primary GraphQL budget classification and does not supersede fix(forge): recover admission reads after GraphQL exhaustion #14773.

Curator enhancement — 2026-11-20

Verified current main at f8f031568344efb9c069333519225d130f5d48b9. The gateway test mutex serializes publishers in its own module. The claim fixture reads the same process-wide state indirectly, without taking that mutex. With no derivable repo, shed_unroutable() calls host_has_read_pool(), which treats forge_gateway::any_live() as a live route. That is the expected production response. The original #14773 test binary reproduced both assertion failures in six repeated three-thread runs and passed all three focused tests with one thread. Current main contains the publisher and the original claim fixture. Reproduce the two-test overlap on the Builder's exact base first. Gateway state · Unscoped reader path · Host probe

Implementation guidance

Use a test-only isolation boundary for synthetic publication into DAEMON. A small, precedent-backed approach is to run the publishing fixture bodies in a child copy of the test executable with an exact opt-in test selector. Keep the public test names as wrappers so the default suite and focused filters still exercise each test. The child must execute the real publication, profile, disable, and cleanup assertions with its own temp directory, the existing fake dependencies, and an inert CLI. Keep fixture credentials and routing inputs synthetic. A fixture-local alternative is acceptable if it keeps the real routing checks and leaves every process-global observer isolated from unrelated fixtures. Existing test subprocess pattern

Cover every gateway fixture body that publishes through the shared singleton. Keep the current claim-label predicates and fake CLI, and leave general test parallelism and production DAEMON, any_live(), host_has_read_pool(), and reader-only routing behavior intact. Keep the positive pooled-reader and no-pool controls. In the isolated publisher context, verify an unscoped reader-only call sheds with a synthetic gateway present and never invokes the stub. In the independent claim fixture, verify the stub actually runs and the expected labels are classified.

Test plan

  • Build the libtest binary once in a private Cargo target. Use an owner-only, canonical temporary directory and offline fake CLI programs.
  • Run the gateway publisher and original claim-label fixture under the same libtest process with parallel threads and repeat the focused invocation six times. If the base includes fix(forge): recover admission reads after GraphQL exhaustion #14773, include its exact REST fallback fixture too. Record the pre-change and post-change results.
  • Run the gateway, reader-only, reader-route and claim suites together with the default parallelism. Confirm that wrapper tests actually invoke and check their isolated bodies, and retain the no-pool and pooled-host controls.
  • Run formatting and git diff --check. Keep work scoped to tests. fix(forge): recover admission reads after GraphQL exhaustion #14773's production GraphQL budget classification and Doctor cycle are separate.

Affected files

  • loom-daemon/src/forge_gateway/tests.rs
  • loom-daemon/src/gh_invocation/reader_only_tests.rs if needed for the positive control
  • Test module of loom-daemon/src/forge_check_claim.rs if needed for stub-execution evidence

Activity

  1. added
    loom:triageNew issue awaiting Curator enhancement. Applied by: humans or any agent filing an issue (intake).
    on Nov 20, 2026
  2. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    PR #14775's loom:curating claim is held by the curator of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the curator works and released when it is done. Another curator that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

  3. added
    loom:curatingCurator is enhancing this issue. Applied by: Curator only (claim label).
    loom:curatedCurator-enriched, awaiting approval. Additive; coexists with loom:issue. Applied by: Curator only.
    points:2Story-point size class 2. Applied by: Curator via loom-daemon curation-complete.
    on Nov 20, 2026
  4. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Curator

    The added guidance isolates the synthetic gateway publication while preserving the real publisher assertions and the production reader-only behavior. It gives the Builder a current-base parallel reproduction, direct checks for both stub execution and deferral, and a combined parallel test plan. The related #14773 PR covers the production GraphQL budget decision, so this test-only issue can proceed independently.

    loom dashboard

  5. removed
    loom:triageNew issue awaiting Curator enhancement. Applied by: humans or any agent filing an issue (intake).
    loom:curatingCurator is enhancing this issue. Applied by: Curator only (claim label).
    on Nov 20, 2026
  6. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    PR #14775's loom:evaluating claim is held by the champion of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

  7. added
    loom:evaluatingChampion is evaluating this proposal (claim label, stale after 15m). Applied by: Champion only.
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    and removed
    loom:evaluatingChampion is evaluating this proposal (claim label, stale after 15m). Applied by: Champion only.
    on Nov 20, 2026
  8. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Champion Review: APPROVED

    The independent replay established a concrete test isolation failure on the original PR binary. The current base already has a publisher and claim-label test that share the relevant path. A test-only isolation boundary is feasible and can preserve the existing gateway and claim routing assertions. The curation specifies the wrong-route and stub-execution controls, repeatable parallel verification, and the affected files.

    The scope covers fixture isolation and leaves Loom's production routing safeguards in use. The focused controls can run with fake dependencies and offline CLI stubs. This makes ordinary parallel test results reliable evidence for future admission and gateway changes.

    Ready for Builder to claim.

    loom dashboard

  9. added
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    and removed
    loom:issueApproved for work. Applied by: humans, Champion, sweep gate — see curator.md.
    on Nov 20, 2026
  10. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    This issue's loom:building claim was acquired by builder step step-builder-14775-1795171259 on host host-7ec0fa4c at 2026-11-20T10:41:00.868567926+00:00. This comment is a lease record (Issue #6179, Epic #6165) — its liveness signal is this comment's own forge-assigned updated_at, never a timestamp embedded in this text. See defaults/docs/lease-record.md for the format contract this establishes, and defaults/docs/lease-renewal.md for how the owning sweep keeps it fresh for the lifetime of its claim.

    loom dashboard

  11. rjwalters commented on Nov 20, 2026

    @rjwalters
    MemberAuthor

    Current-main replay for the active Builder. On unmodified main at f8f03156, the gateway publisher and original claim-label fixture passed together serially. At two threads, the claim fixture returned Unknown instead of Claimed in five of six runs.

    A private one-file trial ran each of the five gateway publication fixture bodies in a child copy of the test executable, using an exact test selector and asserting that the child ran one passing test. The focused parallel pair passed in all six repetitions. The combined gateway, reader and claim suites passed three times with default parallelism, 162 tests per run. Formatting and diff checks passed.

    The trial exercised the real production route inside the publisher child. With the synthetic gateway live and the read unscoped, the reader-only call shed and left the inert stub's launch marker absent. The independent claim fixture still used its CLI stub and asserted the expected labels. This is test-only evidence at the current base. Fleet retains the Builder branch and review workflow.

    loom dashboard

  12. loom-fleet-dispatch commented on Nov 20, 2026

    @loom-fleet-dispatch
    Contributor

    Builder step ended without a PR (reason: probe_failed). It left no .no-changes-needed conclusion, so this is a failed pass, not a deliberate no-op. The daemon will retry under its failure ladder.

  13. removed
    loom:buildingBuilder is implementing this issue. Applied by: Builder only (claim label).
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    loom:curatedCurator-enriched, awaiting approval. Additive; coexists with loom:issue. Applied by: Curator only.points:2Story-point size class 2. Applied by: Curator via loom-daemon curation-complete.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions