Skip to content

test: run gateway publication fixtures in child processes - #14779

Merged
loom-fleet-dispatch[bot] merged 4 commits into
mainfrom
feature/issue-14775
Nov 20, 2026
Merged

loom-fleet-dispatch[bot] merged 4 commits into
mainfrom
feature/issue-14775

Conversation

@loom-fleet-dispatch

@loom-fleet-dispatch loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The five forge_gateway fixtures that drive Refresher::pass_with move the process-global DAEMON singleton (set_active), and the module's serial() mutex only orders publishers inside forge_gateway::tests. An unscoped reader-only read in any other fixture of the same process answers host_has_read_pool() with the synthetic gateway (any_live()) and sheds before its fake CLI can answer — so the claim-label fixtures read Unknown where they expect a classified label, parallel-only.

Each publishing fixture body now runs in a child copy of the test binary behind an exact opt-in selector (the hooks_home umask-child pattern, hooks_home_tests.rs); the public test names stay as wrappers, so the default suite and focused filters still exercise every body. The primary publisher's child also pins the retained production guard: with the synthetic gateway live, an unscoped claim-label read sheds (Unknown) and never consults an inert CLI. The claim fixture gains direct stub-execution evidence — a counting stub whose marker file proves the CLI actually ran.

Closes #14775

Changes

  • loom-daemon/src/forge_gateway/tests.rs — in_a_child_process helper; the five pass_with fixtures (a_pass_publishes_each_owner_and_rewrites_api_host_on_every_publish, a_replaced_gh_is_reported_on_the_next_refresh, an_old_gh_publishes_nothing_and_removes_what_was_there, a_dropped_owner_s_profile_is_removed, with_the_gateway_unset_routing_is_byte_identical_to_today) become wrappers over #[ignore]d child bodies; the primary publisher's child asserts the live-route shed control (unscoped read sheds, inert CLI never invoked).
  • loom-daemon/src/forge_check_claim.rs — read_claim_labels_distinguishes_claimed_clean_and_unknown gains a counting stub asserting the stub itself executed alongside its label classification.

Production DAEMON, any_live(), host_has_read_pool() and reader-only routing are untouched; default Rust test parallelism and the pooled-reader / no-pool controls (reader_only_tests.rs) are kept as-is.

Acceptance Criteria Verification

  • Reproduce the overlapping fixtures on the current base: done pre-change on f8f031568 — the focused pair run with --test-threads=2 (owner-only canonical TMPDIR, offline stub CLIs) failed read_claim_labels_distinguishes_claimed_clean_and_unknown in 4 of 6 invocations (left: Unknown at the AwaitingCurator/claim asserts), passing serially — the exact failure filed.
  • Isolate the synthetic gateway publication … preserving assertions: every pass_with fixture body runs in its own child process; each child executes the original publication/profile/disable/cleanup assertions verbatim with its own tempdir, the existing Fake deps and an inert CLI. All five child bodies pass under the wrappers' exact selectors (--exact forge_gateway::tests::<name>_child --ignored --test-threads=1, 1 passed each).
  • Exercise the original focused interleaving repeatedly after the correction: the same two-test invocation post-change, 6 of 6 green. fix(forge): recover admission reads after GraphQL exhaustion #14773 is not on this base, so its REST fallback fixture is not present.
  • Isolated claim fixtures still run their stub and assert expected labels: the new counting stub asserts Clean classification and that its marker file exists — direct stub-execution evidence.
  • Retain the live-route shed, production guard, default parallelism: the production guard is asserted in the isolated publisher child (any_live() true → unscoped read_claim_labels = Unknown, inert CLI marker absent). No production code changed; default test parallelism kept.
  • Gateway, reader-route and claim suites together, offline fixtures: forge_gateway + forge_check_claim + reader_only + reader_route filters in one process, default parallelism: 109 passed, 0 failed (5 ignored = the child bodies, exercised via their wrappers). Note: 4 front_base/cli_base privacy tests also fail identically on the unmodified main binary on this host — this worker's umask is 0002, so std::fs::write-created marker files are group-writable and safe_base's private-file check refuses them; under umask 022 the combined run is fully green. Pre-existing host artifact, unrelated to and untouched by this change.

Test Plan

  • TDD: the reproduction was the failing test — the pre-change focused interleaving failed 4/6; post-change the same invocation passes 6/6, and the new in-child shed control encodes the interference permanently (it fails if the gateway state ever leaks out of isolation again).
  • cargo test -p loom-daemon --lib --no-run (worktree-local CARGO_TARGET_DIR), focused interleaving ×6, child bodies ×5, combined suites — all green.
  • cargo clippy --package loom-daemon --all-targets with CI's exact flag set: clean. cargo fmt applied; git diff --check clean.
  • CI runs the full suite (build gate Host-side test burden to a bare minimum: CI is the test gate, Builders don't run the suite (operator direction 2026-10-13) #11690); CI was still running at hand-off.

loom dashboard

The five forge_gateway fixtures that drive Refresher::pass_with move the
process-global DAEMON singleton (set_active), and their module mutex only
orders publishers inside forge_gateway::tests: an unscoped reader-only
read in any other fixture of the same process answers host_has_read_pool()
with the synthetic gateway and sheds before its fake CLI can answer, so
the claim-label fixtures read Unknown where they expect a classified label
(parallel-only, 4/6 focused replays pre-change).

Each publishing body now runs in a child copy of the test binary behind an
exact opt-in selector (the hooks_home umask pattern); the public names stay
as wrappers, so the default suite and focused filters exercise every body.
The primary publisher's child also pins the retained production guard: with
the synthetic gateway live, an unscoped claim-label read sheds (Unknown) and
never consults an inert CLI. The claim fixture gains direct stub-execution
evidence — a counting stub whose marker file proves the CLI ran.

Production DAEMON, any_live(), host_has_read_pool() and reader-only routing
are untouched; default Rust test parallelism is kept.

Closes #14775
@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. label Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

PR #14779's loom:reviewing claim is held by the judge of sweep step-judge-14779-1795174625 on host host-d0749b00. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch loom-fleet-dispatch Bot added the loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). label Nov 20, 2026
@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor Author

❌ Changes Requested

Test-only diff, CI green at 406d836e, and the two new positive assertions are the right shape. One blocking defect in the new isolation helper: the wrapper→child link is a bare exit-0, so it cannot tell "the child body ran and passed" from "the child body never ran at all." That is the one thing the whole mechanism has to prove, and it is the acceptance criterion the issue asks to confirm.


Blocking: in_a_child_process treats a bare exit-0 as proof the child ran

loom-daemon/src/forge_gateway/tests.rs:30-42 — the helper's only evidence is assert!(status.success(), …), with the child's stdout sent to Stdio::null().

Measured (not inferred). A libtest binary whose --exact filter matches nothing exits 0:

$ rustc --test t.rs -o t     # t.rs: fn wrapper(), #[ignore] fn real_child()

$ ./t --exact real_child --ignored --test-threads=1
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out
exit=0

$ ./t --exact real_child_typo --ignored --test-threads=1      # nothing matches
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out
exit=0

So a renamed/misspelled _child fn, or a _child fn that loses its #[ignore] (--ignored then selects nothing), makes the wrapper pass green while running no test at all. The entire publish / profile / disable / cleanup assertion set of these fixtures would disappear with no red anywhere — precisely the silent-omission class this PR exists to defend against.

Why this is blocking rather than a nit:

  1. The issue's own acceptance criterion is not met. Isolate synthetic gateway state from parallel claim-label tests #14775's Test plan: "Confirm that wrapper tests actually invoke and check their isolated bodies." The helper cannot confirm invocation.
  2. .loom/docs/fail-closed.md:17-22 names this exact pattern: "Success must be positively proven… Absence, null, an empty result set, '0 rows', or a bare exit-0" is not a result. The new safety net rests on a bare exit-0.
  3. The blessed design included the assertion that got dropped. The operator's current-base replay (issue comment, the trial the Curator's guidance points at) describes it as "using an exact test selector and asserting that the child ran one passing test."
  4. The cited precedent is immune; this helper is not. hooks_home_tests.rs:319-343 makes the same status.success() check, but the parent then asserts on side effects the child must have produced — read_dir(&base).expect("the child unpacked under the base"), copies.len() == 1, the two mode checks. If that child never ran, the parent test fails. in_a_child_process has no parent-side evidence of any kind, so it drops the one property that made the precedent safe.

Class sweep — one helper, five affected call sites (all of the new wrappers; no other instances of this pattern in the diff):

  • tests.rs:235 a_pass_publishes_each_owner_and_rewrites_api_host_on_every_publish
  • tests.rs:314 a_replaced_gh_is_reported_on_the_next_refresh
  • tests.rs:348 an_old_gh_publishes_nothing_and_removes_what_was_there
  • tests.rs:374 a_dropped_owner_s_profile_is_removed
  • tests.rs:438 with_the_gateway_unset_routing_is_byte_identical_to_today

Suggested fix (one place, covers all five). Capture stdout instead of discarding it and require the child to report a test:

let out = std::process::Command::new(std::env::current_exe().unwrap())
    .args(["--exact", &format!("forge_gateway::tests::{child}"), "--ignored", "--test-threads=1"])
    .output()
    .unwrap();
let log = String::from_utf8_lossy(&out.stdout);
assert!(out.status.success(), "the gateway fixture child `{child}` failed: {}\n{log}", out.status);
assert!(
    log.contains("1 passed"),
    "the child `{child}` ran no test — the selector matched nothing (renamed fn, or `#[ignore]` removed):\n{log}"
);

This also fixes a second consequence of Stdio::null(): today a genuinely failing child reports only exit status: 101, with the panic message and assertion text discarded. Asserting a parent-visible side effect, as the precedent does, would be equally acceptable.


What is good (keep as-is)

  • forge_check_claim.rs:1841-1856 — the counting stub is exactly the right discipline: it asserts the classification and ran.is_file(), so "the CLI actually executed" is positively proven rather than assumed. This is the pattern the gateway helper is missing. Verified by reading: the stub emits {"labels":[{"name":"loom:issue"}]}, which falls through is_frozen / is_operator_hold / claim_labels_in / TRIAGE_LABEL to LabelLeg::Clean (forge_check_claim.rs:749-767) — the asserted value is right.
  • The in-child shed control (tests.rs:268-284) genuinely exercises the production route: any_live() true, the read unscoped against a non-repo tempdir → shed_unroutable() → host_has_read_pool() → Unknown, with !ran.exists() proving the inert CLI was never launched. It pins the mechanism the issue diagnosed, and it is a real assertion rather than a circular fixture. Its placement mid-body writes only ws/gh-inert (not under base_dir(ws)), so the later republish / unset assertions are unaffected.
  • Wrapper→child name pairing is correct for all five, and --ignored in the child prevents wrapper recursion.
  • Production DAEMON, any_live(), host_has_read_pool() and reader-only routing are untouched, as Isolate synthetic gateway state from parallel claim-label tests #14775 requires.

Non-blocking note: this helps cargo test only, not CI

Recording this so it is not re-derived later; it is not a reason to change the approach — the operator's replay comment blessed this exact child-process design on the current base.

CI runs cargo nextest run (.github/workflows/ci.yml:504-544, ci-daily.yml), which already runs every test in its own process. .config/nextest.toml and loom-daemon/src/lib.rs:35-49 document that as the structural answer to this whole hazard class — "Process-per-test closes it by construction, including for every future test nobody remembers to mark", and "a failure seen only under cargo test may be this, not your change. CI is the arbiter of green." #14775's reproduction is a plain-cargo test --test-threads=2 failure, so the interference cannot occur in CI. The value here is for developers running plain cargo test (the same reason the crate docs keep #[serial] as "load-bearing for developers running plain cargo test"), plus the two new positive assertions, which stand on their own.

Spec reconciliation

Re-read #14775's current body and trusted comments at review time (no earlier spec baseline available; reviewed against current requirements). Curator enhancement 2026-11-20 and Champion approval are the latest substantive updates; no rescope. The PR's scope matches the issue's four acceptance bullets, with the one gap above ("Confirm that wrapper tests actually invoke and check their isolated bodies"). Closes #14775 present and correct.

Verification performed

  • gh pr diff read in full; reviewed in .loom/worktrees/pr-14779 at 406d836e (worktree HEAD matched PR head, tree clean).
  • CI: LOOM-CHECKS-GREEN at 406d836e — the exact head reviewed (loom-daemon forge wait-checks 14779 --timeout 600 --settle-red). Green CI is consistent with the finding: today the selectors are spelled correctly, so the children do run. The defect is that nothing would detect it if they stopped.
  • check-review-feedback.sh --number 14779 --head-sha 406d836e → REVIEW_FEEDBACK_STATE=CLEAR (0 reviews, 0 unresolved inline threads).
  • mergeStateStatus: BLOCKED at review start (checks pending), now green; no conflicts, no rebase needed.
  • Libtest exit-code behaviour measured locally with a throwaway rustc --test binary (output quoted above).
  • Not re-run locally: the loom-daemon suite itself — CI is the test gate (Host-side test burden to a bare minimum: CI is the test gate, Builders don't run the suite (operator direction 2026-10-13) #11690) and is green on this head.

TDD claim

TDD: line present and the reproduction was the failing test (pre-change 4/6 focused-interleaving failures, post-change 6/6). Not independently re-verified here — reproducing it needs a merge-base build plus repeated two-thread runs, which I did not run; the claim is the Builder's, recorded as reported rather than re-derived. The new in-child shed control is a genuine permanent regression guard for the mechanism.

Test Plan execution

  1. cargo test -p loom-daemon --lib --no-run + focused interleaving ×6 + child bodies ×5 — ⏭️ Skipped: full-suite/build-gate class, CI already ran it green (Host-side test burden to a bare minimum: CI is the test gate, Builders don't run the suite (operator direction 2026-10-13) #11690).
  2. cargo clippy --package loom-daemon --all-targets, cargo fmt, git diff --check — ✅ Covered by green Rust Linting & Build Check at 406d836e.
  3. Libtest selector semantics (the one specific doubt) — ✅ Executed, see above.

Fix the helper's child-ran assertion and this is an approve; nothing else in the diff needs to change.

loom dashboard

@loom-fleet-dispatch loom-fleet-dispatch Bot added loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

PR #14779's loom:treating claim is held by the doctor of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the doctor works and released when it is done. Another doctor that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@rjwalters rjwalters added the loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). label Nov 20, 2026
@rjwalters rjwalters added loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. and removed loom:changes-requested PR requires changes before re-review (Judge requested modifications). Applied by: Judge. loom:treating Doctor is fixing this bug or PR. Applied by: Doctor. Stale after LOOM_STALE_TREATING_MINUTES (60m). labels Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member

Fixed the child-run evidence finding at 1baa88a7b4c3fe7e0fec81e3f09fa56321851681. The shared helper now requires a successful child process and a libtest summary reporting exactly one passing test. Failure messages include the child's stdout and stderr. The permanent negative control selects a missing child and requires the captured zero-test result to be rejected. Judge finding · Fixture helper

The original helper let the missing child return success. The corrected helper rejected that case, accepted one passing child, and surfaced a deliberate child failure in an isolated probe. All six focused parallel replays and 163 default-parallel fixture tests passed. The five isolated child bodies still run through their wrappers. Formatting and diff checks passed. The final base update changed two MCP files, and the focused gateway, claim, and missing-child tests passed again against the same tested Rust files. Exact hosted CI

loom dashboard

@rjwalters

rjwalters commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

PR #14779's loom:reviewing claim is held by the judge of sweep sweep-insession-s38728-ThuNov191721502026 on host host-d9142cf3. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the judge works and released when it is done. Another judge that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@rjwalters rjwalters added the loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). label Nov 20, 2026
@rjwalters

Copy link
Copy Markdown
Member

Approved

The new helper requires a successful child process and a libtest summary with one passing test. I extracted the helper and permanent regression test from the signed revision and ran three inert Rust checks. They verified that one passing child succeeds, a missing selector with exit 0 is rejected, and an actual child failure reports both output streams.

All five gateway fixture wrappers target different ignored bodies. The production reader-only routing is unchanged. The publisher still checks that a live synthetic gateway sheds an unscoped read, and the separate claim fixture proves its stub was invoked. The two test-file changes compose cleanly with current main ddb52920. The exact hosted run passed. The Doctor's full fixture receipt records six focused replays and 163 default-parallel cases. Hosted CI · Doctor verification


Formal review reconciliation (#7647) — check-review-feedback.sh reported NEEDS_RECONCILIATION: 1 formal review(s) and comment verdict(s), 0 outstanding at this head, 1 at an older head, 0 unresolved inline thread(s) (resolution source: graphql).

7038741514: Fixed at loom-daemon/src/forge_gateway/tests.rs:30-58. All five wrappers now require one passing test and log both streams. The permanent missing-child test confirms that the exit-0 zero-test case is refused. An independent failing child also surfaced its named result, stdout, and stderr.

loom dashboard

@rjwalters rjwalters added loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge. and removed loom:review-requested PR ready for Judge to review. Applied by: Builder when opening PR. loom:reviewing Judge is reviewing this PR. Applied by: Judge only. Stale after LOOM_STALE_REVIEWING_MINUTES (30m). labels Nov 20, 2026
@loom-fleet-dispatch

loom-fleet-dispatch Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

PR #14779's (no label: the lease is the claim) claim is held by the champion of sweep step-champion_merge-14779-1795176689 on host host-a6032f0c. This comment is a PR lease record (#9401): its liveness signal is its own forge-assigned updated_at, renewed while the champion works and released when it is done. Another champion that finds this lease live stands down before touching a worktree. See defaults/docs/lease-record.md § "PR-phase leases".

@loom-fleet-dispatch

Copy link
Copy Markdown
Contributor Author

Champion Auto-Merge

This PR meets all safety criteria for automatic merging:

  • Judge approved (loom:pr label, verdict marker matches head 1baa88a)
  • Merge-risk judgment passed: diff is test-only (two #[cfg(test)] modules, +114/-0); no production code path, no high-blast-radius surface; fully revertable
  • Diff size: 114 lines (+114/-0) — informational, not a gate
  • No critical files modified
  • No merge conflicts
  • Updated recently (0 hours ago)
  • All CI checks passing

Proceeding with merge... If this was merged in error, you can revert with:
git revert -m 1 <merge-sha>


Automated by Champion role

loom dashboard

@loom-fleet-dispatch
loom-fleet-dispatch Bot merged commit 9e981e5 into main Nov 20, 2026
24 checks passed
@loom-fleet-dispatch
loom-fleet-dispatch Bot deleted the feature/issue-14775 branch November 20, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

loom:pr PR approved by Judge, ready for Champion auto-merge. Applied by: Judge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Isolate synthetic gateway state from parallel claim-label tests

1 participant