Skip to content

test: de-flake the two assertions ejecting merge groups (#1787, #1952) - #1987

Queued
kkroo wants to merge 2 commits into
masterfrom
test/deflake-agent-action-buttons-render
Queued

kkroo wants to merge 2 commits into
masterfrom
test/deflake-agent-action-buttons-render

Conversation

@kkroo

@kkroo kkroo commented Sep 22, 2026 •

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • Its changes land through a GitHub merge queue, which runs the full suite per group
  • On 2026-09-21, 17 of 33 merge groups with a final verdict produced no merge — each failure ejects its PR back to the queue tail
  • The ejectors are timing-sensitive assertions in unrelated suites, which is the signature of hypervisor CPU steal (BLO-35090) rather than of any diff under test
  • Rebalancing the hypervisor is the root fix and is not in this repo, but each test made steal-immune is one fewer ejector
  • This pull request removes the two that ejected groups that day: fix(metrics): give the Caveman readiness timeout its own error_code (BLO-33441) #1787 and fix(github-webhook): log the reviewer-wake enqueue (BLO-22758) #1952
  • The benefit is two fewer ways for an unrelated PR to lose a two-hour queue slot

What Changed

  • ui/src/components/AgentActionButtons.test.tsx — render() now does createRoot + root.render inside the suite's act() helper (which wraps flushSync), and is awaited at its three call sites. Previously the first paint happened outside any flush and relied on a later flushReact() — one microtask plus one setTimeout(0) — to catch up.
  • packages/mcp-gateway/src/server.test.ts — the aggregate-gateway timeout goes 150 ms → 1 s, named UPSTREAM_TIMEOUT_MS, and the elapsed assertion becomes proportional to it instead of an absolute 1000 ms.

Verification

Both failures were reproduced locally before being fixed, not inferred.

test CI failure reproduced by
AgentActionButtons:175 expected '' to contain 'Pause' — (see limit below)
server.test.ts:1445 expected [] to deeply equal [ 'alpha__search' ] squeezing the shared budget to 1 ms, which reproduces that assertion verbatim
  • AgentActionButtons.test.tsx — 3/3 pass; tsc -p ui/tsconfig.json surfaces no new errors in the file.
  • packages/mcp-gateway/src/server.test.ts — 52/52 pass.
  • Non-vacuity: with the mcp-gateway fix in place, re-squeezing the budget to 1 ms is still red, so the test has not been made blind. That probe now trips the elapsed bound (expected 64 to be less than 5) before reaching the tools assertion.

Limit worth stating. The AgentActionButtons flake only reproduces under contention, so for that one I removed the scheduling dependency by construction rather than watching a red turn green. The mcp-gateway one I did reproduce directly.

Why the mcp-gateway change does not weaken the test

GatewayState.upstreamTimeoutMs is one global budget, not per-upstream, so the 150 ms meant for the deliberately-hanging stuck upstream was also the healthy alpha's deadline. With failureThreshold: 1, a single slow alpha response opens its breaker for the rest of the test. hanging never answers at all, so a larger budget still trips its breaker — it only costs wall-clock.

Risks

Low — test-only, no source change in either file.

  • AgentActionButtons: the render is now forced synchronous via flushSync. No React warning appears in the run. A future test wanting to assert a pre-paint state could not do so through render(); none does.
  • mcp-gateway: that one test now takes ~1 s longer, since it waits out the larger budget before the breaker opens.

Model Used

Claude Opus 5 (1M context), extended thinking, with tool use.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have run tests locally and they pass
  • I have added or updated tests where applicable — this PR is the test change
  • I have considered and documented any risks above
  • If this change affects the UI, I have included before/after screenshots — n/a, no UI source change
  • I have updated relevant documentation to reflect my changes — n/a

🤖 Generated with Claude Code

…not race

`render()` called `createRoot(container)` and `root.render(...)` outside this
suite's `act()` helper, then relied on a later `flushReact()` to catch up.
`flushReact()` is a single fixed pass — one microtask plus one `setTimeout(0)`
— so the first paint depends on the scheduler getting its turn inside that
one window.

On a CPU-starved runner it does not, and the assertion reads
`container.textContent === ""`:

    AssertionError: expected '' to contain 'Pause'
      ui/src/components/AgentActionButtons.test.tsx:175

That ejected the merge group for #1787 on 2026-09-21 with no defect in the PR
under test. Merge groups were ejecting at 17/33 on that day, and the ejectors
are timing-sensitive assertions across unrelated suites — the signature of
hypervisor CPU steal (BLO-35090), not of any diff.

This suite is the outlier: Inbox, Routines, CompanyInvites and ProjectDetail
all render inside `act` already. The local `act()` wraps `flushSync`, so
moving the render inside it flushes the first paint before `render()` returns
and the assertion no longer races the scheduler. Same class of fix as #1962,
which replaced a wall-clock deadline with an attempt budget.

Honest limit: the flake reproduces only under contention, so this is verified
as "3/3 pass, no new tsc errors, and the scheduling dependency is removed by
construction" — not by reproducing the red locally and watching it go green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Sep 22, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-35090

@kkroo

kkroo commented Sep 22, 2026

Copy link
Copy Markdown
Author

@allyblockcast please review current HEAD of #1987.

Exact HEAD: f0d3916

Report current Critical Issues and Important Issues counts; if clear, include exactly one standalone line:

Reviewed head: f0d3916

Focus: AgentActionButtons render wrapped in act and first-paint race coverage.

…e's 150ms budget

`GatewayState.upstreamTimeoutMs` is a single global budget, not per-upstream,
so the 150 ms this test set to trip the deliberately-hanging `stuck` upstream
was also the deadline for the healthy `alpha`. With `failureThreshold: 1`, one
`alpha` response slower than 150 ms opens its breaker for the rest of the test
and `tools/list` returns `[]`:

    AssertionError: expected [] to deeply equal [ 'alpha__search' ]
      packages/mcp-gateway/src/server.test.ts:1445

That ejected the merge group for #1952 on 2026-09-21. A local in-process
upstream is normally single-digit ms, but 150 ms is not much margin on a
runner losing ~50% of its cycles to hypervisor steal (BLO-35090).

Reproduced rather than inferred: squeezing the budget to 1 ms so the healthy
upstream also misses it produces that exact assertion locally.

`hanging` never answers at all, so raising the budget to 1 s does not weaken
what the test checks — the breaker still opens — it only costs wall-clock.
The `elapsed` assertion becomes proportional to the budget for the same
reason: an absolute 1000 ms bound re-introduces the steal sensitivity being
removed.

Verified: 52/52 in the file. Non-vacuity checked by re-squeezing the budget to
1 ms with the fix in place — still red, so the test has not been made blind.
Note the probe now trips the elapsed bound (`expected 64 to be less than 5`)
before reaching the tools assertion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kkroo kkroo changed the title test(ui): render AgentActionButtons inside act so the first paint cannot race test: de-flake the two assertions ejecting merge groups (#1787, #1952) Sep 22, 2026
@kkroo
kkroo requested a review from allyblockcast September 22, 2026 01:06
@kkroo

kkroo commented Sep 22, 2026

Copy link
Copy Markdown
Author
@allyblockcast please review current HEAD of #1987. Exact HEAD: e91f74b. Report current Critical Issues and Important Issues counts; if clear, include exactly one standalone line: Reviewed head: e91f74b

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. All three ran degraded (nested CLI unavailable in this Kubernetes Job pod); prompts applied directly over the diff and exact-head file reads.
Reviewed head: e91f74b

Test-only, 2 files, +40/-12, no source change. Both fixes are correct and I verified the mechanism behind each rather than taking the description's word for it.

Critical Issues (0)

Important Issues (0)

I checked the one thing that could have made this an Important finding and it does not hold. expect(elapsed).toBeLessThan(UPSTREAM_TIMEOUT_MS * 5) is 5000 ms, and packages/mcp-gateway/vitest.config.ts sets no testTimeout, so under a bare vitest run the 5 s default would abort the test before the bound could ever fail. But CI does not run bare: scripts/run-vitest-stable.mjs:69 passes --testTimeout=30000, leaving 6x headroom. The assertion is enforceable in the environment this PR exists to fix.

Suggestions (2)

  • [comments] packages/mcp-gateway/src/server.test.ts:1429 — the comment tells a future reader that "squeezing this to 1 ms reproduces that exact assertion locally", where "that exact assertion" is the expected [] to deeply equal [ 'alpha__search' ] named six lines above. That was true against the old absolute toBeLessThan(1000) bound, but it is no longer true against the proportional bound this PR introduces: with UPSTREAM_TIMEOUT_MS = 1, the elapsed check becomes < 5 and trips first. The PR description already says so precisely (expected 64 to be less than 5). Someone following the comment will see a different failure and may conclude the repro is stale.
    • Suggest matching the description's wording: the 1 ms probe now trips the elapsed bound before reaching the tools assertion, which is still a valid non-vacuity check, just of a different assertion.
  • [tests] packages/mcp-gateway/src/server.test.ts:1452 — the * 5 multiplier only clears the bare-vitest 5 s default by exactly zero margin, so the bound is unreachable for anyone running pnpm --filter @paperclipai/mcp-gateway test directly. * 3 would keep the same steal-immunity (elapsed should land near 1 s) while staying enforceable locally too. Non-blocking — CI is the lane that matters here.

Strengths

  • The mcp-gateway diagnosis is right, and it is not the obvious one. upstreamTimeoutMs is a single global budget (server.test.ts:358, opts?.timeoutMs ?? 60_000) applied to every upstream, so the 150 ms intended for the deliberately-hung stuck was also healthy alpha's deadline — and with failureThreshold: 1 a single slow alpha response opened its breaker for the rest of the test, yielding exactly the [] the CI failure reported. Raising the budget cannot weaken the test because createHangingUpstream (:255) never calls res.end(), so stuck trips its breaker at any budget.
  • The UI fix removes the race by construction rather than by widening a window. The old root.render() ran outside any flush, landing on DefaultLane and depending on the Scheduler's MessageChannel task beating flushReact's single setTimeout(0) — an ordering that is genuinely not guaranteed under CPU starvation, which matches the observed expected '' to contain 'Pause' (empty textContent, i.e. nothing mounted at all). Inside flushSync the mount lands on SyncLane and is flushed before render() returns. I confirmed the asserted text is reachable in that first synchronous paint: Pause and Clear error come from the agent prop and useMutation state (AgentActionButtons.tsx:93,118,209), with no useQuery fetch gating them.
  • Call-site conversion is complete. All three render( call sites (:148, :164, :183) are awaited; no bare render( remains, which a floating-promise regression would have made silent.
  • Cleanup still holds. Moving root = createRoot(container) inside the act callback keeps afterEach's unmount working, including if the render throws after assignment.
  • Scope is correctly drawn, which I checked rather than assumed. The sibling tight budgets at :1849 (timeoutMs: 200) and :1866 (timeoutMs: 150) drive createGateway, which takes a single upstream — there is no healthy upstream sharing the global budget, so they are not exposed to this failure mode and correctly went untouched. Their elapsed < 3000 bounds sit 15-30x above their budgets.
  • The comments are long for test code and earn it. Both encode the non-obvious causal fact (global-not-per-upstream budget; scheduler lane) whose absence is precisely what would let someone re-tighten these values in six months.
  • Non-vacuity was probed, not assumed — re-squeezing the budget with the fix in place still goes red.

Recommended Action

  1. No blocking changes requested.
  2. Merge once the remaining required CI checks finish green.

@kkroo
kkroo added this pull request to the merge queue Sep 22, 2026
Any commits made after this event will not be merged.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026
@github-actions

Copy link
Copy Markdown

Merge-queue ejection detected for PR #1987. The merge-group run failed and GitHub may have removed the PR from the queue and dropped auto-merge. Inspect the merge-group jobs, fix or rerun the failing checks, then re-enqueue the PR.

Run: https://github.com/Blockcast/paperclip/actions/runs/35903530339

@kkroo
kkroo added this pull request to the merge queue Sep 23, 2026
Any commits made after this event will not be merged.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 26, 2026
@github-actions

Copy link
Copy Markdown

Merge-queue ejection detected for PR #1987. The merge-group run failed and GitHub may have removed the PR from the queue and dropped auto-merge. Inspect the merge-group jobs, fix or rerun the failing checks, then re-enqueue the PR.

Run: https://github.com/Blockcast/paperclip/actions/runs/36219010052

@kkroo
kkroo added this pull request to the merge queue Sep 26, 2026
Any commits made after this event will not be merged.
@kkroo

kkroo commented Sep 26, 2026

Copy link
Copy Markdown
Author

Re-enqueued at e91f74b2 (position 52).

The merge group was ejected at 05:56Z as failed_checks. The only failing test was server/src/__tests__/event-loop-stall-log.test.ts:78 ("expected 0 to be greater than 0") in General tests (server 3/4), running on arc-merge-queue. That is the known starved-runner timing flake. This PR only touches packages/mcp-gateway/src/server.test.ts and ui/src/components/AgentActionButtons.test.tsx. Ally is still 0 Critical / 0 Important at this head.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants