Skip to content

test(e2e): quarantine US-9 so the agent-base build fix can merge (BLO-23198) - #1166

Closed
kkroo wants to merge 1 commit into
masterfrom
test/quarantine-us9
Closed

test(e2e): quarantine US-9 so the agent-base build fix can merge (BLO-23198)#1166
kkroo wants to merge 1 commit into
masterfrom
test/quarantine-us9

Conversation

@kkroo

@kkroo kkroo commented Aug 8, 2026

Copy link
Copy Markdown

Thinking Path

paperclip #1129 — the fix for the six-day Docker (agent base) build outage — has been unable to merge. I initially read that as merge-queue depth. It was not.

The merge queue's ruleset uses grouping_strategy: ALLGREEN, and it evaluates mergeability at merge time, not on entry. So #1129 sat at positions 4→2 for hours looking healthy, reached the front twice, and was ejected both times. The measurement:

PR e2e outcome
#1110 SUCCESS merged 03:54:03
#1132 CANCELLED merged
#1129 FAILURE ejected 03:54:03 — the same second its batch-mate merged

CANCELLED is tolerated; FAILURE is not. So e2e red is a hard merge blocker here even though it is not a required status check on the branch — a distinction I had recorded backwards.

The single failing test is US-9, and it is deterministic, not flaky: it failed on the initial attempt and on Playwright's retry #1, same head, same assertion (1 failed / 39 passed).

What Changed

Quarantines US-9 using the mechanism already in this file for US-6/US-7 — no new mechanism invented:

  • mcp-user-stories.catalog.ts: adds a "quarantined" variant to McpUserStoryStatus; US-9 moves to it with a gate string naming BLO-23198.
  • mcp-user-stories.spec.ts: test.skip(true, storyById("US-9").gate) as the first statement, exactly as US-6/US-7 do.

The test still executes and reports as skipped with a reason rather than being deleted or filtered out, so it stays visible in every run. status is only ever written, never read by selection logic, and @mcp-runnable is not a CI selector (there is no --grep anywhere) — so the tag is left alone deliberately.

Risks

This is the risky half of the change, and it should be reviewed as such. US-9 is named "Test-tab bug regressions". The failure is that getByRole('button', { name: 'Allow once' }) never renders on /apps/:id/review for a pending ask_first decision — while pending.decision === "ask_first" itself passes, so the backend is producing the request and the UI is not rendering the approve control. That may well be a genuine product regression, which would mean this PR silences the test that caught it.

I am doing it anyway because Omar explicitly chose to quarantine in order to unblock a six-day build outage. BLO-23198 tracks un-quarantining, with a definition of done that requires root-causing the UI behaviour first and two consecutive green runs before re-enabling.

If you would rather fix US-9 than skip it, say so and I will close this — the outage fix can wait for a correct fix.

Verification

  • Diff is 2 files, +4/−2; no product code touched.
  • Pattern matches the existing US-6/US-7 gating verbatim, including the string | undefined gate argument, which already type-checks in this file today.
  • Confirmed no consumer reads McpUserStoryStatus (only ever set) and no CI tag-selection depends on @mcp-runnable.

Expected effect: e2e goes green on this branch, mergeStateStatus leaves UNSTABLE, and #1129 can rebase onto master and merge.

Model Used

Claude Opus 5 (1M context)

…-23198)

US-9 "Test-tab bug regressions" fails deterministically: `getByRole(
'button', { name: 'Allow once' })` never renders on /apps/:id/review for
a pending ask_first decision. It failed on the initial attempt AND on
Playwright's retry #1 on the same head (1 failed / 39 passed), so it is
not a flake.

This blocks #1129, the fix for the six-day `Docker (agent base)` build
outage. The merge queue uses grouping_strategy: ALLGREEN and evaluates
mergeability at merge time, so a PR with e2e = FAILURE reaches the front
and is then ejected. Measured: #1129 (e2e FAILURE) was ejected at
03:54:03, the same second its batch-mate #1110 (e2e SUCCESS) merged;
#1132 merged with e2e CANCELLED, so CANCELLED is tolerated and FAILURE
is not.

Quarantine via the mechanism already used for US-6/US-7 rather than a
new one: a "quarantined" status variant plus a gate string, and
test.skip(true, storyById("US-9").gate) as the first statement. The test
still runs and reports as skipped-with-reason, so it stays visible.

Note this is deliberately silencing a test named for the regressions it
exists to catch, and the failure looks product-side (the backend does
produce the pending ask_first request; the UI does not render the
approve control). BLO-23198 tracks un-quarantining and requires
root-causing the UI behaviour plus two consecutive green runs first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 8, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-23198

1 similar comment
@allyblockcast

allyblockcast Bot commented Aug 8, 2026

Copy link
Copy Markdown

🔗 Paperclip issue: BLO-23198

@cursor

cursor Bot commented Aug 8, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@kkroo

kkroo commented Aug 8, 2026

Copy link
Copy Markdown
Author

Closing — my root cause was wrong, and this change would have been harmful.

I claimed US-9 was a deterministic product regression blocking #1129. Direct evidence contradicts that: on #1129's merge-queue branch (gh-readonly-queue/master/pr-1129-92ee9613…, run 31235301841) — which is current master plus #1129 — the job list reads:

e2e success
verify success

US-9 passes against current master. It fails only on #1129's own head 1e9b9dc, which is branched from a much older master. So US-9 was evidently already fixed upstream, and this PR would have quarantined a test that works today — silencing exactly the regression coverage it exists to provide, for no benefit.

What I got wrong: I diagnosed from the PR's own stale-base check and never looked at the queue-branch job list, which was available the whole time and answers the question directly. I then built a whole causal story (grouping_strategy: ALLGREEN ejects e2e-FAILURE PRs) that happens to fit the timing but is contradicted by the queue branch being fully green.

I no longer know why #1129 was ejected — a fully green batch is not explained by my theory. Reopening that as an open question rather than acting on a story that fits the timeline.

Correct next step for #1129 is a rebase onto current master, not a quarantine. BLO-23198 updated; no quarantine is warranted.

@kkroo kkroo closed this Aug 8, 2026
@kkroo
kkroo deleted the test/quarantine-us9 branch August 8, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant