test(e2e): quarantine US-9 so the agent-base build fix can merge (BLO-23198) - #1166
test(e2e): quarantine US-9 so the agent-base build fix can merge (BLO-23198)#1166kkroo wants to merge 1 commit into
Conversation
…-23198)
US-9 "Test-tab bug regressions" fails deterministically: `getByRole(
'button', { name: 'Allow once' })` never renders on /apps/:id/review for
a pending ask_first decision. It failed on the initial attempt AND on
Playwright's retry #1 on the same head (1 failed / 39 passed), so it is
not a flake.
This blocks #1129, the fix for the six-day `Docker (agent base)` build
outage. The merge queue uses grouping_strategy: ALLGREEN and evaluates
mergeability at merge time, so a PR with e2e = FAILURE reaches the front
and is then ejected. Measured: #1129 (e2e FAILURE) was ejected at
03:54:03, the same second its batch-mate #1110 (e2e SUCCESS) merged;
#1132 merged with e2e CANCELLED, so CANCELLED is tolerated and FAILURE
is not.
Quarantine via the mechanism already used for US-6/US-7 rather than a
new one: a "quarantined" status variant plus a gate string, and
test.skip(true, storyById("US-9").gate) as the first statement. The test
still runs and reports as skipped-with-reason, so it stays visible.
Note this is deliberately silencing a test named for the regressions it
exists to catch, and the failure looks product-side (the backend does
produce the pending ask_first request; the UI does not render the
approve control). BLO-23198 tracks un-quarantining and requires
root-causing the UI behaviour plus two consecutive green runs first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🔗 Paperclip issue: BLO-23198 |
1 similar comment
|
🔗 Paperclip issue: BLO-23198 |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Closing — my root cause was wrong, and this change would have been harmful. I claimed US-9 was a deterministic product regression blocking #1129. Direct evidence contradicts that: on #1129's merge-queue branch ( US-9 passes against current master. It fails only on #1129's own head What I got wrong: I diagnosed from the PR's own stale-base check and never looked at the queue-branch job list, which was available the whole time and answers the question directly. I then built a whole causal story ( I no longer know why #1129 was ejected — a fully green batch is not explained by my theory. Reopening that as an open question rather than acting on a story that fits the timeline. Correct next step for #1129 is a rebase onto current master, not a quarantine. BLO-23198 updated; no quarantine is warranted. |
Thinking Path
paperclip #1129 — the fix for the six-day
Docker (agent base)build outage — has been unable to merge. I initially read that as merge-queue depth. It was not.The merge queue's ruleset uses
grouping_strategy: ALLGREEN, and it evaluates mergeability at merge time, not on entry. So #1129 sat at positions 4→2 for hours looking healthy, reached the front twice, and was ejected both times. The measurement:CANCELLEDis tolerated;FAILUREis not. Soe2ered is a hard merge blocker here even though it is not a required status check on the branch — a distinction I had recorded backwards.The single failing test is US-9, and it is deterministic, not flaky: it failed on the initial attempt and on Playwright's retry #1, same head, same assertion (
1 failed / 39 passed).What Changed
Quarantines US-9 using the mechanism already in this file for US-6/US-7 — no new mechanism invented:
mcp-user-stories.catalog.ts: adds a"quarantined"variant toMcpUserStoryStatus; US-9 moves to it with agatestring naming BLO-23198.mcp-user-stories.spec.ts:test.skip(true, storyById("US-9").gate)as the first statement, exactly as US-6/US-7 do.The test still executes and reports as skipped with a reason rather than being deleted or filtered out, so it stays visible in every run.
statusis only ever written, never read by selection logic, and@mcp-runnableis not a CI selector (there is no--grepanywhere) — so the tag is left alone deliberately.Risks
This is the risky half of the change, and it should be reviewed as such. US-9 is named "Test-tab bug regressions". The failure is that
getByRole('button', { name: 'Allow once' })never renders on/apps/:id/reviewfor a pendingask_firstdecision — whilepending.decision === "ask_first"itself passes, so the backend is producing the request and the UI is not rendering the approve control. That may well be a genuine product regression, which would mean this PR silences the test that caught it.I am doing it anyway because Omar explicitly chose to quarantine in order to unblock a six-day build outage. BLO-23198 tracks un-quarantining, with a definition of done that requires root-causing the UI behaviour first and two consecutive green runs before re-enabling.
If you would rather fix US-9 than skip it, say so and I will close this — the outage fix can wait for a correct fix.
Verification
string | undefinedgateargument, which already type-checks in this file today.McpUserStoryStatus(only ever set) and no CI tag-selection depends on@mcp-runnable.Expected effect:
e2egoes green on this branch,mergeStateStatusleavesUNSTABLE, and #1129 can rebase onto master and merge.Model Used
Claude Opus 5 (1M context)