What
Two consecutive full pnpm exec vitest run invocations on master at 9ef47cb each failed one test, and a different one each time:
- run 1:
tests/entrypoints/offscreen.test.ts — "keeps the artifact ZIP when the derived summary exceeds its local limit"
- run 2:
tests/extension/brand-assets.test.ts — "exports store PNGs with required dimensions and SHA"
Both pass when their file is run alone. The failure is Test timed out in 5000ms.
Why it matters here more than in most repos
AGENTS.md requires the suite result to be quoted verbatim before a change is called complete, and the release gates treat a green suite as evidence. A suite that fails a different test on each run makes that evidence unreliable in both directions: a real regression looks like flake, and flake looks like a regression. The instinct to re-run until green is exactly what a release gate should not encourage.
It also costs review rounds. During #199 a full run showed one failure that turned out to be load, and confirming that took a stash, two isolated runs, and a second full run.
Likely cause
Both tests are heavy and browser- or IO-bound: one assembles a ZIP through the offscreen path, the other drives Chromium to rasterise Store assets. vitest.config.ts runs serially, so under a full run they compete with everything before them for the same 5s default.
What a fix needs
- A per-test or per-file timeout for the known-heavy tests, rather than raising
testTimeout globally and hiding a genuine slowdown everywhere else.
- A decision on whether asset rasterisation belongs in the default suite at all, or in the release path that already runs
export-chrome-web-store-assets.mjs.
- Confirmation the chosen fix survives repeated full runs, since a single green run is what produced the current belief that the suite is stable.
Not in scope
No test's assertions should be weakened to make it faster. The two tests are correct; the budget they run under is not.
What
Two consecutive full
pnpm exec vitest runinvocations onmasterat9ef47cbeach failed one test, and a different one each time:tests/entrypoints/offscreen.test.ts— "keeps the artifact ZIP when the derived summary exceeds its local limit"tests/extension/brand-assets.test.ts— "exports store PNGs with required dimensions and SHA"Both pass when their file is run alone. The failure is
Test timed out in 5000ms.Why it matters here more than in most repos
AGENTS.mdrequires the suite result to be quoted verbatim before a change is called complete, and the release gates treat a green suite as evidence. A suite that fails a different test on each run makes that evidence unreliable in both directions: a real regression looks like flake, and flake looks like a regression. The instinct to re-run until green is exactly what a release gate should not encourage.It also costs review rounds. During #199 a full run showed one failure that turned out to be load, and confirming that took a stash, two isolated runs, and a second full run.
Likely cause
Both tests are heavy and browser- or IO-bound: one assembles a ZIP through the offscreen path, the other drives Chromium to rasterise Store assets.
vitest.config.tsruns serially, so under a full run they compete with everything before them for the same 5s default.What a fix needs
testTimeoutglobally and hiding a genuine slowdown everywhere else.export-chrome-web-store-assets.mjs.Not in scope
No test's assertions should be weakened to make it faster. The two tests are correct; the budget they run under is not.