Skip to content

Full-suite runs time out on the heaviest tests while passing in isolation #200

Description

@lamemustafa

What

Two consecutive full pnpm exec vitest run invocations on master at 9ef47cb each failed one test, and a different one each time:

  • run 1: tests/entrypoints/offscreen.test.ts — "keeps the artifact ZIP when the derived summary exceeds its local limit"
  • run 2: tests/extension/brand-assets.test.ts — "exports store PNGs with required dimensions and SHA"

Both pass when their file is run alone. The failure is Test timed out in 5000ms.

Why it matters here more than in most repos

AGENTS.md requires the suite result to be quoted verbatim before a change is called complete, and the release gates treat a green suite as evidence. A suite that fails a different test on each run makes that evidence unreliable in both directions: a real regression looks like flake, and flake looks like a regression. The instinct to re-run until green is exactly what a release gate should not encourage.

It also costs review rounds. During #199 a full run showed one failure that turned out to be load, and confirming that took a stash, two isolated runs, and a second full run.

Likely cause

Both tests are heavy and browser- or IO-bound: one assembles a ZIP through the offscreen path, the other drives Chromium to rasterise Store assets. vitest.config.ts runs serially, so under a full run they compete with everything before them for the same 5s default.

What a fix needs

  • A per-test or per-file timeout for the known-heavy tests, rather than raising testTimeout globally and hiding a genuine slowdown everywhere else.
  • A decision on whether asset rasterisation belongs in the default suite at all, or in the release path that already runs export-chrome-web-store-assets.mjs.
  • Confirmation the chosen fix survives repeated full runs, since a single green run is what produced the current belief that the suite is stable.

Not in scope

No test's assertions should be weakened to make it faster. The two tests are correct; the budget they run under is not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions