ci: cut the Playwright run from six minutes to three and a half

[skjnldsv]

The component suite is the slowest check on a pull request, at six minutes. Three changes, each one measured before it was kept:

• four shards instead of two
• the shards run in the Playwright image, so there is no browser install step
• the suite gets a second worker, on a runner that has four cores

Six minutes to three and a half, over three runs of the final shape (35199323059): 3m33, 3m27, 3m31, first job start to merged report, against 6m00 on main. The per-shard test step went from 3m28–3m56 to 1m08–1m59.

Warning

Firefox will not start in this image with the runner's default $HOME: the steps run as root while /github/home belongs to the image's pwuser. The workflow sets HOME: /root on the job, which is what the Playwright documentation prescribes. Without it every Firefox test fails to launch.

Important

The image tag is tied to the playwright-core version in the lockfile, 1.62.1, not to @playwright/test, which is already on 1.63.0. A mismatch is not fatal, Playwright downloads whatever the image lacks, but it silently gives back the seconds this buys, so the tag has to move when that version does.

Why four shards, and why that was not enough on its own

Four shards (35193512023) against the last run of main before this branch (35192646502):

	two shards	four shards
workflow, first job to merged report	6m00	5m10
slowest shard	5m17	4m36
Run Playwright tests per shard	3m28, 3m56	1m44, 2m46, 2m14, 1m55
Install Playwright browsers per shard	46s, 48s	44s, 76s, 43s, 45s

The test time halved, the workflow gained 50 seconds of it. Every shard repeats about 75 seconds of npm ci plus browser install, and the total test time across shards went up, 7m24 to 8m39, because each shard pays its own startup. So the next win had to come from the setup, and more shards alone would have made it worse.

Why the image, and why a cache does not work

Of the 46s install step, about 30s was the apt work behind --with-deps and 15s the browser downloads. Four ways of doing it ran side by side on ci/measure-playwright-install, each ending with the same spec on all three engines, so a variant that skips a step still has to prove the browsers launch. Job start to finish, cold cache then warm:

variant	cold	warm	install step, warm	webkit
A, install --with-deps, what main does	1m48	1m49	61s	✅
B, cache ~/.cache/ms-playwright + --with-deps	1m31	1m58	8s restore + 62s install	✅
C, same cache, no apt	1m06	49s	5s restore + 1s install	❌
D, mcr.microsoft.com/playwright image	1m11	1m09	none, 27s container init	✅

The cache is worthless on its own: warm, B restored a gigabyte in 8 seconds and then still spent 62s in install --with-deps, because the apt work runs whether or not the browsers are on disk. It came out slower than the baseline it was meant to beat.

C shows where those 30 seconds go. With the browsers cached the install drops to one second, chromium and firefox launch fine on a bare runner, and webkit dies on libgtk-4.so.1, the gstreamer set, libopus, libgraphene and libevent. The apt half exists for webkit alone, and there is no split to be had: those are its dependencies.

So D. No install step, the pull becomes 26–40s of container setup, and it does not occasionally take 76 seconds the way apt did. Three runs of four shards in the image (35195497636): 4m36, 4m25, 4m32.

Why two workers and not four, or eight shards

playwright.config.ts pinned CI to a single worker. The whole suite in one job, so nothing but the worker count differs:

workers	suite	flaky
1	6m19	1
2	4m49	1
4	4m22	2

Then the shapes that follow, wall clock from first job start to last job end, merge job excluded:

shape	wall	slowest shard step	runner minutes	flaky
2 shards × 2 workers	4m40	3m07	~9	0
4 shards × 1 worker	3m49	2m51	~12	0
4 shards × 2 workers	3m02	2m02	~11	1
8 shards × 1 worker	2m43	1m40	~17	0

Workers scale badly on their own, 24% for the second and 10% for the third and fourth, while separate runners scale well. Using two of each shard's four idle cores lands within 20 seconds of eight shards for six fewer runner minutes, which is what this takes. Eight shards is the faster shape if runner minutes are free, and past that the setup, ~50s of container and npm ci plus ~15s of vite per job, is most of what a shard does.

Flakiness follows the worker count rather than the shard count. The test that flakes in all three final runs is NcDateTimePicker.spec.ts:179 › Pick a date, and it flakes at a single worker too, so it is not new here.

Left on the table

• Each shard rebuilds the component bundle, 14.5s of vite, four times over.
• Running webkit on main and nightly rather than on every pull request would remove a third of the matrix. That is a coverage decision rather than a speed one.
• actions/setup-node with cache: npm would take npm ci from 13–21s into single digits.

Note

.github/workflows/playwright.yml says it comes from the organization template in nextcloud-libraries/.github. These changes make it diverge, and the next template sync will undo them unless the same edits land there.

👾 This pull request was assisted by Claude Code, commits carry an Assisted-by trailer.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.33%. Comparing base (8e9ec54) to head (96e9be5).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8970   +/-   ##
=======================================
  Coverage   58.32%   58.33%           
=======================================
  Files         123      123           
  Lines        4451     4452    +1     
  Branches     1332     1332           
=======================================
+ Hits         2596     2597    +1     
  Misses       1576     1576           
  Partials      279      279           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[susnux]

Instead we can also just utilize the browser installed in github default runner image just like server does.

Also this should not be changed here but in the template repository.

[skjnldsv]

Also this should not be changed here but in the template repository.

Yeah, I plan to upstream it if we approve. But it's best to have real live data 👍

[skjnldsv]

Instead we can also just utilize the browser installed in github default runner image just like server does.

But here we test all three browsers, I'm not against narrowing it down, but that seemed on purpose. Onn server we're only testing against "chrome"