You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: test what a change reaches, not everything on every push #224
Every pull request and every push to main runs the whole of CI, whatever it changed: every package's tests, the player's browser suite in Chromium, Firefox and WebKit, and the VS Code extension in a real editor. That is about 20 runner-minutes and 7 minutes of waiting per push, for a change to one package or to the docs alike.
Measured on the runs of #221 and #223 (both author-only changes):
The test job took 6.2 to 7.3 minutes, 5.6 of them in pnpm test; Firefox 3.4, WebKit 2.8 to 3.8, VS Code 1.1 to 1.6.
73 of the 105 test files run in under a second. The time is in a few end-to-end files. Three of the player's browser files wait out real time in each engine:
File
Chromium
Firefox
stalling-host (a host silent for longer than the 30 s stall bound)
58 s
59 s
large-media (20 MB archives)
31 s
33 s
worker-update (a worker update's deadlines; flaky under load)
26 s
28 s
A change to packages/author ran the player's suite three times, the verifier's, the embed package's and the extension's, none of which it can affect.
Deleting tests would save little: the slow files guard behaviour the player must keep, and the 41 test titles that name a past issue describe behaviour, not a bug.
Proposal:
A change to design/, docs/ or a root .md alone starts no run.
A pull request tests the packages it changes and those that depend on them (pnpm's ...[<commit>]). A change under skills/ adds the author package and apps/skills. Any other file outside the packages (the lockfile, a workflow, the root's package.json) tests everything.
Firefox and WebKit run only when the player, the runtime or the demo changed; the extension's job only when the extension or what it carries changed.
A pull request's run leaves out the three files above. A push to main tests what it changed with every file, and a weekly run tests everything.
Activity
docs: say exactly what each CI job runs, LICENSE among the docs (#224)