Skip to content

CI: test what a change reaches, not everything on every push #224

Closed

Description

@alekswebnet

Every pull request and every push to main runs the whole of CI, whatever it changed: every package's tests, the player's browser suite in Chromium, Firefox and WebKit, and the VS Code extension in a real editor. That is about 20 runner-minutes and 7 minutes of waiting per push, for a change to one package or to the docs alike.

Measured on the runs of #221 and #223 (both author-only changes):

  • The test job took 6.2 to 7.3 minutes, 5.6 of them in pnpm test; Firefox 3.4, WebKit 2.8 to 3.8, VS Code 1.1 to 1.6.

  • 73 of the 105 test files run in under a second. The time is in a few end-to-end files. Three of the player's browser files wait out real time in each engine:

    File Chromium Firefox
    stalling-host (a host silent for longer than the 30 s stall bound) 58 s 59 s
    large-media (20 MB archives) 31 s 33 s
    worker-update (a worker update's deadlines; flaky under load) 26 s 28 s
  • A change to packages/author ran the player's suite three times, the verifier's, the embed package's and the extension's, none of which it can affect.

Deleting tests would save little: the slow files guard behaviour the player must keep, and the 41 test titles that name a past issue describe behaviour, not a bug.

Proposal:

  1. A change to design/, docs/ or a root .md alone starts no run.
  2. A pull request tests the packages it changes and those that depend on them (pnpm's ...[<commit>]). A change under skills/ adds the author package and apps/skills. Any other file outside the packages (the lockfile, a workflow, the root's package.json) tests everything.
  3. Firefox and WebKit run only when the player, the runtime or the demo changed; the extension's job only when the extension or what it carries changed.
  4. A pull request's run leaves out the three files above. A push to main tests what it changed with every file, and a weekly run tests everything.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions