Skip to content

ci: test what a change reaches, not everything on every push - #225

Merged
alekswebnet merged 2 commits into
mainfrom
ci/test-what-changed
Nov 17, 2026
Merged

alekswebnet merged 2 commits into
mainfrom
ci/test-what-changed

Conversation

@alekswebnet

@alekswebnet alekswebnet commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #224

CI now tests what a change reaches instead of everything on every push.

What a run tests

Event Packages Firefox and WebKit Slow browser files
Pull request the ones it changes and their dependents when the player, the runtime or the demo changed left out
Push to main the same, from where main was the same included
Weekly (Mon 05:00 UTC), by hand all yes included
Only design/, docs/, root .md, LICENSE no run no
  • A new scope job reads the change: pnpm's ...[<commit>] filter for packages and apps, plus the author package and apps/skills for skills/. Any other file outside the packages that is not docs (the lockfile, a workflow, the root's package.json) tests everything.
  • pnpm typecheck still covers every package: each package's check writes the types the next one reads, and it takes seconds.
  • The slow files are stalling-host, worker-update and large-media, listed in the player's vite.config.ts and left out under H5P_TEST_SKIP_SLOW. They take 115 s of test time in Chromium and 120 s in Firefox. Main's run after the merge has them, so design/testing.md now says to wait for it before publishing the player.
  • The VS Code job runs when the extension, the player, the runtime or the libraries changed.
  • to-scorm is not among the slow files: it runs on every pull request that tests apps/skills, being the one check of the h5p-to-scorm skill's text against the embed README.

Testing

  • The scope step, simulated on a fresh shallow clone with a fake commit per kind of change, run as a pull request's run:

    Change Tested Engines VS Code
    packages/author author, demo, skills no no
    skills/h5p-author/SKILL.md author, skills no no
    design/author.md plus packages/author author, demo, skills no no
    packages/player player, verify, embed, author, extension, demo, skills yes yes
    packages/runtime the same, and the runtime yes yes
    apps/demo page demo the app's tests only no
    packages/embed embed, demo, skills no no
    packages/verify verify, author, demo, skills no no
    packages/cmi5 cmi5, demo no no
    packages/vscode the extension no yes
    pnpm-lock.yaml, another workflow everything yes yes

    It found one bug, fixed before this commit: pnpm counts a file outside the packages as the root project's, and the root's test script tests everything. The step now leaves the root out.

  • vitest list --project browser: 36 files, and 33 with H5P_TEST_SKIP_SLOW=1; the three named are the ones missing.

  • The player's typecheck passes. The workflow parses, with each job's needs and if as intended.

Not verified

  • This pull request's own run tests everything, because it changes a workflow, and leaves out the three slow files, as any pull request's run does. Selecting fewer packages shows up first on the next pull request after the merge.
  • A push to main fetches the commit main was at, to compare with. GitHub serves a fetch by commit; a forced push falls back to testing everything.

Release

None: no package's published files change.


Summary by cubic

Closes #224: CI now tests what a change reaches instead of everything on every push. A new scope job picks the packages a change touches and their dependents via pnpm's ...[<commit>] filter. Pull requests test only those, skip the player's three slow browser files, and run Firefox and WebKit only when the player, runtime, or demo changed. Pushes to main run the same selection including the slow files; weekly and manual runs test everything. Docs-only changes (design/, docs/, root markdown, LICENSE) start no run.

  • pnpm typecheck still covers every package, in seconds.
  • The VS Code job runs when the extension or its dependencies changed.
  • Non-doc files outside the packages (lockfile, workflows, root package.json) test everything.

Testing

  • Scope selection verified on a fresh clone with a fake commit per change kind; one bug found and fixed: pnpm counts such files as the root project, whose test script tests everything, so the step now excludes the root.
  • This PR's run tests everything (it changes a workflow) and skips the slow files as any PR does.

Release

  • No published files change.

Written for commit 139b9b5. Summary will update on new commits.

View guided diff Turn on auto-fix

A pull request tests the packages it changes and those that depend on them, as
pnpm's ...[<commit>] filter reads the change; skills/ adds the author package
and apps/skills, and any other file outside the packages that is not docs tests
everything. Firefox and WebKit run when the player, the runtime or the demo
changed, the VS Code job when the extension or what it carries did. A pull
request's run leaves out the player's three browser files that wait out real
time (H5P_TEST_SKIP_SLOW); a push to main tests what it changed with every
file, and a weekly run tests everything. A change to the docs alone starts no
run.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread CONTRIBUTING.md Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread design/testing.md Resolved
Comment thread CONTRIBUTING.md Resolved
Comment thread design/testing.md Resolved
Comment thread design/testing.md Resolved
The engines job runs when the player, the runtime or the demo changed, the
app's tests every time and the player's suite only for the first two. A change
outside the packages tests everything unless it is docs, LICENSE included, and
typecheck runs in every package that has the script.
@alekswebnet
alekswebnet merged commit 65b50bd into main Nov 17, 2026
10 checks passed
@alekswebnet
alekswebnet deleted the ci/test-what-changed branch November 18, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: test what a change reaches, not everything on every push

1 participant