Skip to content

ci: shard the browser e2e suites across runners - #642

Merged
Rinat S (rsolmano) merged 4 commits into
mainfrom
ci-speedup-review
Oct 6, 2026
Merged

Rinat S (rsolmano) merged 4 commits into
mainfrom
ci-speedup-review

Conversation

@rsolmano

Copy link
Copy Markdown
Collaborator

Problem

CI takes ~22 min on every PR and merge-queue entry (measured over the last successful runs, e.g. run 37349114307). Almost all of it is Playwright execution in three jobs, which run the same ~464 no-agent specs against three hosts:

Check Wall time Of which test step
Desktop smoke & e2e 22.3 min 1215 s, 1 worker
Binary smoke & e2e 20.0 min 1126 s, 1 worker
E2E (no-agent) 13.6 min 755 s, 2 lanes (4-vCPU runner)

Setup (checkout, install, Playwright, builds) is only ~1–2 min per job.

Approach

Split each browser suite across several runners and keep the ruleset's required checks stable through aggregate gate jobs.

  • Source suite: 2 jobs. Each runs one slice via THINKRAIL_E2E_JOB_SHARD=k/N, and its local lanes subdivide that slice into global Playwright shards (k-1)·L+i of N·L, so the jobs together cover exactly the whole suite.
  • Binary and desktop suites: 4 jobs each, using Playwright's --shard=k/4 (the runners already forward argv). The artifact configs now distribute individual tests (fullyParallel: true, still workers: 1), so the split is even (116 × 4) instead of file-sized.
  • Gates: the ruleset requires E2E (no-agent), Binary smoke & e2e and Desktop smoke & e2e. Those names now belong to small gate jobs (needs: the matrix, if: !cancelled()) that pass only when every shard succeeded, so no ruleset change is needed.

Trade-off: wall time drops from ~22 to ~7–8 min (estimated from the step timings above), while total runner time grows ~20% (~62 → ~74 min per run), because each shard repeats the ~1–1.5 min setup. Running the full artifact suites only in the merge queue was considered and left out: with sharding it no longer shortens PR wall time; it would only save runner minutes.

Changes

  • .github/workflows/ci.yml: e2e-shard (2), binary-shard (4) and desktop-shard (4) matrices with fail-fast: false and per-shard report artifact names, plus the three gate jobs under the existing required names.
  • e2e/shardPlan.ts, e2e/run.ts: parseJobShard / laneShardArgs. The runner applies the job slice in both serial and lane modes, and rejects THINKRAIL_E2E_JOB_SHARD combined with an explicit --shard.
  • playwright.config.ts: fullyParallel is also on when a job slice is set.
  • e2e/artifactPlaywright.ts: fullyParallel: true for binary and desktop. No spec uses beforeAll/afterAll/describe.serial, and the source lanes already run the same specs split per test.
  • e2e/SPEC.md, .github/SPEC.md: job-slice contract, per-test artifact sharding, and the gate-name ↔ ruleset coupling.

Screenshots

Not applicable — no frontend changes.

Related issues

None.

Checklist

  • Fast gates pass: bun run lint, bun run typecheck, bun run test
  • E2E suite passes for app-affecting changes (bun run e2e, or bun run e2e:full when touching agent behavior) — not app-affecting (test-harness + CI only); targeted sharded runs below, and this PR's own CI runs every shard
  • Before/after screenshots are included for frontend changes, or marked not applicable
  • Relevant SPEC.md / top-level specs updated to reflect any boundary, contract, or behavior change
  • I have read the Contributing guide and agree to the Code of Conduct

Testing

  • bun run lint → clean
  • bun run typecheck → 18/18 successful
  • bun run test → 18/18 tasks, 0 fail
  • bun run check:deps / check:boundaries / check:seams / check:spec-surface → OK
  • playwright test e2e/shard-runner.spec.ts → 8 passed (incl. new job-slice test)
  • bun run e2e --list → 470; THINKRAIL_E2E_JOB_SHARD=1/2 and 2/2 → 235 each; combined with --shard=1/2 → rejected (exit 1)
  • playwright test -c playwright.{binary,desktop}.config.ts --list --shard=k/4 → 116 each of 464
  • bun run build:binary && bun run e2e:binary --shard=1/4 → 116 passed (4.8 min)
  • THINKRAIL_E2E_JOB_SHARD=2/2 bun run e2e → 232 passed (8 local lanes → global shards 9–16 of 16)
  • @action-validator/cli .github/workflows/ci.yml → valid
  • Desktop shard not run locally (needs Linux + Xvfb); this PR's CI exercises all four.

@jetbrains-air jetbrains-air Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — ready to merge.

@rsolmano
Rinat S (rsolmano) added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit ac130a0 Oct 6, 2026
34 of 51 checks passed
@rsolmano
Rinat S (rsolmano) deleted the ci-speedup-review branch October 6, 2026 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant