Skip to content

Record how long CI takes, so a change in it is visible - #838

Merged
Rafael-SOWNet merged 1 commit into
masterfrom
ci-timing
Aug 9, 2026
Merged

Record how long CI takes, so a change in it is visible#838
Rafael-SOWNet merged 1 commit into
masterfrom
ci-timing

Conversation

@Rafael-SOWNet

Copy link
Copy Markdown
Collaborator

There was no way to answer "has CI got slower?". The web UI shows one run at a time and reports created-to-finished, which folds in how long the run waited for a runner — on a busy afternoon that reads as a regression when nothing about the code changed.

It also gets harder the longer it goes unasked: run history is kept for a bounded window and logs for ninety days. The January 2026 logs had already expired when this was written; only the job timestamps survived to be recovered.

Usage

cd Sources/Utils/Utils
dotnet run CiTiming                      # measure, and compare to the baseline
CI_TIMING_RECORD=1 dotnet run CiTiming   # record a new snapshot, keeping the old ones

gh does the talking, so no token lives in this repository.

Two durations, because they answer different questions

  • exec — runner picks the job up → job finishes. This is what moves when a suite grows.
  • queue — run created → runner picks it up. This moves when several PRs are in flight, and says nothing about the code.

What it says today

job exec vs 2026-01-22
C# Test / Test (windows-latest) 13.5m +52%
C# Test / Test (macos-latest) 10.5m +15%
C# Test / Test (ubuntu-latest) 10.0m +56%

Test cases over the same period went 3129 → 5118 (+64%), so the growth is sublinear — time per case fell about a fifth. macOS looks better than its neighbours because those runners moved to ARM, not because less is run there.

Aggregate: 76 minutes of runner time per commit; the slowest single job, which is what a contributor actually waits for, is 13.5m.

Nothing structural to reclaim

I went looking for waste and did not find much worth acting on — recording the negatives so nobody re-derives them:

  • The Build Tests step is 0.2–0.9m, not a hidden second build.
  • Debug vs Release makes no difference to the suite: 5m04s vs 5m10s locally.
  • Coverage collection costs +66s, on the Windows job only, and that one is uploaded.
  • parallelizeAssembly: false is a no-op with a single test assembly; collections already run in parallel.

The one genuinely dead setting: ubuntu and macOS pass /p:CollectCoverage=true, which produces a report that every upload step skips (if: windows-latest). It costs about a second, so this PR leaves it alone rather than touching the workflow.

Baseline is committed, and accumulates

Two snapshots are on file — today, and 22 January 2026 recovered before that window closed. Recording a new one appends the old to history rather than replacing it, so a slow drift over many small steps stays visible.

There was no way to answer "has CI got slower". The web UI shows one run at a time
and reports created-to-finished, which folds in how long the run waited for a
runner; on a busy afternoon that reads as a regression when nothing about the code
changed. Run history is kept for a bounded window and logs for ninety days, so the
question also gets harder to answer the longer it goes unasked -- the January 2026
logs had already expired when this was written, and only the job timestamps
survived to be recovered.

So: dotnet run CiTiming, from Sources/Utils/Utils.

It separates the two durations, because they answer different questions. Exec is
runner-picks-the-job-up to job-finishes, which is what moves when a suite grows.
Queue is run-created to runner-picks-it-up, which moves when several pull requests
are in flight and says nothing about the code.

The baseline is committed rather than derived, for the retention reason above, and
every earlier snapshot is kept when a new one is recorded. Two are on file: today,
and 22 January 2026, recovered from the job timestamps before that window closes.

What it says today, against January:

    C# Test / Test (ubuntu-latest)     10.0m   +56%
    C# Test / Test (windows-latest)    13.5m   +52%
    C# Test / Test (macos-latest)      10.5m   +15%

Against 3129 to 5118 test cases over the same period, +64%. So the growth is
sublinear -- time per case fell about a fifth -- and macOS looks better than its
neighbours because those runners moved to ARM, not because less is run there.

Nothing structural was found to reclaim. The build steps are under a minute, and
the test step is the job.

gh does the talking, so no token lives in this repository.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Rafael-SOWNet
Rafael-SOWNet merged commit 9ae9a84 into master Aug 9, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant