Skip to content

ci: measure the build and tests on standard GitHub-hosted runners (measurement only, do not merge yet) - #211

Closed
rbrenton wants to merge 1 commit into
mainfrom
feature/ci-hosted-runner-measurement
Closed

rbrenton wants to merge 1 commit into
mainfrom
feature/ci-hosted-runner-measurement

Conversation

@rbrenton

@rbrenton rbrenton commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Why

CI here runs on self-hosted runners, and the Linux x86_64 job has not been able to run for some time, so the platform the published native library targets is not being tested. Before deciding how CI should be split between GitHub-hosted and self-hosted runners we need real numbers: do the cold native build, the Java tests and the Rust tests fit on the free standard hosted runners, and how long does each take?

  • ubuntu-24.04: 4 vCPU / 16 GB RAM
  • macos-15 (arm64): 3 vCPU / 7 GB RAM

The same measurement was done for indextables_spark (indextables/indextables_spark#406) with the same helper, so the numbers are comparable.

What this adds

One new workflow, .github/workflows/ci-hosted-measure.yml. Nothing else changes; ci.yml, the scripts, the Makefile, the pom and the native crate are untouched.

  • Runs the same steps as ci.yml on both runners, cold: ./scripts/setup.sh, make compile, make test, cargo test --lib --tests.
  • Writes a table to each job's summary: wall time per phase, free disk before / lowest during / after, peak memory and swap (including Rust toolchain memory), the parallelism used, surefire and cargo test totals, toolchain versions and build directory sizes.
  • Uploads the raw measurement data, and surefire reports on failure.

Things to know when reading the results

  • It is not CI. The workflow and job names say "measure". A red leg means "does not fit or does not pass there yet".
  • Triggers: only workflow_dispatch and pull requests that touch this one file.
  • Phases are measured independently where possible. The Rust tests still run if the Java tests fail, so one red phase does not hide the other numbers.
  • macOS parallelism: the release build uses cargo's default (3 jobs), which fit in 7 GB when indextables_spark built this crate on the same runner. The dev-profile build behind cargo test has not been measured on 7 GB, so it starts at 2 jobs. Both are workflow_dispatch inputs.
  • No secrets, contents: read only, standard runner labels only.

Not verified before opening

The workflow has not run anywhere yet; this PR is the first run. The helper was exercised locally on macOS, and its Linux code paths only by inspection. Linux test or link failures are possible, since Linux has not been exercised in CI recently; they would be a finding rather than a problem with this PR.

The existing CI workflow will also trigger on this PR and wait for runner approval as usual. It does not need to run for this measurement.

We have no numbers for how this repository's CI steps behave on GitHub's
standard hosted runners: whether the cold cargo release build, the Java
suite and the Rust unit tests fit in the disk and memory of ubuntu-24.04
(x86_64, 4 vCPU / 16 GB) and macos-15 (arm64, 3 vCPU / 7 GB), and how
long each step takes there. This adds a separate workflow that runs the
same steps as ci.yml (setup.sh, make compile, make test, cargo test) on
those two labels and reports, per leg, the wall time of each phase, free
disk before / lowest during / after, peak memory and swap, the
parallelism used against the cores and RAM reported, test counts as
surefire and cargo print them, toolchain versions and the size of the
build directories.

It is measurement only: it runs on workflow_dispatch and on pull
requests that touch the workflow file itself, uses no secrets or
environments, and is named so it cannot be read as the CI result.
ci.yml, the scripts, the Makefile, the pom and the native crate are
unchanged. Runs are cold (no caching) so the numbers are the worst case.

The timing helper is the one already used for the same measurement in
indextables_spark, so results are comparable. Two things differ because
this build needs them: the phase log is timestamped by perl rather than
a bash read loop, because surefire prints all test output to the console
and the loop would throttle the tests being timed; and the sampler also
records Rust toolchain memory, since the native build is the heavy part.

Parallelism: the release build keeps cargo's default of one job per
core, which has been measured to fit on macos-15 for this crate. The
dev-profile build behind cargo test has not been measured on 7 GB, so it
defaults to 2 jobs on macOS; both can be overridden per run. The Rust
tests run even if the Java tests fail so that one run measures both.
@claude

claude Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Claude encountered an error —— View job


I'll analyze this and get back to you.

@rbrenton

rbrenton commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the measurement is complete (Linux 25 min, macOS 38 min, cold). Hosted CI itself is in #213.

@rbrenton rbrenton closed this Oct 8, 2026

This branch had an error being deployed

1 failed deployment
test-runners — 9a5309eb Deployed Oct 6, 2026 by rbrenton via Build & Test (macOS ARM64) #125
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant