Skip to content

ci: run CI on GitHub-hosted Linux x86_64 and macOS arm64 runners - #213

Merged
schenksj merged 1 commit into
mainfrom
feature/ci-hosted-legs
Oct 14, 2026
Merged

schenksj merged 1 commit into
mainfrom
feature/ci-hosted-legs

Conversation

@rbrenton

@rbrenton rbrenton commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Why

  • CI here runs on self-hosted runners. The Linux x86_64 job has been unable to run for some time, so the platform the published native library targets is not being tested, and every PR carries a job that never completes.
  • The measurement in ci: measure the build and tests on standard GitHub-hosted runners (measurement only, do not merge yet) #211 showed the build and both test suites fit on the free standard hosted runners: ubuntu-24.04 in 25 minutes and macos-15 in 38 minutes, cold. The self-hosted runners are being retired for this repository.

What changes (.github/workflows/ci.yml only)

Every event (pull requests, pushes to main, workflow_dispatch) runs:

Job Runner Steps
Build & Test (Linux x86_64) ubuntu-24.04 setup.sh, make compile, make test, Rust tests
Build & Test (macOS ARM64) macos-15 same
ci-ok ubuntu-24.04 passes only if both jobs report success
  • The self-hosted jobs are removed, along with the workflow's use of the test-runners environment.
  • The macOS job keeps its exact name, so the existing required status check keeps being satisfied and this can merge without a settings change. ci-ok is added alongside so the required check can be switched to it afterwards.
  • One step is advisory: the Linux Rust test run. parquet_companion::indexing::tests::test_multi_segment_docid_mapping_correctness fails intermittently on Linux (it failed in ci: measure the build and tests on standard GitHub-hosted runners (measurement only, do not merge yet) #211). The step reports a warning annotation and a job-summary note when it is red, but does not fail the job. Linux build, Linux Java tests and compiling the Rust tests all still block. Making it blocking again is a one-line change (remove continue-on-error).
  • Caching: cargo registry/git sources and dependency build outputs, keyed exactly on runner, native toolchain fingerprint and Cargo.lock/Cargo.toml. This crate's own outputs are removed before saving. Only runs on main save; pull requests only restore. Expected saving is roughly 25 to about 14 minutes on Linux and 38 to about 20 on macOS, inferred from the build logs rather than measured.
  • To let the cache survive mvn clean, the release build is directed into native/target (the directory cargo test already uses) through MAVEN_ARGS. The Makefile, pom and scripts are unchanged.
  • Housekeeping: timeout-minutes on every job, permissions: contents: read, no secrets, superseded PR runs are cancelled while runs on main never are.

Things to know

  • Until the required check is switched to ci-ok, only the macOS job gates merges. A red Linux build or Linux Java test shows up but does not block.
  • The macOS Rust tests are blocking and include the test that is flaky on Linux. It passed on hosted macOS in ci: measure the build and tests on standard GitHub-hosted runners (measurement only, do not merge yet) #211.
  • Open PRs will need a new run after this merges (update branch or push); their current runs are waiting on self-hosted runners.
  • The local branch fix/linux-ci-test-failures adds diagnostics for the flaky test and edits the same lines of ci.yml, so it will need a rebase.

Verification

  • ci-ok and the advisory logic were checked offline for every result combination; the scripts were run locally under bash -eo pipefail.
  • This PR's own run exercises both legs cold, the target-directory override, a cache miss, the advisory path if the Linux test fails, ci-ok, and the required check name.
  • Not exercised until after merge: saving the cache (first run on main) and a cache hit (first PR after that).

Follow-ups (not in this PR)

  • Switch the required status check on main to ci-ok.
  • Fix the multi-segment test on Linux, then remove the advisory line.

CI ran on two self-hosted runners. The Linux x86_64 job was not running,
so that platform had no coverage, and the availability of CI depended on
those two machines. Both jobs now run on standard hosted runners
(ubuntu-24.04 and macos-15) for every pull request, push to main and
manual run, with the same steps as before: setup.sh, make compile, make
test and the Rust unit tests. A cold run was measured at about 25
minutes on Linux and 38 minutes on macOS, with all 1313 Java tests
passing on both.

What changes besides the runner labels:

- The macOS job keeps the name "Build & Test (macOS ARM64)", which is
  the status check currently required on main, so the requirement stays
  satisfied. A `ci-ok` job is added that passes only when both legs
  succeeded; it is meant to become the single required check.

- The Rust tests on Linux are advisory for now. One test,
  test_multi_segment_docid_mapping_correctness, fails on the hosted
  Linux runner with a file slice bounds error. The step reports the
  failure as a warning annotation and in the job summary, repeated by
  ci-ok, without failing the leg. The Linux build, the Linux Java tests
  and the build of the Rust tests remain blocking. Removing the single
  `continue-on-error: true` line makes the step blocking again.

- Third-party cargo dependencies and their build outputs are cached.
  A cold run compiles about 650 crates twice, once for the release
  library and once in the dev profile for cargo test. The entry is
  keyed on runner OS and architecture, a fingerprint of the Rust
  toolchain, C compiler, protoc and RUSTFLAGS, and the hash of
  Cargo.lock, Cargo.toml and .cargo/config.toml, and is restored on an
  exact match only. Everything built from this repository's own crate
  is removed before saving, so it is always compiled from the checked
  out sources. Entries are saved only by runs on main; pull requests
  only read them.

- For the cache to survive `mvn clean`, the workflow points the pom's
  rust.target.dir at native/target through MAVEN_ARGS. The Makefile,
  the pom and the scripts are unchanged.

- cargo test is split into a build step and a run step, so that a build
  failure blocks on both platforms and only the test result is advisory
  on Linux. Parallelism is what was measured to fit: 4 cargo jobs on
  Linux, 3 for the release build and 2 for the test build on macOS.

The workflow uses no secrets and no environments, has read-only
permissions and a timeout on every job, and cancels superseded runs on
pull requests only.
@claude

claude Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Claude encountered an error —— View job


I'll analyze this and get back to you.

@schenksj schenksj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@schenksj
schenksj merged commit da75884 into main Oct 14, 2026
3 of 4 checks passed
@schenksj
schenksj deleted the feature/ci-hosted-legs branch October 14, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants