Run benchmarks on CodSpeed in CI#13
Open
kinto0 wants to merge 8 commits into
Open
Conversation
Merging this PR will not alter performance
Performance Changes
Comparing 
Footnotes |
Add a CodSpeed GitHub Actions workflow so the landed criterion benchmarks run continuously: instruction-count (simulation) mode for the deterministic microbenchmarks, walltime mode for the heavy real-world PyTorch benchmarks (which must not run under Valgrind). Swap the `criterion` dev-dependency to `codspeed-criterion-compat` (aliased back to `criterion`) so the existing bench harness emits CodSpeed measurements under `cargo codspeed` while behaving like plain criterion otherwise, and add a `[profile.bench]` that drops LTO/single-codegen-unit to keep the CI build within runner memory limits.
configure() otherwise spawns python3 subprocesses to probe the interpreter and site-packages during state init. Under CodSpeed's instrumented (Valgrind) instrument those syscalls (~65 execve, ~300ms) land in the measured region, triggering the "cannot instrument system calls" warning and polluting instruction counts. The snippets only use stdlib, served from bundled in-memory typeshed, so no interpreter or site-packages are needed.
The walltime job ran `cargo codspeed run --mode walltime` but built with `cargo codspeed build` (default instrumentation mode), so no walltime-built benchmarks existed and the run failed with "No benchmarks found for the walltime mode". Pass `-m walltime` to the build so it matches the run mode.
CodSpeed CI installs the latest cargo-codspeed (v5.0.1), but the compat crate was pinned to major v2. Walltime result collection requires the compat layer and the cargo-codspeed binary to share a major version, so the walltime job failed with a version-incompatibility error. Align the compat crate (and its transitive codspeed runtime) to v5.
Under CodSpeed's instrumented (Valgrind) instrument, dispatching the
check to a rayon pool — even a single-thread one — parks the measured
thread on a futex for the full check while a worker runs it. That
blocking cross-thread syscall can't be instrumented, so CodSpeed drops
it from the measure and warns ("46 system calls, 304.7ms").
Add a ThreadCount::Inline mode that builds no rayon pool, so spawn_many
runs the work directly on the calling thread, and use it for the micro
benchmarks. All work then stays on the measured thread with no handoff.
The type checker times every step, run, and module load with Instant::now()/ elapsed() to populate telemetry counters. Instant::now() reads CLOCK_MONOTONIC via the vDSO, so it is nearly free normally — but under Valgrind (CodSpeed's instrumented instrument) the vDSO is bypassed and each reading becomes a real clock_gettime syscall. A single check makes ~34 of them; CodSpeed cannot instrument syscalls, so they are dropped from the measure and trigger a warning. Add a Timer type in pyrefly_util gated by a global flag (default enabled, so telemetry is unchanged) and route the hot-path timers through it. When disabled the timer never reads the clock, so no clock_gettime is issued.
TaskHeap and per-module step computation call Condvar::notify_all() after every work item / step completes. notify_all() issues a futex wake syscall even when no thread is parked, which under CodSpeed's Valgrind instrument cannot be instrumented (~7 futex per check in the single-threaded case). Track the parked-waiter count under the same lock that guards the wait, and only wake when it is non-zero. The count is read and mutated under the lock, so skipping the wake is race-free: a thread that has not yet parked cannot have incremented the count, and it re-checks the predicate under the lock before waiting.
With timers gate-able, turn them off in the micro benchmark's shared state so CodSpeed's instrumented instrument sees no clock_gettime syscalls in the measured region. Combined with the condvar-wake guard, a measured check now issues zero syscalls (previously 41: ~34 clock_gettime + ~7 futex).
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a CodSpeed GitHub Actions workflow so the landed criterion benchmarks run continuously: instruction-count (simulation) mode for the deterministic microbenchmarks, walltime mode for the heavy real-world PyTorch benchmarks (which must not run under Valgrind).
Swap the
criteriondev-dependency tocodspeed-criterion-compat(aliased back tocriterion) so the existing bench harness emits CodSpeed measurements undercargo codspeedwhile behaving like plain criterion otherwise, and add a[profile.bench]that drops LTO/single-codegen-unit to keep the CI build within runner memory limits.Summary
Fixes #XXXX
Test Plan