Repository navigation
Add option to run suite on PRs? #106
Description
This was previously suggested by @savannahostrowski on Discourse.
The rough idea: let users pick a PR on speed.{python,pypy}.org (to start with) and have the site run a performance comparison for it.
Implementing the UI side here wouldn't be too difficult (some work needed to figure out the dispatch mechanism, but not too bad); the expensive part is the computation. Even with cached results and runs only on demand, a comparison needs the suite run twice in the worst case (base and PR). Some thoughts on that:
- Run only a subset of the suite, or make the subset selectable, to save resources?
- Run on GitHub Actions rather than the current environments? [*]
- Security-wise, we'd either have to trust the users allowed to trigger runs (restricted to the python-core/triage teams) not to start runs on malicious PRs, or implement some sort of sandbox (famous last words ;-).
Leaving this here for now, I'm curious what others think! :-)
[*]: If we're on GitHub Actions anyway, we could do everything in python/cpython with a fairly straightforward label-triggered workflow. We'd lose the tracking on the website and the options for customising runs, though.
Activity
mattip commentedon Nov 9, 2026
I wonder how stable gha runners are. Would you then have a single "environment" label for gha, or try to keep each run separate under a gha environment accordion by branch?
Also: for PyPy we allow kicking off a buildbot run on a branch via the various buildbot builders pages, including benchmarks. But I ran into a UX problem that the executables section on the comparison page got very long, branch names tend to force more than one line of text in the list. By default all branches are displayed. So I added an option to the admin panel to hide old branches at admin/codespeed/branch/. Works for me since I am an admin :) Some thought might be needed to make all that more user friendly.
StanFromIreland commentedon Nov 9, 2026
Indeed, although it's a cost trade-off here. However, looking at memory.python.org where all envisionments are GHAs, it seems reasonable stable.
As for the UI, I'd need to think a bit more about it. We'd likely want the option to run on various runners.
savannahostrowski commentedon Nov 9, 2026
I added a bit more context on Discourse. I was thinking of using
bench_runneron dedicated machines registered as self-hosted GitHub Actions runners, with runs triggered from the PR and results posted back.We’re also looking into getting dedicated benchmarking machines. Thomas has been investigating this, and it’s a standing item for the SC to discuss. Where we host the detailed results feels like a bit of a separate question.
StanFromIreland commentedon Nov 9, 2026
Right, though I was thinking more along the lines of recreating the setup we currently have for Buildbots, which seems to be working reasonably well.
I wonder whether Hetzner might be open to providing a second machine. ;-)
Agreed, and that's (+ the infrastructure for orchestrating) actually the main question I'd like to discuss here.
savannahostrowski commentedon Nov 9, 2026
Maybe we could pick this up on Discourse so folks following that thread can weigh in too, and we can keep the discussion in one place? I’d like to understand what orchestration you have in mind and how you see it fitting with
bench_runner. I was imagining building on its existing GitHub Actions workflows.StanFromIreland commentedon Nov 9, 2026
Hmm, maybe a new thread? That one was for general improvements, a more focused one may be better here?
I've drafted #109
savannahostrowski commentedon Nov 9, 2026
I think it’s premature to move this PR forward before we’ve discussed the overall approach. I’m planning to work on this, but wanted to talk through the design and coordinate with interested folks before starting an implementation. We haven’t worked through how this fits with bench_runner or whether Codespeed should be managing these runs. The UX I’m aiming for is requesting benchmarks and getting results directly on the PR. Going through another site adds friction during review.
I’d like us to discuss those design questions and agree on a plan together on Discourse before deciding on an implementation.
StanFromIreland commentedon Nov 9, 2026
Oh yes, absolutely! The PR is just a quick draft I put together with Claude to illustrate the idea and a possible approach.
I actually have a similar (dual) UX in mind: adding a label to the PR would trigger the benchmarks, and a comment would then be posted with a link to the results, much like Buildbot runs. So there wouldn't be any need to visit another site unless you wanted to dig into the details or select specific options.
I'd be happy to discuss the design first, which is why I opened this issue in the first place. But I'm also happy to start a Discourse thread if that's the preferred place for the discussion!