Implement LLM-driven review assigner for PR bot - #39980
Conversation
Beam's existing PR reviewer assigner matches directory paths against coarse labels (e.g. java, python, kafka) and assigns one reviewer per matching label via round-robin. This creates reviewer bloat on multi-directory PRs, fails to consider who actually worked on the affected files, and offers no rationale to reviewers on why they were tagged. Introduce an intelligent reviewer assigner powered by Gemini: 1. Extract git revision history and churn for touched files (with fallback to parent directory for new files), filtering automated bots and the PR author. 2. Use Gemini to identify true domain experts based on substantive commits rather than mechanical edits (e.g. formatting, version bumps), and minimize the assigned reviewer set (target: 1 reviewer). 3. Post an informative PR comment stating the chosen reviewer(s), their specific expertise rationale, and suggested backup experts. 4. Support seamless handover to backup experts when a reviewer requests reassignment via "assign to next reviewer". 5. Provide a deterministic recency-decayed familiarity scoring fallback when GEMINI_API_KEY is not configured or fails.
343830d to
f27503e
Compare
|
@damccorm I need to give this a more thorough review, but the biggest picture is that I don't really know what it looks like for Beam's infra to make the gemini calls that are in this using a GEMINI_API_KEY or if there is an entirely different appropriate architecture. |
| this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)"; | ||
|
|
||
| return `You are the Apache Beam Code Review Assigner. | ||
| Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn. |
There was a problem hiding this comment.
I haven't done a full review, but I don't think that this has the information needed to do this optimally. Specific things which aren't covered, but seem important:
- How many times has this reviewer been assigned recently (right now we just round robin)
- Committership
- Does this person actually want to be a reviewer for the area/are they willing to respond in a timely manner (I can think of contributors who commit a good amount of code, but aren't necessarily ideal reviewers).
There was a problem hiding this comment.
Looking at the problems, do we need a full system overhaul or can we just make smaller less-invasive changes? The gemini infra is going to be painful since it will need to be a secret we regularly rotate.
Reviewer bloat on multi-directory PRs
Could this be solved by just choosing one of the reviewers from the set of available ones (probably with a preference for code-related expertise)?
Doesn't stay up to date
Is this really an issue? I have not seen this one cause meaningful problems since reviewers tend to opt out fairly quickly once they are out of the loop.
I think the ability to opt in/out is a pretty useful feature.
Is pretty coarse
This is less avoidable (though we certainly could have tighter categories).
There was a problem hiding this comment.
We could ditch gemini and just go with the heuristic score, for example.
There was a problem hiding this comment.
And TBH I think the selected reviewers are just almost always not the right person. I'm biased because I get on there for "java" and "website" and "fallback" and realistically there's always a better choice. I don't think load balancing to people who are that vaguely related is necessarily good.
2 participants
Beam's existing PR reviewer assigner works like so:
The downsides:
This PR is meant to introduce a little more intelligence:
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.