Repository navigation
Conversation
Member
Author
|
I will mark this as a draft for more extensive testing, but it would be nice to hear some feedback because changing container version is not super straightforward and I have to fully verify nothing breaks.. |
kargibora
commented
Oct 2, 2026
| # lm-eval and dependencies (system Python) | ||
| uv pip install --system --break-system-packages \ | ||
| # lm-eval and dependencies in the ROCm/PyTorch base venv. | ||
| uv pip install --python /opt/venv/bin/python \ |
Member
Author
There was a problem hiding this comment.
This is somewhat required to seperate environments. This doesnt change any old way of running the code.
kargibora
commented
Oct 2, 2026
| "vllm==0.19.1+rocm721" \ | ||
| "transformers==5.5.4" \ | ||
| "huggingface-hub>=1.5,<2" \ | ||
| "judgearena @ git+https://github.com/OpenEuroLLM/JudgeArena.git@{{ JUDGEARENA_GIT_REF }}" |
Member
Author
There was a problem hiding this comment.
Currently the main does not have the required code for JudgeArena so I think as the first draft it is better to let this be some reference to a specific working commit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for packaged JudgeArena tasks in oellm-eval. JudgeArena remains responsible for task definitions, prompts, judging, and scoring; oellm-eval handles scheduling and result collection.
judgearena tasks download.--judgearena_kwargs.Verification
I have verified this by reproducing 100-example COPA and BoolQ runs and 50-example Italian XCOPA which scored the same/after.
LUMI container
The LUMI base has been updated from ROCm 6.4 to ROCm 7.2.1 because the JudgeArena vLLM wheel requires ROCm 7.2 libraries. JudgeArena and vLLM should be installed in a separate virtual environment because they require a different version of
torch. In the future, JudgeArena can also be added to other cluster containers.Example
Run a packaged JudgeArena task with a candidate model and a local VLLM judge (given the models are already downloaded)
oellm-eval schedule \ --models Qwen/Qwen2.5-0.5B-Instruct \ --tasks alpaca-eval-ja \ --n_shot 0 \ --limit 10 \ --judgearena_kwargs '{"judge.model":"VLLM/Qwen/Qwen3-4B"}'