Skip to content

Add packaged JudgeArena task support for LUMI - #127

Draft
kargibora wants to merge 10 commits into
OpenEuroLLM:mainfrom
kargibora:feat/judgearena-task-integration-v2
Draft

kargibora wants to merge 10 commits into
OpenEuroLLM:mainfrom
kargibora:feat/judgearena-task-integration-v2

Conversation

@kargibora

Copy link
Copy Markdown
Member

Summary

This PR adds support for packaged JudgeArena tasks in oellm-eval. JudgeArena remains responsible for task definitions, prompts, judging, and scoring; oellm-eval handles scheduling and result collection.

  • Register JudgeArena tasks and language variants in the task groups.
  • Prepare task inputs with judgearena tasks download.
  • Run JudgeArena in the selected container or venv, with judge settings supplied through --judgearena_kwargs.
  • Report failed runs and collect numeric metrics from JudgeArena’s run metadata.

Verification

I have verified this by reproducing 100-example COPA and BoolQ runs and 50-example Italian XCOPA which scored the same/after.

LUMI container

The LUMI base has been updated from ROCm 6.4 to ROCm 7.2.1 because the JudgeArena vLLM wheel requires ROCm 7.2 libraries. JudgeArena and vLLM should be installed in a separate virtual environment because they require a different version of torch. In the future, JudgeArena can also be added to other cluster containers.


Example

Run a packaged JudgeArena task with a candidate model and a local VLLM judge (given the models are already downloaded)

   oellm-eval schedule \
     --models Qwen/Qwen2.5-0.5B-Instruct \
     --tasks alpaca-eval-ja \
     --n_shot 0 \
     --limit 10 \
     --judgearena_kwargs '{"judge.model":"VLLM/Qwen/Qwen3-4B"}'

@kargibora

Copy link
Copy Markdown
Member Author

I will mark this as a draft for more extensive testing, but it would be nice to hear some feedback because changing container version is not super straightforward and I have to fully verify nothing breaks..

Comment thread containers/lumi.def
# lm-eval and dependencies (system Python)
uv pip install --system --break-system-packages \
# lm-eval and dependencies in the ROCm/PyTorch base venv.
uv pip install --python /opt/venv/bin/python \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is somewhat required to seperate environments. This doesnt change any old way of running the code.

Comment thread containers/lumi.def
"vllm==0.19.1+rocm721" \
"transformers==5.5.4" \
"huggingface-hub>=1.5,<2" \
"judgearena @ git+https://github.com/OpenEuroLLM/JudgeArena.git@{{ JUDGEARENA_GIT_REF }}"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the main does not have the required code for JudgeArena so I think as the first draft it is better to let this be some reference to a specific working commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant