Add safety and fairness benchmarks #405

Goal

Integrate the safety and fairness benchmarks listed in the benchmark matrix into oellm-eval.

Implementation plan

Benchmarks differ in their applicability to pretrained, instruction-tuned, and safety-tuned models. We will begin with benchmarks suitable for pretrained/base models, followed by those that require instruction-following or safety-tuned behavior.

This work also includes adding or extending support for inspect_evals and lighteval benchmarks where necessary.

Local implementation progress

Shared infrastructure

• Add and verify inspect_evals integration in oellm-eval

lm-eval

• CrowS-Pairs
• ToxiGen

inspect_evals

• BBQ
• StereoSet
• WMDP
• BOLD
• XSTest
• StrongREJECT
• AgentHarm
• AgentDojo

lighteval

• RealToxicityPrompts (has weird implementation so maybe discard it)

Custom or external integrations

These benchmarks are not currently recorded as implementations in the three suites above and may require custom adapters or their native evaluation code.

• HarmBench
• AdvBench
• JailbreakBench

HPC implementation progress

Shared infrastructure

• Add and verify inspect_evals integration in oellm-eval

lm-eval

• CrowS-Pairs (PR)
• ToxiGen

inspect_evals

• BBQ
• StereoSet
• WMDP
• BOLD
• XSTest
• StrongREJECT
• AgentHarm
• AgentDojo

lighteval

• RealToxicityPrompts (has weird implementation so maybe discard it)

Custom or external integrations

These benchmarks are not currently recorded as implementations in the three suites above and may require custom adapters or their native evaluation code.

• HarmBench
• AdvBench
• JailbreakBench

Recommended benchmarks by model stage

Phase 1: Pretrained/base models

• CrowS-Pairs
• StereoSet
• RealToxicityPrompts
• ToxiGen
• WMDP
• BBQ, using likelihood-based scoring where appropriate

Refusal and jailbreak scores should not be interpreted as safety-alignment measurements for base models.

Phase 2: Instruction-tuned/SFT models

• BBQ
• BOLD
• RealToxicityPrompts
• ToxiGen
• XSTest
• HarmBench
• StrongREJECT
• AdvBench

Phase 3: Post-trained/safety-tuned models

• XSTest
• HarmBench
• JailbreakBench
• StrongREJECT
• AgentHarm
• AgentDojo

Definition of done

For each benchmark:

• The benchmark is runnable through the standard oellm-eval workflow
• The upstream suite and task name are documented
• A minimal configuration or example command is provided
• A smoke test verifies that the benchmark loads and produces results
• Results are written in the standard oellm-eval output format
• Applicable model stages and interpretation caveats are documented

A benchmark may appear in more than one upstream suite. Each suite integration should be tracked separately when the implementations or resulting metrics differ.