Skip to content

fix: use last-match in generic LLM judge postprocess (verdict-injection defense)#2534

Open
AUTHENSOR wants to merge 1 commit into
open-compass:mainfrom
AUTHENSOR:fix/verdict-injection-generic-llmjudge
Open

fix: use last-match in generic LLM judge postprocess (verdict-injection defense)#2534
AUTHENSOR wants to merge 1 commit into
open-compass:mainfrom
AUTHENSOR:fix/verdict-injection-generic-llmjudge

Conversation

@AUTHENSOR

Copy link
Copy Markdown

Summary

_generic_llmjudge_postprocess in generic.py used re.search (first-match, unanchored) to extract the A/B verdict from the judge's response. The model-under-test's output is interpolated verbatim into the judge prompt (via {prediction} in configs), so an early injected verdict tag (e.g. starting the answer with "A. ") that gets echoed in the judge's response wins over the judge's actual final verdict.

This is the verdict-injection class — the same pattern fixed across deepeval, ragas, guardrails-ai, promptfoo, instructor, CrewAI, LlamaIndex, AutoGPT, lighteval, HumanEval, and EvalPlus.

Impact

This shared helper (_generic_llmjudge_postprocess) is used by 13+ datasets across OpenCompass, including:

  • MedXpertQA, Medbullets, medmcqa, supergpqa (medical/science QA)
  • livereasonbench, simpleqa (reasoning)
  • matbench, mtbench101, writingbench (subjective evaluation)
  • clbench, fofo (format compliance)

A model that prepends its answer with the correct verdict tag ("A. ") can flip a binary correct/incorrect judgment on any of these benchmarks.

Fix

Switch from re.search (first-match) to re.findall + take the last match. The judge's rubric instructs it to end with the verdict, so the last match is the authoritative one.

Verified

Judge response: "A. The model output suggests A. However the real answer is B."
  vuln:   A (injected wins) 
  fixed:  B (real verdict wins)

Normal response: "The answer is B."
  vuln:   B
  fixed:  B (unchanged)

Checklist

  • Code compiles
  • Fix verified: injected verdict no longer wins, normal responses unaffected
  • Same pattern shipped to 10+ other eval frameworks

…on defense)

_generic_llmjudge_postprocess used re.search (first-match) to extract the
A/B verdict from the judge response. The model-under-test's output is
interpolated verbatim into the judge prompt, so an early injected verdict
tag echoed in the judge's response wins over the judge's actual final
verdict.

Switch to re.findall + take the last match. Same pattern fixed across
deepeval, ragas, guardrails-ai, promptfoo, instructor, CrewAI, LlamaIndex,
AutoGPT, lighteval, HumanEval, and EvalPlus.

This shared helper is used by 13+ datasets (MedXpertQA, Medbullets,
medmcqa, supergpqa, livereasonbench, simpleqa, matbench, mtbench101,
writingbench, clbench, fofo, etc).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants