[Fix] Extract the final answer in GPQA simple-eval predictions

[Hibbert133]

[Fix] Extract the final answer in GPQA simple-eval predictions

Motivation

GPQA simple-eval prompts require the model to place the final answer on the last line in the form ANSWER: <LETTER>. In practice, some model outputs contain multiple ANSWER: tokens because the model reasons aloud, drafts an intermediate guess, and then self-corrects. The previous postprocessor used the first regex match, so it could pick an earlier intermediate answer instead of the final one and undercount accuracy.

Modification

• Update GPQA_Simple_Eval_postprocess to collect all ANSWER: [A-D] matches and return the last match.
• Keep the existing matching behavior unchanged for:
  • case-insensitive ANSWER
  • optional whitespace around :
  • valid answer letters restricted to A-D
• Add regression tests that cover:
  • a single match
  • multiple matches where the last one should win
  • case and whitespace variants
  • no-match behavior
• Remove the temporary GPQA README note so the branch only contains the actual code fix and regression coverage.

BC-breaking (Optional)

No backward-compatible API break is introduced. This is a behavior fix in post-processing for GPQA simple-eval predictions.

Use cases (Optional)

This affects GPQA simple-eval evaluation, especially for model responses that include:

• a draft answer followed by a corrected final answer
• chain-of-thought outputs with repeated ANSWER: tokens
• prompts that explicitly instruct the model to put the answer on the last line

Technical Analysis

The bug comes from the mismatch between the prompt contract and the extraction strategy:

• The prompt instructs the model to put the answer in the final line.
• The old implementation used re.search, which stops at the first match.
• If the model emits multiple ANSWER: tokens, the first one is often an intermediate guess, not the final answer.

The fix uses re.findall(...)[-1], which preserves the existing regex constraint while selecting the last valid answer token. This makes the evaluator align with the prompt semantics and avoids false negatives caused by intermediate self-corrections.

Test Plan

• python tests/datasets/test_gpqa.py
• python -m unittest tests.datasets.test_gpqa

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[ssiq]

@Hibbert133 You can configure your inference model like this:

dict(
abbr='DeepSeek-V3.1-Reasoning',
type=OpenAISDK,
path='deepseek-ai/DeepSeek-V3.1',
key='ENV',
query_per_second=1,
max_out_len=128000,
max_seq_len=131072,
batch_size=8,
temperature=0.0,
extra_body=dict(thinking=dict(type='enabled')),
openai_extra_kwargs=dict(reasoning_effort='high'),
pred_postprocessor=dict(type=extract_non_reasoning_content),
retry=10)

OpenCompass will call the model’s pred_postprocessor before calling the dataset’s pred_postprocessor. Therefore, before the result enters GPQA_Simple_Eval_postprocess, extract_non_reasoning_content
will have already removed the reasoning part from the inference output.