Add robust HumanEval postprocess for chat outputs

[Ding-god]

Motivation

The current HumanEval postprocess mainly extracts closed Markdown code blocks from model outputs. This works for models that return clean fenced code, but it is less robust for chat/API models that may include reasoning before the final code or produce an opening code fence that is not closed because the generation is truncated.

In these cases, the evaluator may receive natural-language text instead of the intended Python solution, which can cause SyntaxError failures due to output formatting rather than the generated solution itself.

This PR aims to make HumanEval postprocessing more robust for chat-style model outputs while keeping the existing postprocess behavior unchanged.

Modification

This PR adds a new humaneval_chat_postprocess function.

The new postprocessor:

• extracts the first Markdown code block, including unclosed code fences;
• falls back to extracting a clear Python snippet starting with from, import, def, class, or a decorator;
• trims trailing non-code text by keeping the longest compilable Python prefix;
• does not modify the existing humaneval_postprocess_v2 or humaneval_postprocess_v3 behavior.

Unit tests are added for:

• unclosed Markdown code fences;
• Python code appearing after natural-language reasoning;
• trailing explanation after code;
• existing plain function-body behavior.

BC-breaking (Optional)

No backward compatibility breaking change is introduced.

The existing humaneval_postprocess_v2 and humaneval_postprocess_v3 functions are unchanged. The new humaneval_chat_postprocess function is opt-in and only affects users who explicitly configure it.

Use cases (Optional)

This postprocessor can be used when evaluating chat/API models on HumanEval, especially when the model output may contain reasoning text before the final Python code or an unclosed Markdown code fence.

Example use case:

humaneval_eval_cfg = dict(
    evaluator=dict(type=HumanEvalEvaluator),
    pred_role='BOT',
    k=[1, 10, 100],
    pred_postprocessor=dict(type=humaneval_chat_postprocess),
)

The postprocessor only recovers code already produced by the model. It does not repair invalid code, synthesize missing function bodies, or modify model logic.

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[Ding-god]

Hi maintainers, this PR improves HumanEval post-processing for chat-style model outputs. It handles cases such as unclosed markdown fences and reasoning text before code blocks. Could you please review when available? Thanks!