Evaluation: a multiple-choice gold of "A" normalizes to empty, so acc and coverem score any answer 1.0

[chiruu12]

Bug

normalize_text (servers/evaluation/src/evaluation.py:37) strips articles unconditionally:

def _remove_articles(t: str) -> str:
    return re.sub(r"\b(a|an|the)\b", " ", t)

A ground truth of "A" normalizes to the empty string. An empty string is a substring of every prediction, so accuracy_score returns 1.0 whatever the model answered, and cover_exact_match_score does the same because all() over an empty token list is True.

The same gold also scores 0.0 when the model is right, because accuracy_score returns early on an empty normalized prediction and "A" normalizes to empty on that side too. So option A is scored wrong in both directions. Options B through Z are unaffected.

"a", "an" and "the" behave the same way as gold values.

Reproduction

Against main at 0ba52ed, with the four pure functions lifted out of evaluation.py so the MCP app and the rouge scorer are not needed:

import ast, re, string
from typing import List

src = open("servers/evaluation/src/evaluation.py").read()
tree = ast.parse(src)
want = {"normalize_text", "accuracy_score", "cover_exact_match_score"}
mod = ast.Module(body=[n for n in tree.body
                       if isinstance(n, ast.FunctionDef) and n.name in want],
                 type_ignores=[])
ns = {"re": re, "string": string, "List": List}
exec(compile(mod, "evaluation.py", "exec"), ns)

print(repr(ns["normalize_text"]("A")))                       # ''
print(ns["accuracy_score"](["A"], "D"))                      # 1.0
print(ns["cover_exact_match_score"](["A"], "D"))             # 1.0
print(ns["accuracy_score"](["A"], "A"))                      # 0.0
print(ns["accuracy_score"](["B"], "D"))                      # 0.0   control

gold	prediction	acc	coverem	should be
A	D	1.0	1.0	0.0
A	A	0.0	1.0	1.0
A	completely wrong	1.0	1.0	0.0
B	D	0.0	0.0	0.0

Why it is reachable

Both affected metrics are on by default. servers/evaluation/parameter.yaml:5:

metrics: [ 'acc', 'f1', 'em', 'coverem', 'stringem', 'rouge-1', 'rouge-2', 'rouge-l' ]

compute_metrics also falls back to every registered metric when metrics is empty.

The option letters come from the repo's own prompt builder. qa_boxed_multiple_choice (servers/prompt/src/prompt.py:159) labels choices with list(string.ascii_uppercase), and examples/experiments/vanilla_multiple_choice.yaml and rag_multiple_choice.yaml wire that through custom.output_extract_from_boxed into evaluation.evaluate. So a multiple-choice run on either shipped pipeline reports inflated acc and coverem on every question whose answer is A, which is roughly a quarter of a four-option benchmark.

To be clear about what I did not find: the bundled data/sample_nq_10.jsonl has no gold that normalizes to empty, so nothing in the repo's own sample data triggers this. I did not find an existing issue or PR covering it.

Suggested fix

Guarding inside the metrics does not work. Skipping golds that normalize to empty would score a correct "A" as 0.0, which trades a false positive for a false negative. The narrower change is to stop article removal from consuming the entire string:

def _remove_articles(t: str) -> str:
    stripped = re.sub(r"\b(a|an|the)\b", " ", t)
    return t if not stripped.strip() else stripped

"The Beatles" still normalizes to "beatles", so ordinary article stripping is unchanged. Only the case where nothing would be left behaves differently.

I have this written with tests, 5 of which fail on current main and 3 of which are regression guards that pass either way. Happy to open a PR if you want it, or leave it to you.