fix(evaluation): keep answers that are entirely articles

[chiruu12]

Fixes #509.

normalize_text stripped articles unconditionally, so a gold answer of "A" normalized to the empty string. An empty string is a substring of every prediction, so accuracy_score returned 1.0 whatever the model answered, and cover_exact_match_score did the same because all() over an empty token list is True. The same gold scored 0.0 when the model was right, because accuracy_score returns early on an empty normalized prediction.

Both metrics are in the default set (servers/evaluation/parameter.yaml:5), and qa_boxed_multiple_choice labels options with string.ascii_uppercase, so a run of examples/experiments/vanilla_multiple_choice.yaml reported inflated acc and coverem on every question whose answer was A.

The change

Article removal now keeps the un-stripped form when it would otherwise leave nothing:

stripped = re.sub(r"\b(a|an|the)\b", " ", t)
return t if not stripped.strip() else stripped

"The Beatles" still normalizes to "beatles". Only the case where the whole string would disappear behaves differently.

I tried guarding inside the metrics first, skipping golds that normalize to empty. That scores a correct "A" as 0.0, so it trades a false positive for a false negative. The normalization is the right place.

Tests

tests/servers/evaluation/test_normalize_text.py, 8 tests. The evaluation server is not part of the installable package and importing it pulls in the MCP app and the rouge scorer, so the pure functions are lifted out with ast, matching how tests/servers/custom/test_query_extract.py reaches into servers/custom/src.

Five fail on current main and are reproductions:

• test_single_letter_answer_survives_normalization
• test_article_only_answer_survives_normalization
• test_accuracy_rejects_a_wrong_multiple_choice_answer
• test_accuracy_accepts_a_correct_multiple_choice_answer
• test_cover_exact_match_rejects_a_wrong_multiple_choice_answer

Three pass with and without the change and are regression guards, labelled as such in the file:

• test_cover_exact_match_accepts_a_correct_multiple_choice_answer
• test_articles_are_still_stripped_inside_a_longer_answer
• test_unaffected_option_letters_are_unchanged

pytest tests/ is 12 passed.

One thing I did not change: exact_match_score and string_em_score compare normalized strings for equality, so an empty gold only matched an empty prediction and they were never wrong here. They pick up the corrected normalization for free.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

One limit of this change, found while trying to break my own patch.

accuracy_score matches by substring. So once "A" normalizes to "a" rather than "", it still matches any prediction containing that letter:

acc(["A"], "apple")           -> 1.0
acc(["A"], "The answer is B") -> 1.0
acc(["A"], "D")               -> 0.0

cover_exact_match_score is clean here. It compares token membership, not substrings, and returns 0.0 on all three.

The shipped multiple-choice pipelines are unaffected either way, because custom.output_extract_from_boxed reduces the prediction to a bare letter before evaluation, so A through D all score correctly. The residual only bites if acc runs against free-text predictions with a short gold.

That substring behaviour predates this change and is not specific to single letters. "cat" matches "concatenate" the same way. Tightening it to whole-word matching would move numbers for every dataset, so folding it into a bug fix seemed wrong. Happy to raise it separately if you want it changed.