Reject duplicate qa_id entries instead of silently keeping the last prediction

[jiajieZeng]

Problem

The official evaluator silently keeps the last prediction when a submission contains the same qa_id more than once. This makes the evaluated answer depend on record order and can hide malformed submissions.

Minimal reproduction

Create a submission containing two records with the same ID:

[
  {
    "dataset": "angry",
    "qa_results": [
      {"qa_id": "angry:conv-0#q0000", "predicted_answer": "(A)"},
      {"qa_id": "angry:conv-0#q0000", "predicted_answer": "(B)"}
    ]
  }
]

Then load it with the evaluator:

PYTHONPATH=src python - <<'PY'
from pathlib import Path
from evaluate import load_submission

print(load_submission(Path("duplicate-submission.json")))
PY

Actual result:

{'angry:conv-0#q0000': '(B)'}

The (A) prediction is overwritten without an error or diagnostic because load_submission() assigns directly to predictions[str(qa_id)].

Expected behavior

Reject the submission with an explicit error, for example:

duplicate prediction for qa_id: angry:conv-0#q0000

Suggested scope

• Detect duplicate IDs in every accepted submission shape.
• Add a regression test covering duplicate records within a group and across groups.
• Keep the existing missing- and extra-prediction reporting unchanged.

I also ran the published 457-item submission template through scripts/run_eval.py; all template IDs resolve normally, so this report is limited to duplicate-ID validation.

[Z-Chealse]

I agree that rejecting duplicated qa_id entries would be helpful. When preparing prediction files for evaluation, silently keeping the last entry could make it difficult to notice formatting mistakes.