Rescore referee arms from committed caches using declared-only accounting

[armaanvgrewal]

Summary

The declaration-aware accounting added in #417 introduced a distinction between:

• parser-derived answers, and
• explicitly declared answers (declared_mcq_choice).

The self-inconsistency lane was updated to report:

• declared_pairs
• undeclared_pairs
• undeclared_draws

and to compute metrics over declared pairs only.

This issue tracks a retrospective audit of previously reported referee results using the same declared-only accounting.

Motivation

Historical referee results may have included cases where a legacy parser fallback produced an answer despite the model never explicitly declaring one.

The goal is to quantify whether using declared-only accounting would change any previously reported results.

Scope

Starting with:

• referee_deployable
• plausible_distractor

Recompute reported metrics from committed caches without issuing new API calls.

For each arm:

1. Replay from committed cache artifacts.
2. Compute results using current behavior.
3. Compute results using declared-only behavior.
4. Produce side-by-side comparisons.

Deliverables

For each audited arm:

• current metric value
• declared-only metric value
• number of declared cases
• number of undeclared cases
• absolute difference
• interpretation of whether conclusions change

Constraints

• No new model calls.
• Replay entirely from committed cache artifacts.
• Preserve a reproducible audit trail.
• Do not change historical results in-place as part of this work.

Related

Follow-up from review of PR #418.

[maximinl]

Claiming. Offline declared-only rescore of referee_deployable and plausible_distractor from committed caches (MedQA + MedMCQA); new audit artifacts only, no historical overwrite.