Audit published results and compare TREC evaluation policies

[majdabd]

The Nature Communications results could not be audited from the current CLI, and a fresh run silently used different retrieval and evaluation semantics. This adds checksum-pinned CPU-only commands for auditing the published artifact and re-evaluating completed local runs without repeating retrieval or model inference.

trialmatchai reproduce-paper verifies the official Zenodo archive and TREC qrels, then keeps stored aggregation, ranking-file recalculation, retrieval recall, and current-evaluator metrics separate. trialmatchai trec-evaluate compares two explicit unjudged-document policies: removing unjudged trials before cutoffs and retaining them at rank with zero gain. Reports include macro means, medians, per-topic metrics, qrels hashes, and fingerprints over the evaluated result files.

The audit covers all 125 paper topics. Six complete local configurations were also re-evaluated across TREC 2021 and 2022. For the closest local paper-stack analogue, h100_rrf, mean nDCG@10 changes from 0.790937 to 0.608344 on TREC 2021 and from 0.752821 to 0.653702 on TREC 2022 when unjudged trials are retained as zero-gain results. The tracked benchmark reports preserve both policies and all per-topic values.

The archived per-topic metrics aggregate to the archived summaries, but archived rankings do not regenerate at least one stored metric for 20/75 TREC 2021 topics and 19/50 TREC 2022 topics. Documentation therefore presents this as an artifact audit rather than an exact fresh-inference reproduction. The TREC runner remains lazily loaded for the CPU-only commands, and archive extraction now validates every metric, ranking, candidate, and recall-cutoff input before accepting a cache.

Validation:

• focused evaluator, reproduction, CLI, and package tests pass
• Ruff passes
• strict MkDocs build passes
• wheel build and installed-wheel CLI evaluation pass
• checksum-verified live paper audit covers all 125 topics
• all required GitHub quality, security, package, installed-CLI, model-import, and build checks pass

[Copilot]

Copilot review overview

🟡 Changes recommended

The extraction cache/validation logic can incorrectly accept an incomplete archive extraction (missing recall cutoff files), leading to later failures or misleading “cache complete” behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Cache validation omits required recall input files · New

What changed in this PR

Adds a reproducible, checksum-pinned “artifact audit” workflow to validate and re-evaluate the published TrialMatchAI TREC 2021/2022 result bundle (without rerunning inference), and exposes it via a new trialmatchai reproduce-paper CLI subcommand while keeping the TREC runner import lazy for CPU-only usage.

Changes:

• Introduces trialmatchai.trec.paper_reproduction to download/verify/extract the Zenodo archive, validate qrels checksums, and generate a structured audit report (stored metrics vs ranking recomputation vs current evaluator).
• Adds a trialmatchai reproduce-paper CLI command, plus README/MkDocs documentation for running and interpreting the audit.
• Updates trialmatchai.trec to lazily import the full runner via __getattr__, with tests covering import laziness and archive extraction safety.

File	Description
tests/​test_paper_reproduction.py	Adds tests for lazy TREC runner import, safe zip extraction, metric convention behavior, and audit report separation.
src/​trialmatchai/​trec/​paper_reproduction.py	Implements archive/qrels verification, safe selective extraction, paper-metric recalculation, recall computation, and track-level audit reporting.
src/​trialmatchai/​trec/​__init__.py	Switches run_tracks import to a lazy __getattr__ to avoid importing the full pipeline on CPU-only commands.
src/​trialmatchai/​cli/​reproduce_paper.py	Adds the reproduce-paper command wiring to produce a JSON report and human-readable summary.
src/​trialmatchai/​cli/​main.py	Registers the new reproduce-paper subcommand and dispatches to its CLI entry.
README.md	Documents the new paper reproduction/audit command and links to detailed docs.
pyproject.toml	Updates the scripts section comment to include reproduce-paper among subcommands.
mkdocs.yml	Adds the new paper reproduction page to the documentation nav.
docs/​paper-reproduction.md	Provides detailed audit methodology, outputs, and interpretation guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.