Audit published results and compare TREC evaluation policies

README.md

@@ -10,7 +10,7 @@
 [![Python 3.11](https://img.shields.io/badge/python-3.11-2563EB)](https://github.com/cbib/TrialMatchAI/blob/main/pyproject.toml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-64748B)](https://github.com/cbib/TrialMatchAI/blob/main/LICENSE)
 
-[Try the demo](#try-the-demo) · [Run your own data](#run-your-own-data) · [Architecture](#how-matching-works) · [CLI](#cli-reference) · [Development](#development-and-delivery)
+[Try the demo](#try-the-demo) · [Run your own data](#run-your-own-data) · [Reproduce the paper](#reproduce-the-paper) · [Architecture](#how-matching-works) · [CLI](#cli-reference) · [Development](#development-and-delivery)
 
 </div>
 
@@ -234,6 +234,8 @@ Run `trialmatchai <command> --help` for the current options.
 | `report` | Render a patient report or the multi-patient front page |
 | `update-registry` | Fetch and upsert ClinicalTrials.gov studies; optional watch mode |
 | `trec` | Run TREC Clinical Trials evaluation presets |
+| `trec-evaluate` | Re-score completed TREC rankings with explicit unjudged policies; no GPU inference |
+| `reproduce-paper` | Verify and recalculate the published TREC 2021/2022 result artifact |
 | `finetune` | Train supported reasoning, reranker, or NER components |
 | `healthcheck` | Inspect configured dependencies, paths, and services |
 | `artifacts` | Create or verify portable SHA-256 manifests |
@@ -272,6 +274,30 @@ The current evaluator and retrieval pipeline have open audit findings; this
 release does not claim a newly measured ranking improvement or a validated
 comparison against another system.
 
+Completed rankings can be evaluated without repeating retrieval or model
+inference. `trialmatchai trec-evaluate` reports both condensed metrics that
+exclude unjudged trials and metrics that retain unjudged trials with gain zero.
+See the [TREC evaluation guide](https://github.com/cbib/TrialMatchAI/blob/main/docs/trec-evaluation.md)
+for commands, input fingerprints, and measured policy sensitivity across the
+existing complete runs.
+
+## Reproduce the paper
+
+The published TREC result artifact can be audited on CPU with one command:
+
+```bash
+trialmatchai reproduce-paper --workdir ./paper-reproduction
+```
+
+This verifies the pinned Zenodo archive, aggregates its stored per-topic metrics,
+recalculates metrics from the archived rankings, and evaluates the same rankings
+with the current metric implementation. It does not rerun model inference. The
+archived summaries are internally consistent, but some archived rankings do not
+regenerate their stored topic metrics, and the present tie-aware evaluator is not
+the evaluator used for the paper values. See the [full reproduction record](https://github.com/cbib/TrialMatchAI/blob/main/docs/paper-reproduction.md)
+for the measured differences, retrieval recall, offline usage, and the inputs
+still required for an exact historical rerun.
+
 The [research paper](https://doi.org/10.1038/s41467-026-70509-w) describes the
 published study. Its results should be distinguished from this release's software
 checks. The roadmap prioritizes retrieval recall, eligibility semantics, evidence

docs/paper-reproduction.md

@@ -0,0 +1,134 @@
+# Reproducing the published results
+
+The Nature Communications study evaluated TrialMatchAI on 75 TREC 2021 topics,
+50 TREC 2022 topics, and a separate synthetic “ideal candidate” experiment. The
+current repository can verify the immutable result artifact and recalculate its
+TREC metrics without a GPU. It cannot yet perform an exact fresh inference rerun
+of the historical pipeline.
+
+## Run the artifact audit
+
+From an installed release containing this command:
+
+```bash
+trialmatchai reproduce-paper --workdir ./paper-reproduction
+```
+
+The command downloads the 124.7 MB `matching_results.zip` from the paper's
+[Zenodo data record](https://zenodo.org/records/15045515), verifies its pinned
+SHA-256 digest, extracts only the files needed for the audit, downloads the
+official TREC qrels, and writes `paper-reproduction/reproduction-report.json`.
+It reports three separate views:
+
+1. **Stored per-topic aggregation** checks whether the archived topic metrics
+   aggregate to the archived `average_metrics.json`.
+2. **Recalculated from rankings** applies the paper artifact's metric convention
+   to `ranked_trials.json`: remove unjudged trials, use linear relevance gains,
+   retain file order within score ties, and compute graded precision as
+   `sum(grade) / (2*k)`.
+3. **Current evaluator** applies the repository's present tie-aware, condensed
+   evaluator to the same rankings. These values answer a different metric
+   question and should not be substituted for the published values.
+
+Use [`trialmatchai trec-evaluate`](trec-evaluation.md) to compare the current
+evaluator with unjudged trials excluded versus retained as zero-gain results.
+That comparison can reuse any completed result directory and performs no model
+inference.
+
+For an offline or CI run, supply the already-downloaded official archive and
+qrels:
+
+```bash
+trialmatchai reproduce-paper \
+  --workdir ./paper-reproduction \
+  --archive /artifacts/matching_results.zip \
+  --qrels-dir /artifacts/qrels \
+  --no-download --json
+```
+
+The official archive accepted by the command has SHA-256
+`dbfd11f19ffbfd78acfeaeee634c3e7e7ca8cb4d72b526ff8d191f9e880312c8`.
+The TREC 2021 and 2022 qrels are also pinned by SHA-256. Changing any input byte
+causes the audit to stop.
+
+## What the published artifacts reproduce
+
+The following values were obtained with the command above on 19 September 2026.
+Means and medians are macro-averages across all topics in each track.
+
+| Track and metric | Stored mean | Stored median | Recalculated mean | Current-evaluator mean |
+| :--- | ---: | ---: | ---: | ---: |
+| TREC 2021 nDCG@5 | 0.729721 | 0.746939 | 0.727659 | 0.669658 |
+| TREC 2021 nDCG@10 | 0.716471 | 0.735478 | 0.706848 | 0.669746 |
+| TREC 2021 nDCG@20 | 0.724413 | 0.733114 | 0.692088 | 0.688335 |
+| TREC 2021 graded P@10 | 0.705333 | 0.750000 | 0.692667 | 0.692667 |
+| TREC 2022 nDCG@5 | 0.722144 | 0.819318 | 0.717099 | 0.587426 |
+| TREC 2022 nDCG@10 | 0.708484 | 0.762500 | 0.683137 | 0.598333 |
+| TREC 2022 nDCG@20 | 0.719456 | 0.766262 | 0.656213 | 0.632342 |
+| TREC 2022 graded P@10 | 0.663000 | 0.725000 | 0.661000 | 0.661000 |
+
+For the same archived rankings, retaining unjudged trials as zero-gain results
+instead of excluding them changes current tie-aware mean nDCG@10 from 0.669746
+to 0.435800 on TREC 2021 and from 0.598333 to 0.458934 on TREC 2022. The JSON
+report stores both policies explicitly.
+
+Across all 125 topics, weighting both tracks by topic count gives a stored mean
+nDCG@10 of 0.713276 and stored mean graded P@10 of 0.688400. These do not exactly
+regenerate the paper text's pooled values of 0.7232 and 0.6865 from the published
+per-topic metric files.
+
+The stored per-topic files aggregate to the stored summaries. Recalculation from
+the archived rankings finds at least one metric difference in 20 of 75 TREC 2021
+topics and 19 of 50 TREC 2022 topics. The report records every affected topic and
+value. This means the aggregate tables are verifiable as an aggregation of stored
+metrics, but some stored topic metrics cannot be regenerated from the ranking
+files in the same archive.
+
+The archived first-stage cutoff files produce the following grade-1-or-higher
+retrieval recall:
+
+| Track | Recall@500 mean | Recall@500 median | Recall@1000 mean | Recall@1000 median |
+| :--- | ---: | ---: | ---: | ---: |
+| TREC 2021 | 0.693768 | 0.738095 | 0.757988 | 0.817518 |
+| TREC 2022 | 0.676977 | 0.719643 | 0.716526 | 0.769697 |
+
+These archived values do not establish the paper text's statement of greater
+than 90% retrieval recall at roughly the top 3% of trials under this relevance
+definition and macro-averaging. That statement may use a different run,
+denominator, or aggregation, but the distinction is not recoverable from the
+published artifact.
+
+The paper's source-data workbook supports the reported ideal-candidate result:
+92 of 100 ground-truth trials are ranked first, 95 are in the top two, and all
+are in the top nine. The matching archive's `IDEAL_CANDIDATES` directory contains
+102 runs and does not regenerate that workbook table from its final ranking
+files, so the current command does not present it as a ranking-file
+recalculation.
+
+## Why a fresh exact rerun is not yet available
+
+The [paper](https://doi.org/10.1038/s41467-026-70509-w) describes Elasticsearch,
+BGE-M3 retrieval, Gemma-2-2B reranking, and Phi-4 eligibility reasoning. The
+current pipeline uses LanceDB and has changed ranking, tie, configuration, and
+resume behavior. The [archived v0.01 code](https://doi.org/10.5281/zenodo.18329084)
+does not include a single executable workflow that reconstructs all figures, and
+its checked-in defaults do not fully match the methods text. The published
+bundle also lacks enough lineage to bind every output to exact registry records,
+model revisions, container images, prompts, random seeds, and hardware/runtime
+versions.
+
+An exact inference reproduction therefore needs a new immutable experiment
+manifest containing:
+
+- the historical full trial corpus and a content manifest;
+- exact topics, qrels, preprocessing outputs, and candidate pools;
+- model and adapter revisions with checksums;
+- the Elasticsearch version, mappings, analyzers, and index snapshot;
+- effective configuration and prompt templates for every stage;
+- deterministic seeds, runtime/container lock, and GPU details;
+- all per-topic run files, failures, timings, and figure-generation code.
+
+Until those inputs are recovered or rebuilt, call the command an **artifact
+audit**, not an exact end-to-end replication. New experiments should use the
+current evaluator and publish a complete run manifest; compare them with the
+paper only after labeling the changed corpus, engine, models, and metrics.

docs/trec-evaluation.md

@@ -0,0 +1,86 @@
+# Evaluating completed TREC runs
+
+TrialMatchAI can recompute ranking metrics from completed result directories
+without rebuilding an index or loading an LLM:
+
+```bash
+trialmatchai trec-evaluate \
+  --track 21 \
+  --results-dir h100_rrf/results_trec21 \
+  --results-dir qwen36_medcpt/results_trec21 \
+  --qrels data/trec/qrels/qrels_21.txt \
+  --include-per-query \
+  --output trec21-evaluation.json
+```
+
+The command evaluates both supported unjudged-document policies by default:
+
+- `exclude` removes unjudged trials before applying each rank cutoff. This is
+  the condensed-list convention previously used by the current evaluator.
+- `include_as_zero` preserves every ranked position and assigns unjudged trials
+  relevance grade zero. This is the conventional trec_eval interpretation.
+
+Both policies use the same tie-aware nDCG calculation. The report includes
+macro means and medians, every topic when `--include-per-query` is supplied,
+the qrels SHA-256, and a SHA-256 over every evaluated `ranked_trials.json` and
+`nct_ids.txt`. The two policies must never be compared without their names.
+
+## Existing completed analysis
+
+Six local configurations have complete TREC 2021 and 2022 results: 75 and 50
+topic directories respectively, each with first-stage candidates, criterion
+assessments, final rankings, and evaluation output. Re-running model inference
+would duplicate these expensive runs. The closest current-code analogue to the
+paper stack is `h100_rrf`: BGE-M3 retrieval, Gemma reranking, and Phi-4
+eligibility assessment. Its original effective configuration was not stored
+beside its outputs, so it is evidence from a completed experiment rather than a
+fully provenance-bound replication.
+
+The matching pipeline uses vLLM by default for the reranker and eligibility
+reasoner. vLLM batches many criterion pairs and trial prompts within a patient.
+Independent tracks and experiment arms can run as Slurm arrays on separate
+GPUs; tensor parallelism is available for models that need multiple GPUs. For
+these completed runs, metric evaluation is CPU-only and finishes faster than
+queueing a GPU job.
+
+## Measured policy sensitivity
+
+These values were regenerated from the complete local rankings on 19 September
+2026. `P@10` below is graded precision, `sum(relevance grade)/(2*10)`.
+
+| Track | Run | nDCG@10 exclude | nDCG@10 include-zero | P@10 exclude | P@10 include-zero | Unjudged top 10 |
+| :--- | :--- | ---: | ---: | ---: | ---: | ---: |
+| 2021 | h100_rrf | 0.790937 | 0.608344 | 0.775333 | 0.586667 | 26.3% |
+| 2021 | h100_medcpt | 0.798727 | 0.585323 | 0.782667 | 0.572000 | 28.3% |
+| 2021 | h100_baichuan | 0.822496 | 0.621061 | 0.802667 | 0.601333 | 26.1% |
+| 2021 | qwen36_medcpt | 0.821458 | 0.613718 | 0.812667 | 0.601333 | 27.9% |
+| 2021 | huatuo_medcpt | 0.681722 | 0.430457 | 0.676000 | 0.427333 | 39.1% |
+| 2021 | iimedical_medcpt | 0.753910 | 0.545734 | 0.737333 | 0.534667 | 29.6% |
+| 2022 | h100_rrf | 0.752821 | 0.653702 | 0.700000 | 0.603000 | 15.2% |
+| 2022 | h100_medcpt | 0.714433 | 0.625418 | 0.671000 | 0.584000 | 17.8% |
+| 2022 | h100_baichuan | 0.727787 | 0.622698 | 0.695000 | 0.589000 | 17.6% |
+| 2022 | qwen36_medcpt | 0.783205 | 0.691356 | 0.731000 | 0.639000 | 14.2% |
+| 2022 | huatuo_medcpt | 0.610370 | 0.490513 | 0.598000 | 0.486000 | 22.8% |
+| 2022 | iimedical_medcpt | 0.690797 | 0.576935 | 0.659000 | 0.547000 | 18.8% |
+
+The inclusive policy lowers nDCG@10 by 0.089–0.251 across these runs. This is
+expected because the systems retrieve trials outside the original judgment pool;
+the inclusive policy treats those trials as incorrect even though they were
+never assessed. The condensed policy avoids that assumption but measures
+ordering only among judged retrieved trials. Report both, and use recall-aware
+`ndcg_full@k` alongside them when retrieval coverage matters.
+
+The complete per-topic reports are tracked as
+`benchmarks/trec/unjudged-policy-trec21.json` and
+`benchmarks/trec/unjudged-policy-trec22.json`.
+
+## When another GPU run is warranted
+
+Launch a new Slurm inference job only for a configuration whose effective model,
+adapter, corpus, index, prompts, candidate budgets, or ranking code differs from
+an existing run. A production experiment should write that effective
+configuration, Git revision, model revisions, index fingerprint, Slurm job ID,
+GPU type, and output manifest into the result directory before processing the
+first topic. Use one array arm per independent track/configuration and vLLM
+batching within each arm. Resume incomplete patient and trial outputs rather
+than passing `--rematch` unless invalidation is intentional.

mkdocs.yml

@@ -61,6 +61,8 @@ nav:
   - Patient interoperability: interoperability.md
   - Registry updater: registry-updater.md
   - Fine-tuning: finetuning.md
+  - Paper reproduction: paper-reproduction.md
+  - TREC evaluation: trec-evaluation.md
   - API reference: api.md
   - Release process: release.md
   - Production roadmap: production-roadmap.md

pyproject.toml

@@ -78,7 +78,7 @@ finetune = [
 [project.scripts]
 # Single entry point. Every capability is a subcommand: `trialmatchai <cmd>`
 # (pipeline, healthcheck, bootstrap-data, index, build-concepts, update-registry,
-# import-patient, build, run, e2e, trec, report, finetune).
+# import-patient, build, run, e2e, trec, trec-evaluate, reproduce-paper, report, finetune).
 trialmatchai = "trialmatchai.cli.main:main"
 
 [project.urls]

src/trialmatchai/cli/main.py

@@ -64,6 +64,16 @@ def main() -> int:
         help="End-to-end TREC CT evaluation (preset over e2e)",
         add_help=False,
     )
+    subparsers.add_parser(
+        "reproduce-paper",
+        help="Verify and recalculate the published TREC 2021/2022 result artifact",
+        add_help=False,
+    )
+    subparsers.add_parser(
+        "trec-evaluate",
+        help="Re-evaluate completed TREC rankings under explicit unjudged policies",
+        add_help=False,
+    )
     subparsers.add_parser(
         "report",
         help="Render a self-contained HTML match report from a patient's results",
@@ -102,6 +112,10 @@ def main() -> int:
         from trialmatchai.cli.e2e import main as command
     elif args.command == "trec":
         from trialmatchai.cli.trec import main as command
+    elif args.command == "reproduce-paper":
+        from trialmatchai.cli.reproduce_paper import main as command
+    elif args.command == "trec-evaluate":
+        from trialmatchai.cli.trec_evaluate import main as command
     elif args.command == "report":
         from trialmatchai.cli.report import main as command
     elif args.command == "finetune":