diff --git a/.gitignore b/.gitignore index 034b5fd..90691e1 100644 --- a/.gitignore +++ b/.gitignore @@ -24,8 +24,12 @@ venv/ review_results/ benchmarks/conference_study/results/ benchmarks/perturbation/perturbation_results/ +# Perturbation benchmark run outputs (run_benchmark.py resolves results_dir here) +/results/ -# conference_study study artifacts (not code) +# conference_study study artifacts (not code). The full paper set regenerates +# via select_papers.py, so manifests/ is ignored. The frontier subset has no +# regeneration script; it is published as a Google Drive zip (see the READMEs). benchmarks/conference_study/manifests/ benchmarks/conference_study/reports/ # Symlink targets — trailing-slash patterns wouldn't match the symlinks diff --git a/README.md b/README.md index 4c6f61f..d7df253 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # OpenAIReview -[![PyPI version](https://img.shields.io/pypi/v/openaireview.svg)](https://pypi.org/project/openaireview/) +[PyPI version](https://pypi.org/project/openaireview/) Our goal is provide thorough and detailed reviews to help researchers conduct the best research. See more examples [here](https://openaireview.github.io/). -![Example](assets/example.png) +Example ## Installation @@ -14,11 +14,13 @@ uv venv && uv pip install openaireview ``` For fast PDF processing (requires `MISTRAL_API_KEY`): + ```bash uv pip install "openaireview[mistral]" ``` For development: + ```bash git clone https://github.com/ChicagoHAI/OpenAIReview.git cd OpenAIReview @@ -42,12 +44,14 @@ uv venv && uv pip install -e . PDF extraction quality matters — math symbols, tables, and reading order all affect review quality. Four engines are supported, tried in order: -| Engine | Install | Best for | Notes | -|--------|---------|----------|-------| -| **Mistral OCR** | `pip install "openaireview[mistral]"` + set `MISTRAL_API_KEY` | Best overall quality, math, tables | Cloud API, ~$0.001/page | -| **DeepSeek OCR** | `pip install "openaireview[deepseek]"` + local backend | Privacy-sensitive docs | Local model via Ollama/vLLM | -| **Marker** | `uv tool install marker-pdf --with psutil` | Math-heavy PDFs (offline) | Slow without GPU | -| **pymupdf4llm** | (included) | Fallback, always available | No math symbol support | + +| Engine | Install | Best for | Notes | +| ---------------- | ------------------------------------------------------------- | ---------------------------------- | --------------------------- | +| **Mistral OCR** | `pip install "openaireview[mistral]"` + set `MISTRAL_API_KEY` | Best overall quality, math, tables | Cloud API, ~$0.001/page | +| **DeepSeek OCR** | `pip install "openaireview[deepseek]"` + local backend | Privacy-sensitive docs | Local model via Ollama/vLLM | +| **Marker** | `uv tool install marker-pdf --with psutil` | Math-heavy PDFs (offline) | Slow without GPU | +| **pymupdf4llm** | (included) | Fallback, always available | No math symbol support | + The engine is auto-detected: if `MISTRAL_API_KEY` is set, Mistral OCR is tried first; then DeepSeek (if installed); then Marker (if on PATH); finally pymupdf4llm. You can force a specific engine with `--ocr`: @@ -96,34 +100,40 @@ openaireview serve Review an academic paper for technical and logical issues. Accepts a local file path or an arXiv URL. -| Option | Default | Description | -|---|---|---| -| `--method` | `progressive` | Review method: `zero_shot`, `local`, `progressive`, `progressive_full` | -| `--model` | `anthropic/claude-opus-4-6` | Model to use | -| `--provider` | (auto) | LLM provider: `openrouter`, `openai`, `anthropic`, `gemini`, `mistral` | -| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | -| `--max-pages` | (all) | Only process first N pages of a PDF (saves OCR cost) | -| `--max-tokens` | (all) | Truncate input text to first N tokens before review | -| `--output-dir` | `./review_results` | Directory for output JSON files | -| `--name` | (from filename) | Paper slug name | + +| Option | Default | Description | +| -------------- | --------------------------- | ---------------------------------------------------------------------- | +| `--method` | `progressive` | Review method: `zero_shot`, `local`, `progressive`, `progressive_full` | +| `--model` | `anthropic/claude-opus-4-6` | Model to use | +| `--provider` | (auto) | LLM provider: `openrouter`, `openai`, `anthropic`, `gemini`, `mistral` | +| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | +| `--max-pages` | (all) | Only process first N pages of a PDF (saves OCR cost) | +| `--max-tokens` | (all) | Truncate input text to first N tokens before review | +| `--output-dir` | `./review_results` | Directory for output JSON files | +| `--name` | (from filename) | Paper slug name | + ### `openaireview extract ` Run OCR extraction only and save as markdown with metadata frontmatter. Useful for a two-stage workflow: extract first, then review the markdown. -| Option | Default | Description | -|---|---|---| -| `-o`, `--output` | `.md` | Output markdown path | -| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | + +| Option | Default | Description | +| ---------------- | ----------- | ---------------------------------------------------------- | +| `-o`, `--output` | `.md` | Output markdown path | +| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | + ### `openaireview serve` Start a local visualization server to browse review results. -| Option | Default | Description | -|---|---|---| + +| Option | Default | Description | +| --------------- | ------------------ | -------------------------------------- | | `--results-dir` | `./review_results` | Directory containing result JSON files | -| `--port` | `8080` | Server port | +| `--port` | `8080` | Server port | + ## Supported Input Formats @@ -135,15 +145,17 @@ Start a local visualization server to browse review results. ## Environment Variables -| Variable | Default | Description | -|---|---|---| -| `OPENROUTER_API_KEY` | | OpenRouter API key (supports all models) | -| `OPENAI_API_KEY` | | OpenAI native API key | -| `ANTHROPIC_API_KEY` | | Anthropic native API key | -| `GEMINI_API_KEY` | | Google Gemini native API key | -| `MISTRAL_API_KEY` | | Mistral API key (also used for Mistral OCR) | -| `MODEL` | `anthropic/claude-opus-4-6` | Default model | -| `REVIEW_PROVIDER` | (auto) | Force a specific LLM provider | + +| Variable | Default | Description | +| -------------------- | --------------------------- | ------------------------------------------- | +| `OPENROUTER_API_KEY` | | OpenRouter API key (supports all models) | +| `OPENAI_API_KEY` | | OpenAI native API key | +| `ANTHROPIC_API_KEY` | | Anthropic native API key | +| `GEMINI_API_KEY` | | Google Gemini native API key | +| `MISTRAL_API_KEY` | | Mistral API key (also used for Mistral OCR) | +| `MODEL` | `anthropic/claude-opus-4-6` | Default model | +| `REVIEW_PROVIDER` | (auto) | Force a specific LLM provider | + Set one API key. The provider is auto-detected from whichever key is set (priority: OpenRouter > OpenAI > Anthropic > Gemini > Mistral). See `.env.example` for a template. @@ -151,12 +163,14 @@ Set one API key. The provider is auto-detected from whichever key is set (priori All models available on [OpenRouter](https://openrouter.ai) are supported — use any model ID via `--model`. The following models have built-in pricing for accurate cost tracking in the visualization: -| Model | Input ($/1M tokens) | Output ($/1M tokens) | -|---|---|---| -| `anthropic/claude-opus-4-6` | $5.00 | $25.00 | -| `anthropic/claude-opus-4-5` | $5.00 | $25.00 | -| `openai/gpt-5.2-pro` | $21.00 | $168.00 | -| `google/gemini-3.1-pro-preview` | $2.00 | $12.00 | + +| Model | Input ($/1M tokens) | Output ($/1M tokens) | +| ------------------------------- | ------------------- | -------------------- | +| `anthropic/claude-opus-4-6` | $5.00 | $25.00 | +| `anthropic/claude-opus-4-5` | $5.00 | $25.00 | +| `openai/gpt-5.2-pro` | $21.00 | $168.00 | +| `google/gemini-3.1-pro-preview` | $2.00 | $12.00 | + For models not listed above, a default rate of $5.00/$25.00 per 1M tokens is used. @@ -212,59 +226,50 @@ uv pip install -e ".[benchmarks]" export OPENROUTER_API_KEY=... ``` -Run scripts from inside each benchmark's directory unless noted. +Run scripts from inside each benchmark's directory unless otherwise noted. -### Outcomes study (`benchmarks/conference_study/`) +### Quality-proxy study (`benchmarks/conference_study/`) -Compares OpenAIReview output on accepted vs. rejected conference submissions. Papers are sampled via the 4-pair SNOR signal matrix (top-cited vs. never-published, awarded vs. rejected, top vs. bottom scores, and a composed pair). +Tests whether AI review systems comment more on weaker papers than on stronger ones. Papers are split into high- and low-quality groups by four quality proxies built from citations, venue awards, review scores, and a composite of the three. `configs/baseline.yaml` (our system) and `coarse.yaml` (coarse, an external review system) are the committed example configs. ```bash cd benchmarks/conference_study -# 1. Build manifests (manifests/v1/{pair_1..4,combined}.json) -python select_papers.py --venues iclr neurips --years 2021 2022 +# 1. Build the canonical paper manifest from SNOR (deterministic; downloads ~448 MB once) +python select_papers.py # writes manifests/canonical/ -# 2. Download PDFs flat under papers/scaleup/, write pages back into the manifest -python download_papers.py --source snor +# 2. Download PDFs into papers/, writing pages back into the manifest +python download_papers.py --source snor --manifest manifests/canonical/full.json -# 3. Optional cost preview (drops PDF parsing; estimate = pages × tokens_per_page × multipliers) -python estimate_cost.py --config configs/scaleup_progressive.yaml +# 3. Optional cost preview (estimate = pages × tokens_per_page × multipliers) +python estimate_cost.py --config configs/baseline.yaml -# 4. Run OpenAIReview and/or competitor systems on the same paper × model grid -python run_study.py --config configs/scaleup_progressive.yaml -python run_competitors.py --config configs/coarse_v2.yaml +# 4. Run our system and/or external systems on the same paper × model grid +python run_study.py --config configs/baseline.yaml +python run_competitors.py --config configs/coarse.yaml -# 5. Aggregate -python analyses/report_scaleup.py results/scaleup_progressive +# 5. Aggregate into the tables reported in the paper +python analyses/generate_report.py --config configs/baseline.yaml ``` -`run_study.py` and `run_competitors.py` are idempotent — rerunning skips paper × model combos already complete. Per-paper locks let multiple models share the same result JSON. See `benchmarks/conference_study/README.md` for the config schema, concurrency model, and result format. +The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontier subset the paper mainly reports on has no regeneration script; download it (bundled with the full manifest) from [Google Drive](https://drive.google.com/file/d/1mor434X2tTJKBYsNXgdpxPW7guzqbJAV/view?usp=sharing) (or `gdown 1mor434X2tTJKBYsNXgdpxPW7guzqbJAV`) and unzip inside `benchmarks/conference_study/`. `run_study.py` and `run_competitors.py` are idempotent, so rerunning skips paper × model combos already complete. See `benchmarks/conference_study/README.md` for more details, include how to run `coarse`. ### Perturbation benchmark (`benchmarks/perturbation/`) -Injects controlled errors (math edits, false claims, faulty reasoning, experimental flaws) into clean papers and measures per-comment recall by error type and domain. +Injects controlled errors (surface math edits, false claims, faulty reasoning, experimental flaws) into clean papers and measures recall by model, method, and error category. + +Pipeline: `extract → generate → validate → verify → inject → review → score`. -Pipeline: `extract → generate → validate → verify → inject → review → score`. `run_benchmark.py` drives all stages from a single YAML. +We ship the perturbed papers used in the paper, so the generation pipeline (everything through inject) is optional. The recommended way to reproduce is to download and unzip the released set ([Google Drive](https://drive.google.com/file/d/10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3/view?usp=sharing), or `gdown 10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3`) inside `benchmarks/perturbation/` so it lands at `data/perturbations_filtered/`, then run every paper domain at once with the committed per-domain configs: ```bash cd benchmarks/perturbation - -# One-shot: prepare papers, run reviews, score against the perturbation manifest -python run_benchmark.py configs/default.yaml - -# Or run a subset of stages -python run_benchmark.py configs/default.yaml --stages prepare,review -python run_benchmark.py configs/default.yaml --stages score - -# Multi-config sweep with parallel workers reused across configs -python run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 - -# Aggregate recall tables across all (paper, model, method) cells -python generate_report.py results/ +python run_benchmark.py --configs configs/full_*.yaml --stages prepare,review,score,report ``` -The config picks the review system per run via `system: openaireview | coarse | reviewer3`; adapter setup for third-party systems is in `systems/README.md`. Scoring uses a two-stage filter: a fuzzy substring match on the perturbed text against the comment quote, then an LLM judge rating (≥3/5) on whether the explanation identifies the same error. See `benchmarks/perturbation/README.md` for error-type taxonomy, results layout, and known limitations. +The config picks the review system per run via `system: openaireview | coarse | reviewer3` (adapter setup for the external systems is in `systems/README.md`). A comment counts as a detection when it passes two stages: a fuzzy substring match (the perturbed text covers the comment quote), then an LLM judge rating (a 1-5 score >= 3) on whether the explanation identifies the same error. + +To regenerate a fresh perturbed set instead, see `benchmarks/perturbation/README.md`, which documents the full generation pipeline, error taxonomy, config schema, dataset layout, and results layout. ## Related Resources @@ -273,4 +278,4 @@ The config picks the review system per run via `system: openaireview | coarse | ## License -[MIT](LICENSE) +[MIT](LICENSE) \ No newline at end of file diff --git a/benchmarks/README.md b/benchmarks/README.md deleted file mode 100644 index dfebe75..0000000 --- a/benchmarks/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# Benchmarks - -Reproducibility code for the two studies in the paper: - -- **Outcomes study** ([`conference_study/`](conference_study/)) — Does the proposed reviewer produce more (or qualitatively different) comments on rejected vs. accepted conference submissions? Papers are sampled via a 4-pair signal matrix (top-cited vs. never-published, awarded vs. rejected, top vs. bottom review scores, and a composed pair). -- **Perturbation benchmark** ([`perturbation/`](perturbation/)) — Can the proposed reviewer detect *seeded* errors injected into clean papers? Errors span four categories: surface math edits, false claims, faulty reasoning, and experimental-design flaws. Recall is reported per (model, method) cell. - -Both share a common setup. Throughout this document, the **system under evaluation** is the proposed reviewer (referred to as a "system" or "method" below); **competitor systems** are external review tools we compare against. - -## Setup - -```bash -uv pip install -e ".[benchmarks]" -export OPENROUTER_API_KEY=... # all model calls route through OpenRouter -``` - -Run scripts from inside each benchmark's directory. - -## Outcomes study - -Configs that produced the main-text tables: `conference_study/configs/scaleup_progressive.yaml` (proposed system) and `conference_study/configs/coarse_v2.yaml` (competitor). End-to-end: - -```bash -cd conference_study - -# 1. Build manifests (manifests/v1/{pair_1..4,combined}.json) -python select_papers.py --venues iclr neurips --years 2021 2022 - -# 2. Download PDFs flat under papers/scaleup/, write pages back into the manifest -python download_papers.py --source snor - -# 3. Optional cost preview (estimate = pages × tokens_per_page × method-specific multipliers) -python estimate_cost.py --config configs/scaleup_progressive.yaml - -# 4. Run the proposed system and/or competitor systems on the same paper × model grid -python run_study.py --config configs/scaleup_progressive.yaml -python run_competitors.py --config configs/coarse_v2.yaml - -# 5. Aggregate into the tables reported in the paper -python analyses/report_scaleup.py results/scaleup_progressive -``` - -Both runners are idempotent — rerunning skips (paper × model) combos already complete. Per-paper locks let multiple models share the same result JSON. See [`conference_study/README.md`](conference_study/README.md) for the config schema, concurrency model, and result format. - -## Perturbation benchmark - -Pipeline: `extract → generate → validate → verify → inject → review → score`. The first five stages produce a *corrupted paper* and a ground-truth manifest of injected errors; the last two stages run the reviewer and score its output. `run_benchmark.py` drives all stages from a single YAML. - -```bash -cd perturbation - -# One-shot: prepare papers, run reviews, score against the perturbation manifest -python run_benchmark.py configs/default.yaml - -# Or run a subset of stages (useful when iterating on scoring or rerunning a single model) -python run_benchmark.py configs/default.yaml --stages prepare,review -python run_benchmark.py configs/default.yaml --stages score - -# Multi-config sweep with parallel workers reused across configs -python run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 - -# Aggregate the recall-by-{model,method,error-type,domain} tables in the paper -python generate_report.py results/ -``` - -The `system:` field in each config selects the reviewer under test. Scoring uses a two-stage filter: (1) a fuzzy substring match on the perturbed text against each review comment's quote (after whitespace + math-delimiter normalization, with a 0.75 coverage threshold), then (2) an LLM judge rating (≥3/5) on whether the comment's explanation identifies the same error described in the perturbation's `why_wrong` field. See [`perturbation/README.md`](perturbation/README.md) for the error-type taxonomy, results layout, and known limitations. - -## Repository map - -``` -benchmarks/ -├── conference_study/ # outcomes study (§ above) -│ ├── select_papers.py # build the 4-pair signal-matrix manifest -│ ├── download_papers.py # fetch PDFs from OpenReview -│ ├── run_study.py # batch runner, proposed system -│ ├── run_competitors.py # batch runner, external systems -│ ├── analyses/ # report-generation scripts -│ └── configs/ # per-experiment YAMLs -└── perturbation/ # perturbation benchmark (§ above) - ├── run_benchmark.py # single-entry pipeline driver - ├── extract.py / generate.py / validate.py / verify.py / inject.py - ├── score.py / generate_report.py - ├── systems/ # adapters for the proposed system + competitors - └── configs/ # per-experiment YAMLs -``` - -The reviewer code itself is at the repository root (`src/`), exposed as a Python package and a CLI; both benchmarks invoke it through their adapter layers rather than calling its internals directly. diff --git a/benchmarks/conference_study/.gitignore b/benchmarks/conference_study/.gitignore index 215905b..58d4641 100644 --- a/benchmarks/conference_study/.gitignore +++ b/benchmarks/conference_study/.gitignore @@ -8,5 +8,6 @@ papers/ # Local configs — ignore all, un-ignore the canonical ones. configs/*.yaml !configs/baseline.yaml +!configs/frontier.yaml !configs/coarse.yaml !configs/reviewer3.yaml diff --git a/benchmarks/conference_study/README.md b/benchmarks/conference_study/README.md index c696800..51052c5 100644 --- a/benchmarks/conference_study/README.md +++ b/benchmarks/conference_study/README.md @@ -1,177 +1,170 @@ -# Conference Study: Accepted vs Rejected +# Quality-Proxy Study -**Research question:** Does OpenAIReview's progressive method produce more (or -different) comments on rejected papers than on accepted ones? +Do AI review systems comment more on weaker papers than on stronger ones? +This study tests whether comment volume tracks paper quality on real +ICLR/NeurIPS papers. Paper quality has no gold-standard label, so we split +papers into high- and low-quality groups using four noisy **quality proxies** +built from citation, award, and review-score signals, and check whether each +system produces more comments on the low-quality group. -**Venue:** ICLR 2024. The only major ML venue where rejected submissions are -publicly available on OpenReview. NeurIPS, ICML, CVPR etc. hide rejected -papers, so this study can't easily port to them. +The paper has the full method, metric, and results +([arXiv:2606.19749](https://arxiv.org/abs/2606.19749), Section "AI Review +Systems Correlate with Human Quality Signals"). This README covers how to +reproduce the runs. -## Directory layout - -``` -configs/ # YAML run configs (one per experiment) -reports/ # Markdown write-ups (one per experiment) -results/ # Output JSONs + run_log.jsonl (gitignored) -papers/ # Downloaded PDFs (gitignored; regenerate via download_papers.py) - accepted/ # 5 ICLR 2024 Outstanding Paper Award winners - rejected/ # 5 substantive rejections (ratings 2.5–3.5, ≥3 reviewers) -competitors/ # Adapters for external review systems (coarse, etc.) -download_papers.py # Fetches PDFs + writes manifest.json -estimate_cost.py # Pre-run USD cost estimate (progressive method) -run_study.py # Batch runner for OpenAIReview progressive (multi-model, multi-paper) -run_competitors.py # Batch runner for competitor systems (same paper/model grid) -generate_report.py # Auto-detects systems and emits per-system summary tables -manifest.json # Curated paper list (forum IDs, titles, groups) -``` +## The four quality proxies -One experiment = `configs/.yaml` + `reports/.md` + -`results//`. The triad shares a name. +Each proxy splits papers into a high-quality and a low-quality group of 30, +drawn from ICLR/NeurIPS 2021-2022 papers with at least three reviews and a +non-null average score: -## Workflow +- **Community-level**: top 30 by citations-per-year vs 30 rejected papers never published elsewhere. +- **Conference-level**: 30 papers highlighted by the venue (Outstanding, Best, Oral, Spotlight) vs 30 rejected papers. +- **Reviewer-level**: top 30 by mean review score vs bottom 30. +- **Composite**: top 30 awarded papers by combined citation-and-score rank vs bottom 30 rejected, never-published papers by review score. -### 1. Download papers (one-time) - -PDFs are gitignored. Download them before the first run: - -```bash -python download_papers.py -``` +That gives 60 papers per proxy and 240 in total (197 unique, since some papers +satisfy more than one criterion). Frontier models too expensive to run on the +full set use a **frontier subset** of 74 unique papers, which the paper mainly +reports on. -If `manifest.json` exists, the script downloads the exact papers listed -there, skipping any already on disk. The manifest is never modified. +Citations, awards, and review scores are noisy proxies of quality. We pick the +top and bottom groups as a tractable approximation, not as ground-truth quality. -**Adding more papers** to an existing manifest: +## Reproducing the paper set -```bash -python download_papers.py --add -n 5 # 5 more per group (papercopilot) -python download_papers.py --add -n 3 --group rejected # 3 more rejected only -``` +Selection draws from SNOR ([Zenodo 15866613](https://zenodo.org/records/15866613)), +which links OpenReview submissions to Semantic Scholar citation counts, +decisions, and reviewer scores. -**Bootstrapping a new study** (no manifest yet): +**Full set (197 papers)** regenerates from code: ```bash -python download_papers.py -n 10 # papercopilot, ICLR 2024 -python download_papers.py --source hf --venue ICLR --year 2023 -n 10 # HuggingFace -python download_papers.py --source hf --venue NeurIPS --year 2022 -n 10 +cd conference_study +python select_papers.py # downloads SNOR (~448 MB, cached), writes manifests/canonical/ ``` -Two data sources via `--source`: - -- **papercopilot** (default): ICLR only, uses explicit accept/reject decisions. -- **hf**: [AlgorithmicResearchGroup/openreview-papers-with-reviews](https://huggingface.co/datasets/AlgorithmicResearchGroup/openreview-papers-with-reviews) - on HuggingFace. Multi-venue (ICLR, NeurIPS, CoRL, UAI, MIDL), selects - by review score thresholds (`--accepted-threshold`, `--rejected-threshold`) - since the dataset has no explicit decisions. - -### 2. Estimate cost before a run - -```bash -python estimate_cost.py -``` +Selection is deterministic (fixed seed, forum-id tiebreak), so this reproduces +the exact 197-paper set used in the paper. -Uses observed progressive-method token multipliers (~6× input for the chain -of running-summary + window replay, ~0.15× output). Sanity-check before -burning API credits. +**Frontier subset (74 papers)** is a fixed subsample of the full set with no +regeneration script. Download it (bundled with the full manifest) from +[Google Drive](https://drive.google.com/file/d/1mor434X2tTJKBYsNXgdpxPW7guzqbJAV/view?usp=sharing) +(or `gdown 1mor434X2tTJKBYsNXgdpxPW7guzqbJAV`) and unzip inside `conference_study/` +so the files land at `manifests/canonical/`. The same archive includes +`full.json`, so you can use it to skip the SNOR download above. -### 3. Run an experiment +## Run flow ```bash -export OPENROUTER_API_KEY=... -python run_study.py --config configs/baseline.yaml -``` +cd conference_study +export OPENROUTER_API_KEY=... # all model calls route through OpenRouter -Writes results to `results//.json` and a run log to -`results//run_log.jsonl`. Idempotent — rerunning skips paper/model -combos already complete. +# 1. Build the full paper manifest. +python select_papers.py -**Useful flags:** +# 2. Download the PDFs listed in a manifest. Add --limit 5 for a cheap end-to-end smoke test first. +python download_papers.py --source snor --manifest manifests/canonical/full.json -- `--dry-run` — print the commands without calling the API. -- `--paper ` / `--model ` — restrict to one paper or model. -- `--force` — re-run even if already complete. -- `--max-pages` / `--max-tokens` / `--timeout-sec` / `--max-per-model` — ad-hoc -overrides of any YAML value. +# 3. Preview the API cost before the full run. estimate_cost.py prints a per-model +# breakdown; the full efficient set costs ~$25-50. +python estimate_cost.py --config configs/baseline.yaml -### 4. Run a competitor system (optional) +# 4. Run the systems on the same paper x model grid. +python run_study.py --config configs/baseline.yaml # our system, efficient models, full set +python run_competitors.py --config configs/coarse.yaml # coarse, in its own venv (see "Running coarse") -Competitors are external review systems wrapped via adapters in -`competitors/`. They run on the same paper × model grid as `run_study.py` -and their outputs merge into the same result-JSON schema, so the report -tooling handles them alongside OpenAIReview's progressive. - -```bash -python run_competitors.py --config configs/coarse.yaml +# 5. Build the paper's tables (pairwise accuracy + 95% CIs) from the results. +# Pass several configs to compare systems in one set of tables. +python analyses/generate_report.py --config configs/baseline.yaml configs/coarse.yaml ``` -Each competitor has its own config (e.g. `configs/coarse.yaml`) with a -`competitor:` field naming the adapter. Adapters live in -`competitors/_adapter.py` and are registered in -`competitors/registry.py`. See that file's docstring for how to add a new -one. +Start with the `--limit 5` smoke run in step 2 plus one model to confirm the +pipeline end to end for a few cents before launching the full set. Both runners +are idempotent: rerunning skips (paper, model) combos already complete. -Competitor outputs get their own `results//` directory, -parallel to `results//`. To report on both, point -`generate_report.py` at the combined results directory or at each -separately. +To reproduce the paper's frontier-subset numbers (all six models, including the +two frontier ones), use `manifests/canonical/frontier.json` from the download +above, then run steps 4 and 5 with `configs/frontier.yaml` in place of +`baseline.yaml`. -### 5. Generate report tables +## Running coarse + +`coarse` (PyPI `coarse-ink`) has a large, version-sensitive dependency set, so +it runs in its own virtual environment and the adapter subprocess-calls into it. +Set it up once: ```bash -python generate_report.py --config configs/baseline.yaml -python generate_report.py --config configs/baseline.yaml --papers # include papers table (parses PDFs, slow) +uv venv /path/to/coarse-venv +uv pip install --python /path/to/coarse-venv coarse-ink +export COARSE_VENV_PYTHON=/path/to/coarse-venv/bin/python ``` -Auto-detects every system present in the result JSONs (e.g. `progressive`, -`coarse`) by scanning method prefixes, and emits a per-system block of -tables (overall, per-model, consolidation if applicable, cost) plus a -runtime table spanning all systems. Pipe to a file or paste into -`reports/.md`. +`run_competitors.py --config configs/coarse.yaml` finds the venv through +`COARSE_VENV_PYTHON` (or a `venv_python:` field in the config). coarse runs on +the same paper x model grid, and its output merges into the same result schema +as the other systems. -### 6. Add a new experiment +## Metric -```bash -cp configs/baseline.yaml configs/my_experiment.yaml -# edit name, caps, parallelism as needed -python run_study.py --config configs/my_experiment.yaml -python generate_report.py --config configs/my_experiment.yaml -# write up findings in reports/my_experiment.md -``` +For each (system, model, proxy) we compare mean comments on the high- and +low-quality groups and report **pairwise accuracy**: the fraction of (low, +high) paper pairs where the low-quality paper received more comments (ties +count 0.5). A value of 0.5 means no separation, and above 0.5 means the system +tracks the proxy direction. Confidence intervals come from a cluster bootstrap +over papers (`analyses/ci_auc.py`); see the paper's appendix for the +construction. ## Config schema ```yaml -name: baseline # Results -> results// -max_pages: 20 # Passed to openaireview CLI -max_tokens: 20000 # Passed to openaireview CLI -timeout_sec: 3600 # Per-subprocess timeout -max_per_model: 2 # Concurrent runs per model +name: baseline # results -> results// +manifest: manifests/canonical/full.json # paper set to run on +models: # backbone models (falls back to the manifest's list if omitted) + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b +max_pages: 20 # passed to the openaireview CLI +max_tokens: 20000 # passed to the openaireview CLI +timeout_sec: 3600 # per-subprocess timeout +max_per_model: 2 # concurrent runs per model ``` -Precedence: **CLI flag > YAML value > built-in default**. CLI flag names -mirror YAML keys (hyphens for underscores). +## Directory layout -## Concurrency model +``` +configs/ # run configs (baseline = full set, frontier = 74-subset, coarse = external system) +manifests/ # paper set (canonical/full.json regenerates via select_papers.py) +papers/ # downloaded PDFs (gitignored; regenerate via download_papers.py) +results/ # output JSONs + run_log.jsonl (gitignored) +competitors/ # adapters for external review systems +analyses/ # report + AUC + confidence-interval scripts +select_papers.py # build the canonical manifest from SNOR +download_papers.py # fetch PDFs listed in a manifest +run_study.py # batch runner for our system +run_competitors.py # batch runner for external review systems +``` -Each model gets its own queue + `max_per_model` worker threads — keeps -OpenRouter per-model rate limits isolated. Per-paper locks prevent two -models from clobbering the same result JSON during the CLI's -read-modify-write merge. Total in-flight = `max_per_model × num_models` -(default 6 for 3 models). +## Output structure -## Results format +Each run config writes to its own results directory: -Each `results//.json` contains a `methods` dict keyed by -`__`. OpenAIReview's progressive writes two keys: +``` +results// + .json # one file per paper, with every model and system merged in + run_log.jsonl # one line per (paper, model) run: timing, exit code, output tails +``` -- `progressive__` — comments after the consolidation pass. -- `progressive_original__` — raw comments before consolidation. +Each `.json` holds the paper identity (`slug`, `title`), the parsed +`paragraphs` the systems reviewed, and a `methods` dict keyed by +`__`: -Both are written in a single CLI invocation. `run_study.py` considers a -(paper, model) combo "done" only when both keys exist. +``` +methods: + progressive__ # OpenAIReview comments after consolidation + progressive_original__ # OpenAIReview comments before consolidation + coarse__ # one key per external system run +``` -Competitor systems typically write a single post-editorial key -`__` (no raw/consolidated pair). The report -generator scans the method keys to decide which tables apply to which -system — systems without a `_original` partner skip the consolidation -table. \ No newline at end of file +Each method entry has `label`, `model`, `overall_feedback`, `cost_usd`, +`prompt_tokens`, `completion_tokens`, and a `comments` list. Each comment has a +`title`, `quote`, `explanation`, `paragraph_index`, and `severity`. \ No newline at end of file diff --git a/benchmarks/conference_study/analyses/compute_auc.py b/benchmarks/conference_study/analyses/compute_auc.py index 3ff04a6..11293ee 100644 --- a/benchmarks/conference_study/analyses/compute_auc.py +++ b/benchmarks/conference_study/analyses/compute_auc.py @@ -1,5 +1,5 @@ #!/usr/bin/env python -"""Mann-Whitney AUC for the conference outcomes study. +"""Mann-Whitney AUC for the quality-proxy study. For each (method, model) found in the supplied result directories, computes: @@ -190,7 +190,7 @@ def main() -> int: """CLI: load the manifest and result dirs, compute each (method, model) cell's point summary, and print them as a markdown table.""" p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--manifest", required=True, type=Path, help="Path to manifest JSON (e.g., manifests/v2/combined.json).") + p.add_argument("--manifest", required=True, type=Path, help="Path to manifest JSON (e.g., manifests/canonical/full.json).") p.add_argument("--dir", nargs="+", required=True, dest="dirs", help="One or more result subdirectories under results/ (e.g., scaleup_v2_progressive).") p.add_argument("--method", default=None, help="Filter to one method (e.g., progressive_original, zero_shot, coarse).") diff --git a/benchmarks/conference_study/analyses/generate_report.py b/benchmarks/conference_study/analyses/generate_report.py index 99e5b51..ce2bf6c 100644 --- a/benchmarks/conference_study/analyses/generate_report.py +++ b/benchmarks/conference_study/analyses/generate_report.py @@ -1,536 +1,108 @@ #!/usr/bin/env python -"""Compute summary tables from conference study results. - -Reads the result JSONs and run log to produce the markdown tables found in -reports/baseline.md. Can optionally parse PDFs to compute per-paper page -counts and effective token counts (requires papers/ on disk). +"""Produce the paper's quality-proxy tables from a run's results. + +Reads the result JSONs for a run (selected by --config or --results-dir) and the +manifest's high/low group memberships, then prints the pairwise-accuracy +(Mann-Whitney AUC) tables with 95% cluster-bootstrap confidence intervals: the +per-model comment-volume table, the per-quality-proxy table, and the +per-severity-tier table. + +The (method, model) cells are discovered from whatever is present in the +results, so no separate table config is needed: run the systems (step 4), then +point this at their results directories. Pass several to compare systems in one +table. By default it shows the manifest's roster models and openaireview's +pre-consolidation progressive output (the variant the paper reports). Use +--all-models and --consolidated to change that. The AUC and bootstrap math lives +in compute_auc.py and ci_auc.py, and this script is the reporting entry point +over them. Usage: - python generate_report.py # legacy results/ - python generate_report.py --config configs/baseline.yaml # results// - python generate_report.py --results-dir results/baseline # explicit + python analyses/generate_report.py --config configs/baseline.yaml + # merge systems into one set of tables + python analyses/generate_report.py --config configs/baseline.yaml configs/coarse.yaml + python analyses/generate_report.py --results-dir results/baseline results/coarse \ + --manifest manifests/canonical/full.json """ from __future__ import annotations import argparse import json import sys -from collections import defaultdict from pathlib import Path import yaml -HERE = Path(__file__).resolve().parent -REPO = HERE.parent.parent - -# Models in display order for the baseline three. Any model not in this list -# still appears (sorted alphabetically after these). -MODEL_ORDER = ["gemini-3-flash-preview", "glm-4.6", "qwen3-235b-a22b-2507"] - - -def ordered_models(models: set[str]) -> list[str]: - """MODEL_ORDER first, then any others sorted.""" - head = [m for m in MODEL_ORDER if m in models] - tail = sorted(m for m in models if m not in MODEL_ORDER) - return head + tail - - -# --------------------------------------------------------------------------- -# System detection -# --------------------------------------------------------------------------- -# -# A "system" is a review producer (e.g. `progressive`, `coarse`). Each system -# writes methods of the form `__` into result JSONs. Some -# systems (openaireview progressive) also emit a raw-vs-consolidated pair -# (`progressive_original__` alongside `progressive__`); others -# (competitors) emit only a single prefix. Detection: scan the actual method -# keys and pair up any `*_original` with its base. - -def detect_systems(results: dict[str, dict]) -> list[dict]: - """Return [{name, cons_prefix, raw_prefix | None}, ...] detected from JSONs.""" - prefixes: set[str] = set() - for res in results.values(): - for key in res.get("methods", {}): - if "__" in key: - prefixes.add(key.split("__", 1)[0]) - - systems = [] - for p in sorted(prefixes): - if p.endswith("_original"): - continue # shown via its consolidated partner's shrink column - raw = f"{p}_original" - systems.append({ - "name": p, - "cons_prefix": p, - "raw_prefix": raw if raw in prefixes else None, - }) - return systems - - -# --------------------------------------------------------------------------- -# Config loading (same pattern as run_study.py / estimate_cost.py) -# --------------------------------------------------------------------------- - -def load_config(path: str) -> dict: - if not path: - return {} - cfg_path = Path(path) - if not cfg_path.is_absolute(): - cfg_path = cfg_path if cfg_path.exists() else HERE / path - if not cfg_path.exists(): - sys.exit(f"config file not found: {path}") - with cfg_path.open() as f: - return yaml.safe_load(f) or {} - - -# --------------------------------------------------------------------------- -# Data loading -# --------------------------------------------------------------------------- - -def load_results(results_dir: Path, manifest: dict) -> dict[str, dict]: - """Load result JSONs for all papers in the manifest.""" - results = {} - for p in manifest["papers"]: - slug = p["slug"] - path = results_dir / f"{slug}.json" - if not path.exists(): - print(f" warning: missing {path}", file=sys.stderr) - continue - results[slug] = json.loads(path.read_text()) - return results - - -def load_run_log(results_dir: Path) -> list[dict]: - """Load run_log.jsonl entries.""" - log_path = results_dir / "run_log.jsonl" - if not log_path.exists(): - return [] - entries = [] - for line in log_path.read_text().splitlines(): - if line.strip(): - entries.append(json.loads(line)) - return entries - - -def model_short(model: str) -> str: - return model.split("/")[-1] if "/" in model else model - - -def paper_group(slug: str, manifest: dict) -> str: - for p in manifest["papers"]: - if p["slug"] == slug: - return p["group"] - return "unknown" - - -# --------------------------------------------------------------------------- -# Papers table (requires PDFs on disk) -# --------------------------------------------------------------------------- - -def compute_papers_table(manifest: dict, max_pages: int, max_tokens: int) -> list[dict]: - """Parse PDFs to get page counts and effective token counts.""" - sys.path.insert(0, str(REPO / "src")) - from reviewer.parsers import parse_document - from reviewer.utils import count_tokens - - rows = [] - for p in manifest["papers"]: - slug = p["slug"] - pdf = HERE / "papers" / p["group"] / f"{slug}.pdf" - if not pdf.exists(): - rows.append({"slug": slug, "group": p["group"], - "title": p["title"], "pages": None, "eff_tokens": None}) - continue - try: - import fitz - doc = fitz.open(pdf) - pages = len(doc) - doc.close() - except Exception: - pages = None - - try: - _title, content, _ocr = parse_document(pdf, max_pages=max_pages) - eff_tokens = min(count_tokens(content), max_tokens) - except Exception: - eff_tokens = None - - rows.append({"slug": slug, "group": p["group"], - "title": p["title"], "pages": pages, "eff_tokens": eff_tokens}) - return rows - - -# --------------------------------------------------------------------------- -# Comment counting helpers -# --------------------------------------------------------------------------- - -def count_comments(result: dict, method_prefix: str) -> dict[str, int]: - """Count comments per model for a given method prefix. - - Returns {model_short: total_count}. - """ - counts: dict[str, int] = {} - methods = result.get("methods", {}) - for key, data in methods.items(): - if not key.startswith(method_prefix + "__"): - continue - model = key[len(method_prefix) + 2:] - counts[model] = len(data.get("comments", [])) - return counts - - -def get_cost(result: dict, method_prefix: str) -> dict[str, float]: - """Return {model_short: cost_usd} for the given prefix.""" - costs: dict[str, float] = {} - methods = result.get("methods", {}) - for key, data in methods.items(): - if not key.startswith(method_prefix + "__"): - continue - model = key[len(method_prefix) + 2:] - costs[model] = data.get("cost_usd") or 0.0 - return costs - - -# --------------------------------------------------------------------------- -# Table builders -# --------------------------------------------------------------------------- - -def build_overall_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Overall: accepted vs rejected, for one system.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - groups: dict[str, dict] = { - "accepted": {"n": 0, "raw": 0, "consolidated": 0}, - "rejected": {"n": 0, "raw": 0, "consolidated": 0}, - } - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) if raw_pref else cons_counts - for model, cons in cons_counts.items(): - groups[group]["n"] += 1 - groups[group]["consolidated"] += cons - groups[group]["raw"] += raw_counts.get(model, cons) - - rows = [] - for g in ["accepted", "rejected"]: - d = groups[g] - n = d["n"] - if n == 0: - continue - raw_avg = d["raw"] / n - cons_avg = d["consolidated"] / n - shrink = 1 - cons_avg / raw_avg if raw_avg > 0 else 0 - rows.append({"group": g, "n": n, - "raw_total": d["raw"], "raw_avg": raw_avg, - "cons_total": d["consolidated"], "cons_avg": cons_avg, - "shrink": shrink}) - return rows +import ci_auc +from compute_auc import load_counts, load_manifest +HERE = Path(__file__).resolve().parent +STUDY = HERE.parent # benchmarks/conference_study -def build_per_model_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Per-model breakdown for one system.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - cells: dict[tuple[str, str], dict] = defaultdict( - lambda: {"n": 0, "raw": 0, "consolidated": 0} - ) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) if raw_pref else cons_counts - for model, cons in cons_counts.items(): - seen_models.add(model) - cells[(group, model)]["n"] += 1 - cells[(group, model)]["consolidated"] += cons - cells[(group, model)]["raw"] += raw_counts.get(model, cons) - - rows = [] - for group in ["accepted", "rejected"]: - for model in ordered_models(seen_models): - d = cells.get((group, model)) - if not d or d["n"] == 0: - continue - raw_avg = d["raw"] / d["n"] - cons_avg = d["consolidated"] / d["n"] - shrink = 1 - cons_avg / raw_avg if raw_avg > 0 else 0 - rows.append({"group": group, "model": model, "n": d["n"], - "raw_total": d["raw"], "raw_avg": raw_avg, - "cons_total": d["consolidated"], "cons_avg": cons_avg, - "shrink": shrink}) - return rows - - -def build_consolidation_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Consolidation shrink per model, split by group. Only meaningful when - the system exposes a raw_prefix.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - if not raw_pref: - return [] - cells: dict[tuple[str, str], dict] = defaultdict( - lambda: {"raw": 0, "consolidated": 0} - ) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) - for model in raw_counts: - seen_models.add(model) - cells[(group, model)]["raw"] += raw_counts[model] - cells[(group, model)]["consolidated"] += cons_counts.get(model, 0) - - rows = [] - for model in ordered_models(seen_models): - acc = cells.get(("accepted", model), {"raw": 0, "consolidated": 0}) - rej = cells.get(("rejected", model), {"raw": 0, "consolidated": 0}) - acc_shrink = 1 - acc["consolidated"] / acc["raw"] if acc["raw"] > 0 else 0 - rej_shrink = 1 - rej["consolidated"] / rej["raw"] if rej["raw"] > 0 else 0 - rows.append({"model": model, "acc_shrink": acc_shrink, "rej_shrink": rej_shrink}) - return rows - - -def build_cost_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Cost per model, split by group, for one system.""" - cells: dict[tuple[str, str], float] = defaultdict(float) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - costs = get_cost(res, system["cons_prefix"]) - for model, cost in costs.items(): - seen_models.add(model) - cells[(group, model)] += cost - - rows = [] - for model in ordered_models(seen_models): - acc = cells.get(("accepted", model), 0.0) - rej = cells.get(("rejected", model), 0.0) - rows.append({"model": model, "accepted": acc, "rejected": rej, - "total": acc + rej}) - return rows - - -def build_runtime_table(run_log: list[dict], manifest: dict) -> list[dict]: - """Runtime per model from run_log.jsonl.""" - manifest_slugs = {p["slug"] for p in manifest["papers"]} - # {model_short: [durations]} - durations: dict[str, list[float]] = defaultdict(list) - for entry in run_log: - if not entry.get("ok") or entry["slug"] not in manifest_slugs: - continue - model = model_short(entry["model"]) - durations[model].append(entry["duration_sec"]) - - rows = [] - for model in ordered_models(set(durations)): - durs = durations.get(model, []) - if not durs: - continue - avg_min = (sum(durs) / len(durs)) / 60 - min_min = min(durs) / 60 - max_min = max(durs) / 60 - rows.append({"model": model, "avg_min": avg_min, - "min_min": min_min, "max_min": max_min}) - return rows - - -# --------------------------------------------------------------------------- -# Markdown formatting -# --------------------------------------------------------------------------- - -def fmt_papers(rows: list[dict]) -> str: - lines = [] - lines.append("| Slug | Group | Title | Pages | Eff. tokens |") - lines.append("| --- | --- | --- | --- | --- |") - for r in rows: - pages = str(r["pages"]) if r["pages"] is not None else "N/A" - tokens = f"{r['eff_tokens']:,}" if r["eff_tokens"] is not None else "N/A" - lines.append(f"| {r['slug']} | {r['group']} | {r['title']} | {pages} | {tokens} |") - return "\n".join(lines) - - -def fmt_overall(rows: list[dict], has_raw: bool) -> str: - lines = [] - if has_raw: - lines.append("| Group | N runs | Raw (total) | Raw (avg) | Consolidated (total) | Consolidated (avg) | Shrink |") - lines.append("| --- | --- | --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| **{r['group'].title()}** | {r['n']} | {r['raw_total']} | {r['raw_avg']:.1f} " - f"| {r['cons_total']} | {r['cons_avg']:.1f} | {r['shrink']:.0%} |") - else: - lines.append("| Group | N runs | Comments (total) | Comments (avg) |") - lines.append("| --- | --- | --- | --- |") - for r in rows: - lines.append(f"| **{r['group'].title()}** | {r['n']} | " - f"{r['cons_total']} | {r['cons_avg']:.1f} |") - return "\n".join(lines) - - -def fmt_per_model(rows: list[dict], has_raw: bool) -> str: - lines = [] - if has_raw: - lines.append("| Group | Model | N | Raw (total) | Raw (avg) | Cons. (total) | Cons. (avg) | Shrink |") - lines.append("| --- | --- | --- | --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| {r['group']} | {r['model']} | {r['n']} | {r['raw_total']} | {r['raw_avg']:.1f} " - f"| {r['cons_total']} | {r['cons_avg']:.1f} | {r['shrink']:.0%} |") - else: - lines.append("| Group | Model | N | Comments (total) | Comments (avg) |") - lines.append("| --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| {r['group']} | {r['model']} | {r['n']} | " - f"{r['cons_total']} | {r['cons_avg']:.1f} |") - return "\n".join(lines) - - - -def fmt_consolidation(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Accepted shrink | Rejected shrink |") - lines.append("| --- | --- | --- |") - for r in rows: - # Use short display name - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | {r['acc_shrink']:.0%} | {r['rej_shrink']:.0%} |") - return "\n".join(lines) - - -def fmt_cost(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Accepted | Rejected | Total |") - lines.append("| --- | --- | --- | --- |") - grand_acc = grand_rej = grand_total = 0.0 - for r in rows: - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | ${r['accepted']:.2f} | ${r['rejected']:.2f} " - f"| ${r['total']:.2f} |") - grand_acc += r["accepted"] - grand_rej += r["rejected"] - grand_total += r["total"] - lines.append(f"| **Total** | **${grand_acc:.2f}** | **${grand_rej:.2f}** " - f"| **${grand_total:.2f}** |") - return "\n".join(lines) - +# (heading, ci_auc table kind) — printed in order over the discovered cells. +TABLES = [ + ("Comment volume and overall accuracy, per model", "comment_volume"), + ("Accuracy by quality proxy", "by_proxy"), + ("Accuracy by severity tier", "by_severity"), +] -def fmt_runtime(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Avg per run | Range |") - lines.append("| --- | --- | --- |") - for r in rows: - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | {r['avg_min']:.1f} min " - f"| {r['min_min']:.1f}-{r['max_min']:.1f} min |") - return "\n".join(lines) +def _resolve(path: Path, base: Path) -> Path: + return path if path.is_absolute() else (base / path) -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- def main() -> None: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--config", help="YAML config file (same schema as run_study.py).") - ap.add_argument("--results-dir", help="Explicit results directory.") - ap.add_argument("--papers", action="store_true", - help="Include papers table (parses PDFs, slow).") + ap.add_argument("--config", type=Path, nargs="+", + help="one or more run config YAMLs, merged (e.g. baseline.yaml coarse.yaml)") + ap.add_argument("--results-dir", type=Path, nargs="+", + help="one or more results directories, merged") + ap.add_argument("--manifest", type=Path, + help="manifest JSON (defaults to the first config's, then manifests/canonical/full.json)") + ap.add_argument("--all-models", action="store_true", + help="show every model found, not just the manifest's roster") + ap.add_argument("--consolidated", action="store_true", + help="show openaireview's consolidated progressive output instead of the " + "pre-consolidation progressive_original the paper reports") args = ap.parse_args() - - cfg = load_config(args.config) - name = cfg.get("name") - max_pages = cfg.get("max_pages", 20) - max_tokens = cfg.get("max_tokens", 20_000) - - if args.results_dir: - results_dir = Path(args.results_dir) - elif name: - results_dir = HERE / "results" / name - else: - results_dir = HERE / "results" - - if not results_dir.exists(): - sys.exit(f"results directory not found: {results_dir}") - - manifest = json.loads((HERE / "manifest.json").read_text()) - results = load_results(results_dir, manifest) - run_log = load_run_log(results_dir) - - if not results: - sys.exit("No result files found.") - - print(f"Loaded {len(results)} result files from {results_dir}\n") - - # --- Papers table --- - pdfs_available = any((HERE / "papers" / p["group"] / f"{p['slug']}.pdf").exists() - for p in manifest["papers"]) - if args.papers and pdfs_available: - try: - print("### Papers\n") - papers_rows = compute_papers_table(manifest, max_pages, max_tokens) - print(fmt_papers(papers_rows)) - # Summary stats - acc_pages = [r["pages"] for r in papers_rows if r["group"] == "accepted" and r["pages"]] - rej_pages = [r["pages"] for r in papers_rows if r["group"] == "rejected" and r["pages"]] - acc_tok = [r["eff_tokens"] for r in papers_rows if r["group"] == "accepted" and r["eff_tokens"]] - rej_tok = [r["eff_tokens"] for r in papers_rows if r["group"] == "rejected" and r["eff_tokens"]] - if acc_pages and rej_pages: - print(f"\nAccepted avg: {sum(acc_pages)/len(acc_pages):.1f} pages, " - f"~{sum(acc_tok)/len(acc_tok):,.0f} tokens") - print(f"Rejected avg: {sum(rej_pages)/len(rej_pages):.1f} pages, " - f"~{sum(rej_tok)/len(rej_tok):,.0f} tokens") - except Exception as e: - print(f"(Papers table skipped — {e})\n", file=sys.stderr) - elif args.papers: - print("(Papers table skipped — PDFs not on disk. Run download_papers.py first.)\n") - - # --- Per-system tables --- - systems = detect_systems(results) - if not systems: - print("\n(No methods found in result JSONs.)") - for system in systems: - name = system["name"] - has_raw = system["raw_prefix"] is not None - banner = f"System: {name}" - print(f"\n\n## {banner}\n") - - overall = build_overall_table(manifest, results, system) - if overall: - print(f"\n### Overall: accepted vs rejected ({name})\n") - print(fmt_overall(overall, has_raw)) - - per_model = build_per_model_table(manifest, results, system) - if per_model: - print(f"\n### Per-model breakdown ({name})\n") - print(fmt_per_model(per_model, has_raw)) - - if has_raw: - consolidation = build_consolidation_table(manifest, results, system) - if consolidation: - print(f"\n### Consolidation effect ({name})\n") - print(fmt_consolidation(consolidation)) - - cost = build_cost_table(manifest, results, system) - if cost and any(r["total"] for r in cost): - print(f"\n### Cost ({name})\n") - print(fmt_cost(cost)) - - # --- Runtime (single table spanning all systems; derived from log) --- - if run_log: - print("\n\n## Runtime (all systems)\n") - runtime = build_runtime_table(run_log, manifest) - print(fmt_runtime(runtime)) - else: - print("\n(Runtime table skipped — no run_log.jsonl found.)") + if not args.config and not args.results_dir: + ap.error("pass --config or --results-dir") + + configs = [yaml.safe_load(p.read_text()) for p in (args.config or [])] + + dirs = [_resolve(p, Path.cwd()) for p in (args.results_dir or [])] + dirs += [STUDY / "results" / cfg["name"] for cfg in configs] + missing = [str(d) for d in dirs if not d.exists()] + if missing: + sys.exit("results directory not found: " + ", ".join(missing)) + + default_manifest = configs[0].get("manifest") if configs else None + manifest = _resolve(args.manifest or Path(default_manifest or "manifests/canonical/full.json"), STUDY) + if not manifest.exists(): + sys.exit(f"manifest not found: {manifest}") + + # Discover the (method, model) cells across the result dirs. By default keep + # only the manifest's roster models (dropping exploratory runs that linger in + # the result dirs) and the consolidated progressive output. A model-less + # system keyed as __ (e.g. reviewer3) is always kept. + slug_to_mem = load_manifest(manifest) + roster = {m.rsplit("/", 1)[-1] for m in json.loads(manifest.read_text()).get("models", [])} + cells = sorted(load_counts(dirs, set(slug_to_mem))) + if roster and not args.all_models: + cells = [c for c in cells if c[1] in roster or c[1] == c[0]] + # openaireview writes two variants per run: progressive_original (raw, the + # variant the paper reports) and progressive (after consolidation). Show the + # paper's by default; --consolidated switches to the consolidated one. + drop_variant = "progressive_original" if args.consolidated else "progressive" + cells = [c for c in cells if c[0] != drop_variant] + if not cells: + sys.exit("no (method, model) results found under: " + ", ".join(str(d) for d in dirs)) + + print(f"Results: {', '.join(d.name for d in dirs)}\nManifest: {manifest.name} " + f"({len(slug_to_mem)} papers)\nCells: {len(cells)} (method, model)\n") + for heading, kind in TABLES: + print(f"\n########## {heading} ##########") + ci_auc.DISPATCH[kind](manifest, dirs, cells) if __name__ == "__main__": diff --git a/benchmarks/conference_study/analyses/report_scaleup.py b/benchmarks/conference_study/analyses/report_scaleup.py index 79dce76..771ecb8 100644 --- a/benchmarks/conference_study/analyses/report_scaleup.py +++ b/benchmarks/conference_study/analyses/report_scaleup.py @@ -2,7 +2,7 @@ """Pair-grouped report for the 4-pair signal-matrix scale-up. Reads: - manifests/combined.json paper list + pair_memberships + word_count + manifests/canonical/full.json paper list + pair_memberships + word_count results//.json methods dict with __ keys Emits one markdown table per pair showing high-tail vs low-tail comment @@ -27,7 +27,7 @@ from statistics import mean HERE = Path(__file__).resolve().parent -MANIFEST = HERE / "manifests" / "combined.json" +MANIFEST = HERE / "manifests" / "canonical" / "full.json" RESULTS_BASE = HERE / "results" BASELINE_MANIFEST = HERE / "manifest.json" diff --git a/benchmarks/conference_study/competitors/coarse_adapter.py b/benchmarks/conference_study/competitors/coarse_adapter.py index 03634fb..7f57307 100644 --- a/benchmarks/conference_study/competitors/coarse_adapter.py +++ b/benchmarks/conference_study/competitors/coarse_adapter.py @@ -65,13 +65,14 @@ def method_key(self, model: str) -> str: return f"coarse__{model_short(model)}" def review(self, pdf: Path, model: str, cfg: dict) -> NormalizedReview: - venv_python = cfg.get("venv_python") + venv_python = cfg.get("venv_python") or os.environ.get("COARSE_VENV_PYTHON") if not venv_python: raise RuntimeError( - "coarse adapter requires `venv_python` in config " - "(path to the coarse venv's python executable)" + "coarse adapter needs the coarse venv's python: set the " + "COARSE_VENV_PYTHON env var, or `venv_python` in the config. " + "See benchmarks/conference_study/README.md for coarse setup." ) - venv_python = str(Path(venv_python).expanduser()) + venv_python = str(Path(os.path.expandvars(venv_python)).expanduser()) if not Path(venv_python).exists(): raise RuntimeError(f"coarse venv python not found: {venv_python}") diff --git a/benchmarks/conference_study/configs/baseline.yaml b/benchmarks/conference_study/configs/baseline.yaml index 8dfdb7c..d25e05e 100644 --- a/benchmarks/conference_study/configs/baseline.yaml +++ b/benchmarks/conference_study/configs/baseline.yaml @@ -1,9 +1,24 @@ -# Baseline run config for the ICLR accept-vs-reject study. +# Baseline run config for the quality-proxy study. # Results -> benchmarks/conference_study/results/baseline/ # Log -> benchmarks/conference_study/results/baseline/run_log.jsonl name: baseline +# Paper set. Regenerate the full canonical manifest with select_papers.py +# (deterministic, seed 42), or point at manifests/canonical/frontier.json +# for the 74-paper frontier subset. +manifest: manifests/canonical/full.json + +# Backbone models. These four efficient models run on the full set. The two +# frontier models (openai/gpt-5.5, anthropic/claude-opus-4.7) run only on the +# frontier subset; for that run, point `manifest` at +# manifests/canonical/frontier.json and add them here. +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + # Review caps passed to the openaireview CLI. max_pages: 20 max_tokens: 20000 diff --git a/benchmarks/conference_study/configs/coarse.yaml b/benchmarks/conference_study/configs/coarse.yaml index 258a6c4..0fdd3b4 100644 --- a/benchmarks/conference_study/configs/coarse.yaml +++ b/benchmarks/conference_study/configs/coarse.yaml @@ -4,16 +4,23 @@ # # Prerequisites: # - coarse installed in its own venv at `venv_python` below -# - OPENROUTER_API_KEY set in env (coarse routes all three manifest models +# - OPENROUTER_API_KEY set in env (coarse routes all backbone models # through OpenRouter) name: coarse competitor: coarse -# Path to the coarse venv's python. The adapter subprocess-calls a runner -# inside this venv so coarse's deps (litellm, instructor, docling, ...) stay -# isolated from openaireview's venv. -venv_python: /data/dangnguyen/openaireview_project/coarse/.venv/bin/python +# Backbone models (same efficient roster as the baseline run). +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + +# The adapter subprocess-calls a runner inside coarse's own venv so its deps +# (litellm, instructor, docling, ...) stay isolated from openaireview's. Point +# the COARSE_VENV_PYTHON env var at that venv's python (see the README section +# "Running the coarse competitor"). To hardcode it instead, set `venv_python:`. # Batch runner knobs. timeout_sec: 1800 # 30 min per (paper, model) run (coarse is 2-5 min typical) diff --git a/benchmarks/conference_study/configs/frontier.yaml b/benchmarks/conference_study/configs/frontier.yaml new file mode 100644 index 0000000..46f1bff --- /dev/null +++ b/benchmarks/conference_study/configs/frontier.yaml @@ -0,0 +1,26 @@ +# Frontier-subset run config for the quality-proxy study. +# Runs all six backbone models on the 74-paper frontier subset, the set the +# paper mainly reports on. The two frontier models (gpt-5.5, claude-opus-4.7) +# are too expensive for the full set, so they appear only here. +# The frontier manifest is not shipped; request it from the authors. +# Results -> benchmarks/conference_study/results/frontier/ + +name: frontier + +manifest: manifests/canonical/frontier.json + +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + +# Review caps passed to the openaireview CLI. +max_pages: 20 +max_tokens: 20000 + +# Batch runner knobs. +timeout_sec: 3600 +max_per_model: 2 diff --git a/benchmarks/conference_study/download_papers.py b/benchmarks/conference_study/download_papers.py index a38891e..2db9aaf 100644 --- a/benchmarks/conference_study/download_papers.py +++ b/benchmarks/conference_study/download_papers.py @@ -7,7 +7,7 @@ on HuggingFace. Multi-venue (ICLR, NeurIPS, CoRL, etc.), uses review score thresholds instead of decisions. --source snor Scale-up 4-pair signal matrix. Reads - manifests/combined.json (produced by select_papers.py). + manifests/canonical/full.json (produced by select_papers.py). Downloads PDFs to papers/scaleup/.pdf and updates the manifest with pdf_path + pages. @@ -590,7 +590,7 @@ def main() -> None: default="papercopilot", help="Data source (default: papercopilot).") ap.add_argument("--manifest", type=Path, - help="[snor] Path to combined.json (default: manifests/combined.json).") + help="[snor] Path to the manifest (default: manifests/canonical/full.json).") ap.add_argument("--limit", type=int, help="[snor] Only download the first N manifest entries " "(for smoke tests).") @@ -615,11 +615,11 @@ def main() -> None: manifest_path = out_dir / "manifest.json" cache_dir = out_dir / ".cache" - # snor source has its own flow: read combined.json, download flat. + # snor source has its own flow: read full.json, download flat. if args.source == "snor": snor_manifest = (args.manifest if args.manifest - else out_dir / "manifests" / "v1" / "combined.json") + else out_dir / "manifests" / "canonical" / "full.json") # The papercopilot/hf default of 15 pages filters accidentally # tiny papers. SNOR already selects from real conference decisions, # so accept anything at least 6 pages. User can override explicitly. diff --git a/benchmarks/conference_study/estimate_cost.py b/benchmarks/conference_study/estimate_cost.py index 920147d..218d75a 100644 --- a/benchmarks/conference_study/estimate_cost.py +++ b/benchmarks/conference_study/estimate_cost.py @@ -73,6 +73,9 @@ "openai/gpt-5-mini": {"prompt": 0.25e-6, "completion": 2.00e-6}, "openai/gpt-5.4-mini": {"prompt": 0.75e-6, "completion": 4.50e-6}, "openai/gpt-5.5": {"prompt": 5.00e-6, "completion": 30.00e-6}, + # TODO: add the OpenRouter rate for x-ai/grok-4.1-fast (part of the efficient + # roster). Until it is filled in, estimate_cost skips Grok and the printed + # total excludes it. } diff --git a/benchmarks/conference_study/paper_metrics.py b/benchmarks/conference_study/paper_metrics.py index 461bb78..6fbda51 100644 --- a/benchmarks/conference_study/paper_metrics.py +++ b/benchmarks/conference_study/paper_metrics.py @@ -20,7 +20,7 @@ from pathlib import Path HERE = Path(__file__).resolve().parent -DEFAULT_MANIFEST = HERE / "manifests" / "combined.json" +DEFAULT_MANIFEST = HERE / "manifests" / "canonical" / "full.json" # Make the openaireview package importable when running from this dir. sys.path.insert(0, str(HERE.parent.parent / "src")) diff --git a/benchmarks/conference_study/run_study.py b/benchmarks/conference_study/run_study.py index d9a34c6..31fbc7a 100644 --- a/benchmarks/conference_study/run_study.py +++ b/benchmarks/conference_study/run_study.py @@ -235,7 +235,7 @@ def main() -> None: RESULTS_DIR = DEFAULT_RESULTS_BASE / name if name else DEFAULT_RESULTS_BASE LOG_FILE = RESULTS_DIR / "run_log.jsonl" - manifest_path = args.manifest or Path(cfg.get("manifest", "manifest.json")) + manifest_path = args.manifest or Path(cfg.get("manifest", "manifests/canonical/full.json")) if not manifest_path.is_absolute(): manifest_path = HERE / manifest_path manifest = json.loads(manifest_path.read_text()) diff --git a/benchmarks/conference_study/select_papers.py b/benchmarks/conference_study/select_papers.py index 98cd7ab..df0afd9 100644 --- a/benchmarks/conference_study/select_papers.py +++ b/benchmarks/conference_study/select_papers.py @@ -1,10 +1,10 @@ #!/usr/bin/env python """Select papers for the 4-pair signal matrix from SNOR v1. -Writes per-pair manifests to manifests/pair_{1,2,3,4}.json plus a -combined manifest (manifests/combined.json) with deduped papers and -pair-membership metadata — that combined manifest is what the downloader -and runner consume. +Writes per-pair manifests (pair_{1,2,3,4}.json) plus a full-set manifest +(full.json) with deduped papers and pair-membership metadata into the output +dir (default manifests/canonical/). full.json is what the downloader and runner +consume. Pairs: 1 top-cited (high) vs never-published (low) community-wide @@ -40,7 +40,7 @@ from snor_loader import load_cohort HERE = Path(__file__).resolve().parent -MANIFESTS_DIR = HERE / "manifests" / "v1" +MANIFESTS_DIR = HERE / "manifests" / "canonical" CURRENT_YEAR = 2026 @@ -421,18 +421,22 @@ def build_manifests( "years": years, "n_per_tail": n_per_tail, "papers": combined_papers, + # Efficient roster run on the full set. The two frontier models + # (openai/gpt-5.5, anthropic/claude-opus-4.7) run only on the frontier + # subset (manifests/canonical/frontier.json), which pins all six. "models": [ - "google/gemini-3-flash-preview", - "z-ai/glm-4.6", - "qwen/qwen3-235b-a22b-2507", + "deepseek/deepseek-v4-flash", + "qwen/qwen3.6-35b-a3b", + "google/gemini-3.1-flash-lite-preview", + "x-ai/grok-4.1-fast", ], "review_caps": {"max_pages": 20, "max_tokens": 20_000}, } - (out_dir / "combined.json").write_text( + (out_dir / "full.json").write_text( json.dumps(combined, indent=2) + "\n" ) print(f"\nCombined manifest: {len(combined_papers)} unique papers " - f"-> {out_dir / 'combined.json'}") + f"-> {out_dir / 'full.json'}") def main() -> None: diff --git a/benchmarks/perturbation/.gitignore b/benchmarks/perturbation/.gitignore index 5cce8cb..db8465e 100644 --- a/benchmarks/perturbation/.gitignore +++ b/benchmarks/perturbation/.gitignore @@ -9,8 +9,11 @@ data/ # Reports (regeneratable from results/) reports/ -# Local experiment configs (we don't check any of these in). -configs/ +# Experiment configs are local except the committed example and the per-domain +# configs that reproduce the paper (run with --configs configs/full_*.yaml). +configs/* +!configs/default.yaml +!configs/full_*.yaml # Python __pycache__/ diff --git a/benchmarks/perturbation/README.md b/benchmarks/perturbation/README.md index a9521f3..7e9aa8c 100644 --- a/benchmarks/perturbation/README.md +++ b/benchmarks/perturbation/README.md @@ -1,126 +1,166 @@ -# Perturbation Benchmark (v2) +# Perturbation Benchmark -Benchmark for evaluating how well LLM reviewers detect **seeded errors** in mathematical papers. The pipeline takes real arxiv papers, injects controlled perturbations, runs automated reviews, and measures recall. +Do AI review systems catch errors deliberately injected into otherwise-good +papers? This benchmark takes real arXiv papers, injects controlled errors, runs +automated reviews, and measures recall (the fraction of injected errors a system +detects). + +See the paper for full method and results ([arXiv:2606.19749](https://arxiv.org/abs/2606.19749)). This README covers how to +reproduce a run. ## Pipeline ``` -extract → generate → validate → inject → review → score +extract → generate → validate → verify → inject → review → score ``` -1. **Extract** (`extract.py`): Identify candidate math spans (equations, symbols) in the paper. -2. **Generate** (`generate.py`): Use an LLM to create perturbations for each candidate span. -3. **Validate** (`validate.py`): Check that perturbations are valid (original exists, no overlaps, no garbled text). -4. **Inject** (`inject.py`): Apply valid perturbations to produce a corrupted paper. -5. **Review**: Run `openaireview review` on the corrupted paper with each (model, method) combination. -6. **Score** (`score.py`): Compare review comments against the perturbation manifest using fuzzy substring matching + LLM-as-judge. +- **extract → generate → validate** (`perturb_automated.py`, the `openaireview perturb` CLI): find candidate spans in a paper, generate candidate errors, and +apply the structural checks, writing a ground-truth manifest per paper. +- **verify** (`analyses/verify_existing.py`): rate each surviving error and drop +the ones that read as typos or do not constitute a real error. +- **inject** (`reinject_existing.py`): apply the kept errors to the clean source, +writing a corrupted paper (`*_recorrupted.md`) plus its filtered manifest +(`*_kept_perturbations.json`). These files are the input to the review stage. +- **review** (`run_benchmark.py`): run each (model, method) over the corrupted paper. +- **score** (`run_benchmark.py`): match each review comment against the manifest +and compute recall. + +## Dataset + +The recommended way to reproduce the paper is to use the released perturbed +papers, so most users can skip generation entirely. The exact set is published as +a zip archive on +[Google Drive](https://drive.google.com/file/d/10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3/view?usp=sharing) +(download with `gdown 10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3`). Each paper +ships as a corrupted markdown file (`*_recorrupted.md`) and a ground-truth +manifest (`*_kept_perturbations.json`), laid out as +`/all///`. Unzip it inside `benchmarks/perturbation/` +so it lands at `data/perturbations_filtered/`, then run every domain at once with +the committed per-domain configs (each `full_*.yaml` already points `input_dir` at +its split): -## Error Types +```bash +cd benchmarks/perturbation +# unzip the released archive here first (creates data/perturbations_filtered/) +python run_benchmark.py --configs configs/full_*.yaml --stages prepare,review,score,report +``` -### Surface errors -Minimal single-token edits to math expressions: -- `operator_or_sign` — flip `+`/`-`, `≤`/`≥`, `∪`/`∩` -- `symbol_binding` — swap a symbol (`α`→`β`) -- `index_or_subscript` — change sub/superscript (`x_i`→`x_{i+1}`) -- `numeric_parameter` — change a number (`0.5`→`0.25`) +For a single domain, pass one config instead, for example +`run_benchmark.py configs/full_math_all.yaml --stages ...`. -### Formal errors -Deeper structural corruptions to definitions, theorems, and proofs: -- `def_wrong`, `thm_wrong_condition`, `thm_wrong_conclusion`, `thm_wrong_scope` -- `proof_wrong_direction`, `proof_missing_case`, `proof_wrong_assumption`, `proof_mismatch` +The released set was filtered with an LLM verifier (the verify stage below) and +spot-checked by a manual audit (82.5% of a 40-perturbation sample judged valid +errors, see the paper). To build a fresh set of your own instead, follow the +Quick start below. Error generation is model-driven, so a fresh run produces +different papers and errors from the released set. -## Quick Start +## Error types -```bash -# Install benchmark dependencies -pip install -e ".[benchmarks]" +Injected errors span four categories (the underlying error type is in parentheses): -# Run a single config (prepare → review → score → report) -python benchmarks/perturbation/run_benchmark.py benchmarks/perturbation/configs/default.yaml +- **Surface** (`surface`): minimal single-token math edits (flip an operator or sign, change a number, alter a subscript). +- **Claim** (`statement_empirical`, `claim_theoretical`): a stated empirical or theoretical claim is made wrong. +- **Reasoning** (`logic`): a logical or derivation step is broken. +- **Experimental** (`experimental`): an experimental-design or setup detail is corrupted. -# Run only specific stages -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages prepare,review -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages score +## Quick start -# Run many domains in one process (workers stay busy across config boundaries) -python benchmarks/perturbation/run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 -``` - -`run_benchmark.py` selects a review system per config via the `system:` field -(`openaireview` | `coarse` | `reviewer3`); see `systems/README.md` for setup -of the third-party systems. +Install the benchmark dependencies and set an API key: -## Configuration +```bash +uv pip install -e ".[benchmarks]" +export OPENROUTER_API_KEY=... +``` -Configs are YAML files in `configs/`. Copy `default.yaml` and edit to create experiment variants. Committed configs serve as the experiment log. +This rebuilds the perturbed papers from scratch. Most users do not need it: the +released set (see Dataset above) is the recommended input. Regenerate only to +make a fresh set. Steps 1-3 build the perturbed papers (the generation pipeline), +steps 4-7 run the benchmark over them. Step 2, verify, is an LLM filter that drops +non-substantive perturbations; it is how the released set was built, and inject +keeps only the perturbations it marks substantive. The verify and inject steps are +package modules, so they run from the repo root (the `(cd ../.. && ...)` subshells +below); the others run from this directory. -```yaml -max_papers: 5 -length: short # short (2k-7k words) | medium (7k-17k) | long (>17k) -error_type: surface # surface | formal | all -score_method: llm # llm | fuzzy | semantic +```bash +cd benchmarks/perturbation + +# 1. extract → generate → validate: sample papers and write a clean copy plus a +# candidate manifest (_clean.md + _perturbations.json) per paper. Generation +# is model-driven, so this produces a fresh set each run. +python perturb_automated.py --arxiv-category "math.*" --category theoretical \ + --error-type all --target 10 --min-year 2015 \ + --output-dir data/perturbations/math_all + +# 2. verify: rate each candidate error and record verdicts under +# results/error_verification/. +(cd ../.. && python -m benchmarks.perturbation.analyses.verify_existing) + +# 3. inject: apply the kept (substantive) errors to the clean source, writing the +# corrupted papers (_recorrupted.md) and filtered manifests +# (_kept_perturbations.json). This is the benchmark input. +(cd ../.. && python -m benchmarks.perturbation.reinject_existing \ + --output-root benchmarks/perturbation/data/perturbations_filtered) + +# 4-7. the config's input_dir already points at the injected set +# (data/perturbations_filtered/math_all/all); run the benchmark stages. +python run_benchmark.py configs/default.yaml --stages prepare # 4. stage corrupted papers +python run_benchmark.py configs/default.yaml --stages review # 5. run the reviews +python run_benchmark.py configs/default.yaml --stages score # 6. score against the manifests +python run_benchmark.py configs/default.yaml --stages report # 7. aggregate the recall tables +``` -perturb_model: google/gemini-3-flash-preview -score_model: google/gemini-3-flash-preview +Run several stages at once with `--stages prepare,review,score,report`, or sweep +many configs with `--configs configs/*.yaml`. `run_benchmark.py` selects the +review system per config via the `system:` field (`openaireview` | `coarse` | +`reviewer3`). See `systems/README.md` for setting up the external systems. To skip +generation entirely, download the released set (see Dataset above) and point +`input_dir` at it. -review_models: - - google/gemini-3-flash-preview - - z-ai/glm-4.6 +## Config schema -review_methods: +```yaml +system: openaireview # openaireview | coarse | reviewer3 +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all # injected papers from reinject_existing.py (or the released set) +results_dir: results/default # staged papers, reviews, scores + +models: # backbone models to review with + - openai/gpt-5.5 + - deepseek/deepseek-v4-flash +methods: # zero_shot | local | progressive - zero_shot - progressive -results_dir: benchmarks/perturbation/results +score_method: llm # llm | fuzzy | semantic +score_model: google/gemini-3-flash-preview # the llm judge ``` -Papers are streamed from the [proof-pile](https://huggingface.co/datasets/hoskinson-center/proof-pile) dataset and binned by word count. - -## Results Layout +Configs in `configs/` are local except the committed `default.yaml` and the +per-domain `full_*.yaml` set (one per paper domain), which reproduce the paper via +`--configs configs/full_*.yaml`. Copy any of them to define experiment variants. -``` -/ - config.yaml # resolved config - perturb//paper_001/ - paper_001.md # original paper - paper-001_clean.md # clean copy - paper-001_corrupted.md # with injected errors - paper-001_perturbations.json # ground-truth manifest - ///paper_001/ - review/*.json # review results - score//*_score.json # recall scores -``` +## Scoring -## Reports +A comment counts as detecting an injected error when it passes two stages: -Experiment reports are in `reports/`. See `reports/surface_3models_short_medium.md` for the first benchmark run (3 cheap models, 10 papers, surface errors). +1. **Substring match** — the perturbed text approximately covers the comment's + quote (normalized, 0.75 coverage), so the judge only sees comments pointing at + the right span. +2. **LLM judge** — Gemini-3 Flash Preview rates whether the comment's explanation + identifies the same error as the manifest's `why_wrong`, on a 1-5 scale, and a + rating >= 3 counts as detected. -### Generating report statistics +Recall is the fraction of injected errors detected, reported per (model, method) +and per error category by the `report` stage. -After a pipeline run completes, use `generate_report.py` to aggregate results across all papers, models, and methods: +## Results layout -```bash -python benchmarks/perturbation/generate_report.py benchmarks/perturbation/results_short +``` +/ + perturb//paper_NNN/ + paper_NNN_corrupted.md # injected paper + paper_NNN_perturbations.json # ground-truth manifest + ///paper_NNN/ + review/*.json # review output + score//*_score.json # recall scores ``` -This prints markdown tables to stdout covering: -- Configuration summary -- Ground truth counts by length and error type -- Recall by model × method (split by paper length and overall) -- Recall by error type -- Token usage and cost - -The output is meant to be reviewed and edited into a final report in `reports/`. - -## Scoring - -The scorer uses a two-stage filter: -1. **Fuzzy substring match** — checks if the perturbed text appears (approximately) in the review comment's quote, using normalized text coverage with a 0.75 threshold. -2. **LLM-as-judge** — asks a model to rate whether the reviewer's explanation identifies the same error described in the perturbation's `why_wrong` field (score >= 3/5 = match). - -## Known Limitations - -- The perturb stage targets `n_per_error=2` perturbations per error type (8 total per paper), but the LLM often reuses the same candidate span for both, causing the validator to reject the duplicate. Typical yield is ~4-5 per paper. -- Fuzzy substring matching can miss catches where the reviewer heavily paraphrases the quoted text. -- `cost_usd` from OpenRouter metadata is unreliable for some models (notably qwen). diff --git a/benchmarks/perturbation/configs/default.yaml b/benchmarks/perturbation/configs/default.yaml new file mode 100644 index 0000000..2442be3 --- /dev/null +++ b/benchmarks/perturbation/configs/default.yaml @@ -0,0 +1,26 @@ +# Default perturbation benchmark config (one experiment = one config). +# +# input_dir: injected papers + filtered manifests from reinject_existing.py, or +# the released dataset. Resolved relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all +results_dir: results/default + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_cs_CC.yaml b/benchmarks/perturbation/configs/full_cs_CC.yaml new file mode 100644 index 0000000..44d2f84 --- /dev/null +++ b/benchmarks/perturbation/configs/full_cs_CC.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the cs_CC split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/cs_CC/all +results_dir: results/full_cs_CC + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_cs_LG.yaml b/benchmarks/perturbation/configs/full_cs_LG.yaml new file mode 100644 index 0000000..5c82dc9 --- /dev/null +++ b/benchmarks/perturbation/configs/full_cs_LG.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the cs_LG split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/cs_LG/all +results_dir: results/full_cs_LG + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_econ_EM.yaml b/benchmarks/perturbation/configs/full_econ_EM.yaml new file mode 100644 index 0000000..43e50ba --- /dev/null +++ b/benchmarks/perturbation/configs/full_econ_EM.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the econ_EM split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/econ_EM/all +results_dir: results/full_econ_EM + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_hep-ex.yaml b/benchmarks/perturbation/configs/full_hep-ex.yaml new file mode 100644 index 0000000..931588e --- /dev/null +++ b/benchmarks/perturbation/configs/full_hep-ex.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the hep-ex split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/hep-ex/all +results_dir: results/full_hep-ex + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_math_all.yaml b/benchmarks/perturbation/configs/full_math_all.yaml new file mode 100644 index 0000000..0ba8d48 --- /dev/null +++ b/benchmarks/perturbation/configs/full_math_all.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the math_all split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all +results_dir: results/full_math_all + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_physics_atm-clus.yaml b/benchmarks/perturbation/configs/full_physics_atm-clus.yaml new file mode 100644 index 0000000..6987c8a --- /dev/null +++ b/benchmarks/perturbation/configs/full_physics_atm-clus.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the physics_atm-clus split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/physics_atm-clus/all +results_dir: results/full_physics_atm-clus + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_q-bio_GN.yaml b/benchmarks/perturbation/configs/full_q-bio_GN.yaml new file mode 100644 index 0000000..e30faf0 --- /dev/null +++ b/benchmarks/perturbation/configs/full_q-bio_GN.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the q-bio_GN split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/q-bio_GN/all +results_dir: results/full_q-bio_GN + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_stat_AP.yaml b/benchmarks/perturbation/configs/full_stat_AP.yaml new file mode 100644 index 0000000..328e95c --- /dev/null +++ b/benchmarks/perturbation/configs/full_stat_AP.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the stat_AP split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/stat_AP/all +results_dir: results/full_stat_AP + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview